Minki/linux
1
0
Fork 1
mirror of https://github.com/torvalds/linux.git synced 2024-11-23 04:31:50 +00:00
Code Issues Packages Projects Releases Wiki Activity

block: only check previous entry for plug merge attempt

Browse Source 
Currently we scan the entire plug list, which is potentially very
expensive. In an IOPS bound workload, we can drive about 5.6M IOPS with
merging enabled, and profiling shows that the plug merge check is the
(by far) most expensive thing we're doing:

  Overhead  Command   Shared Object     Symbol
  +   20.89%  io_uring  [kernel.vmlinux]  [k] blk_attempt_plug_merge
  +    4.98%  io_uring  [kernel.vmlinux]  [k] io_submit_sqes
  +    4.78%  io_uring  [kernel.vmlinux]  [k] blkdev_direct_IO
  +    4.61%  io_uring  [kernel.vmlinux]  [k] blk_mq_submit_bio

Instead of browsing the whole list, just check the previously inserted
entry. That is enough for a naive merge check and will catch most cases,
and for devices that need full merging, the IO scheduler attached to
such devices will do that anyway. The plug merge is meant to be an
inexpensive check to avoid getting a request, but if we repeatedly
scan the list for every single insert, it is very much not a cheap
check.

With this patch, the workload instead runs at ~7.0M IOPS, providing
a 25% improvement. Disabling merging entirely yields another 5%
improvement.

Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
This commit is contained in:
Jens Axboe 2021-10-14 07:24:07 -06:00
parent 4c928904ff
commit d38a9c04c0
1 changed files with 13 additions and 23 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

36
block/blk-merge.c
View File

@ -1084,8 +1084,8 @@ static enum bio_merge_status blk_attempt_bio_merge(struct request_queue *q,
* another request associated with @q is found on the plug list * another request associated with @q is found on the plug list
* (optional, may be %NULL) * (optional, may be %NULL)
* *
* Determine whether @bio being queued on @q can be merged with a request * Determine whether @bio being queued on @q can be merged with the previous
* on %current's plugged list. Returns %true if merge was successful, * request on %current's plugged list. Returns %true if merge was successful,
* otherwise %false. * otherwise %false.
* *
* Plugging coalesces IOs from the same issuer for the same purpose without * Plugging coalesces IOs from the same issuer for the same purpose without
@ -1102,32 +1102,22 @@ bool blk_attempt_plug_merge(struct request_queue *q, struct bio *bio,
{ {
struct blk_plug *plug; struct blk_plug *plug;
struct request *rq; struct request *rq;
struct list_head *plug_list;
plug = blk_mq_plug(q, bio); plug = blk_mq_plug(q, bio);
if (!plug) if (!plug || list_empty(&plug->mq_list))
return false; return false;
plug_list = &plug->mq_list; /* check the previously added entry for a quick merge attempt */
rq = list_last_entry(&plug->mq_list, struct request, queuelist);
list_for_each_entry_reverse(rq, plug_list, queuelist) { if (rq->q == q && same_queue_rq) {
if (rq->q == q && same_queue_rq) { /*
/* * Only blk-mq multiple hardware queues case checks the rq in
* Only blk-mq multiple hardware queues case checks the * the same queue, there should be only one such rq in a queue
* rq in the same queue, there should be only one such */
* rq in a queue *same_queue_rq = rq;
**/
*same_queue_rq = rq;
}
if (rq->q != q)
continue;
if (blk_attempt_bio_merge(q, rq, bio, nr_segs, false) ==
BIO_MERGE_OK)
return true;
} }
if (blk_attempt_bio_merge(q, rq, bio, nr_segs, false) == BIO_MERGE_OK)
return true;
return false; return false;
} }