John Garry fea9f92f17 blk-mq: Optimise blk_mq_queue_tag_busy_iter() for shared tags ... Kashyap reports high CPU usage in blk_mq_queue_tag_busy_iter() and callees using megaraid SAS RAID card since moving to shared tags [0]. Previously, when shared tags was shared sbitmap, this function was less than optimum since we would iter through all tags for all hctx's, yet only ever match upto tagset depth number of rqs. Since the change to shared tags, things are even less efficient if we have parallel callers of blk_mq_queue_tag_busy_iter(). This is because in bt_iter() -> blk_mq_find_and_get_req() there would be more contention on accessing each request ref and tags->lock since they are now shared among all HW queues. Optimise by having separate calls to bt_for_each() for when we're using shared tags. In this case no longer pass a hctx, as it is no longer relevant, and teach bt_iter() about this. Ming suggested something along the lines of this change, apart from a different implementation. [0] https://lore.kernel.org/linux-block/e4e92abbe9d52bcba6b8cc6c91c442cc@mail.gmail.com/ Signed-off-by: John Garry <john.garry@huawei.com> Reviewed-by: Hannes Reinecke <hare@suse.de> Reviewed-by: Ming Lei <ming.lei@redhat.com> Reported-and-tested-by: Kashyap Desai <kashyap.desai@broadcom.com> Fixes: e155b0c238 ("blk-mq: Use shared tags for shared sbitmap support") Link: https://lore.kernel.org/r/1638794990-137490-4-git-send-email-john.garry@huawei.com Signed-off-by: Jens Axboe <axboe@kernel.dk>		2021-12-06 13:18:47 -07:00
..
partitions	block: remove GENHD_FL_EXT_DEVT	2021-11-29 06:38:35 -07:00
badblocks.c	treewide: Use fallthrough pseudo-keyword	2020-08-23 17:36:59 -05:00
bdev.c	block: Remove redundant initialization of variable ret	2021-11-29 06:41:29 -07:00
bfq-cgroup.c	block, bfq: fix UAF problem in bfqg_stats_init()	2021-10-19 15:18:30 -06:00
bfq-iosched.c	block: simplify ioc_lookup_icq	2021-11-29 06:41:29 -07:00
bfq-iosched.h	bfq: Provide helper to generate bfqq name	2021-11-29 06:38:52 -07:00
bfq-wf2q.c	block: Introduce IOPRIO_NR_LEVELS	2021-08-18 07:21:12 -06:00
bio-integrity.c	block: convert the rest of block to bdev_get_queue	2021-10-18 06:17:37 -06:00
bio.c	block: avoid extra iter advance with async iocb	2021-10-27 06:54:58 -06:00
blk-cgroup-rwstat.c	blk-cgroup: Fix the recursive blkg rwstat	2021-03-05 11:32:15 -07:00
blk-cgroup-rwstat.h	blk-cgroup: separate out blkg_rwstat under CONFIG_BLK_CGROUP_RWSTAT	2019-11-07 12:28:13 -07:00
blk-cgroup.c	block: don't include <linux/part_stat.h> in blk.h	2021-11-29 06:38:44 -07:00
blk-core.c	blk-mq: move srcu from blk_mq_hw_ctx to request_queue	2021-12-03 14:51:29 -07:00
blk-crypto-fallback.c	blk-crypto: rename blk_keyslot_manager to blk_crypto_profile	2021-10-21 10:49:32 -06:00
blk-crypto-internal.h	block: move struct request to blk-mq.h	2021-10-18 06:17:02 -06:00
blk-crypto-profile.c	blk-crypto: remove blk_crypto_unregister()	2021-11-29 06:38:51 -07:00
blk-crypto.c	blk-crypto: rename blk_keyslot_manager to blk_crypto_profile	2021-10-21 10:49:32 -06:00
blk-flush.c	block: switch to atomic_t for request references	2021-12-03 14:51:29 -07:00
blk-ia-ranges.c	block: fix kerneldoc for disk_register_independent_access__ranges()	2021-11-11 11:52:30 -07:00
blk-integrity.c	blk-crypto: remove blk_crypto_unregister()	2021-11-29 06:38:51 -07:00
blk-ioc.c	block: simplify ioc_lookup_icq	2021-11-29 06:41:29 -07:00
blk-iocost.c	block: convert the rest of block to bdev_get_queue	2021-10-18 06:17:37 -06:00
blk-iolatency.c	mm: don't include <linux/blk-cgroup.h> in <linux/backing-dev.h>	2021-10-18 06:17:01 -06:00
blk-ioprio.c	blk-ioprio: don't set bio priority if not needed	2021-11-29 06:38:35 -07:00
blk-ioprio.h	block: Introduce the ioprio rq-qos policy	2021-06-21 15:03:40 -06:00
blk-lib.c	block: export blk_next_bio()	2021-06-17 15:51:20 +02:00
blk-map.c	Merge branch 'akpm' (patches from Andrew)	2021-09-03 10:08:28 -07:00
blk-merge.c	block: don't check ->rq_disk in merges	2021-11-29 06:41:29 -07:00
blk-mq-cpumap.c	blk-mq: remove the calling of local_memory_node()	2020-10-20 07:08:17 -06:00
blk-mq-debugfs-zoned.c	block: Cleanup license notice	2019-01-17 21:21:40 -07:00
blk-mq-debugfs.c	blk-mq: check q->poll_stat in queue_poll_stat_show	2021-12-02 08:20:49 -07:00
blk-mq-debugfs.h	blk-mq: no need to check return value of debugfs_create functions	2019-06-13 03:00:30 -06:00
blk-mq-pci.c	block: Fix blk_mq_*_map_queues() kernel-doc headers	2019-05-31 15:12:34 -06:00
blk-mq-rdma.c	block: Fix blk_mq_*_map_queues() kernel-doc headers	2019-05-31 15:12:34 -06:00
blk-mq-sched.c	blk-mq: run dispatch lock once in case of issuing from list	2021-12-03 14:51:29 -07:00
blk-mq-sched.h	block: move blk_mq_sched_assign_ioc to blk-ioc.c	2021-11-29 06:41:29 -07:00
blk-mq-sysfs.c	blk-mq: move srcu from blk_mq_hw_ctx to request_queue	2021-12-03 14:51:29 -07:00
blk-mq-tag.c	blk-mq: Optimise blk_mq_queue_tag_busy_iter() for shared tags	2021-12-06 13:18:47 -07:00
blk-mq-tag.h	blk-mq: Delete busy_iter_fn	2021-12-06 13:18:47 -07:00
blk-mq-virtio.c	blk-mq: Fix typo in comment	2020-03-17 20:55:21 +01:00
blk-mq.c	blk-mq: Drop busy_iter_fn blk_mq_hw_ctx argument	2021-12-06 13:18:47 -07:00
blk-mq.h	blk-mq: don't run might_sleep() if the operation needn't blocking	2021-12-06 09:40:42 -07:00
blk-pm.c	scsi: block: Fix a race in the runtime power management code	2020-12-09 11:41:41 -05:00
blk-pm.h	block: Remove unused blk_pm_*() function definitions	2021-02-22 06:33:48 -07:00
blk-rq-qos.c	rq-qos: fix missed wake-ups in rq_qos_throttle try two	2021-06-08 15:12:57 -06:00
blk-rq-qos.h	block: only mark bio as tracked if it really is tracked	2021-10-18 08:50:47 -06:00
blk-settings.c	block: Fix partition check for host-aware zoned block devices	2021-10-27 06:58:01 -06:00
blk-stat.c	block: only allocate poll_stats if there's a user of them	2021-11-29 06:38:35 -07:00
blk-stat.h	block: only allocate poll_stats if there's a user of them	2021-11-29 06:38:35 -07:00
blk-sysfs.c	blk-mq: move srcu from blk_mq_hw_ctx to request_queue	2021-12-03 14:51:29 -07:00
blk-throttle.c	block: don't include blk-mq.h in blk.h	2021-11-29 06:38:44 -07:00
blk-throttle.h	block: move blk-throtl fast path inline	2021-10-18 06:17:03 -06:00
blk-timeout.c	block: blk-timeout: delete duplicated word	2020-07-31 16:29:47 -06:00
blk-wbt.c	blk-wbt: prevent NULL pointer dereference in wb_timer_fn	2021-10-19 06:13:41 -06:00
blk-wbt.h	blk-wbt: introduce a new disable state to prevent false positive by rwb_enabled()	2021-06-21 15:03:41 -06:00
blk-zoned.c	block: Hold invalidate_lock in BLKRESETZONE ioctl	2021-11-11 11:52:46 -07:00
blk.h	blk-mq: move srcu from blk_mq_hw_ctx to request_queue	2021-12-03 14:51:29 -07:00
bounce.c	mm: don't include <linux/blk-cgroup.h> in <linux/backing-dev.h>	2021-10-18 06:17:01 -06:00
bsg-lib.c	block: remove the gendisk argument to blk_execute_rq	2021-11-29 06:41:29 -07:00
bsg.c	scsi: bsg: Fix device unregistration	2021-09-14 00:22:15 -04:00
disk-events.c	block: return errors from disk_alloc_events	2021-08-23 12:55:45 -06:00
elevator.c	block: remove the e argument to elevator_exit	2021-11-29 06:38:44 -07:00
elevator.h	block: move elevator.h to block/	2021-10-18 06:17:01 -06:00
fops.c	block: move direct_IO into our own read_iter handler	2021-12-03 14:51:29 -07:00
genhd.c	blk-mq: move srcu from blk_mq_hw_ctx to request_queue	2021-12-03 14:51:29 -07:00
holder.c	block: drop unused includes in <linux/genhd.h>	2021-10-18 06:17:02 -06:00
ioctl.c	block: merge disk_scan_partitions and blkdev_reread_part	2021-11-29 06:35:21 -07:00
ioprio.c	block: Check ADMIN before NICE for IOPRIO_CLASS_RT	2021-11-15 14:28:59 -07:00
Kconfig	block: move menu "Partition type" to block/partitions/Kconfig	2021-10-18 06:17:35 -06:00
Kconfig.iosched	block: simplify Kconfig files	2021-10-18 06:17:35 -06:00
kyber-iosched.c	blk-mq: Stop using pointers for blk_mq_tags bitmap tags	2021-10-18 06:17:03 -06:00
Makefile	block: remove blk-exec.c	2021-11-29 06:34:50 -07:00
mq-deadline.c	blk-mq: Stop using pointers for blk_mq_tags bitmap tags	2021-10-18 06:17:03 -06:00
opal_proto.h	block: sed-opal: Change the check condition for regular session validity	2020-03-12 08:00:10 -06:00
sed-opal.c	block: sed-opal: Change the check condition for regular session validity	2020-03-12 08:00:10 -06:00
t10-pi.c	block: move integrity handling out of <linux/blkdev.h>	2021-10-18 06:17:02 -06:00