Sebastian Andrzej Siewior b4c6f86ec2 irq_work: Handle some irq_work in a per-CPU thread on PREEMPT_RT ... The irq_work callback is invoked in hard IRQ context. By default all callbacks are scheduled for invocation right away (given supported by the architecture) except for the ones marked IRQ_WORK_LAZY which are delayed until the next timer-tick. While looking over the callbacks, some of them may acquire locks (spinlock_t, rwlock_t) which are transformed into sleeping locks on PREEMPT_RT and must not be acquired in hard IRQ context. Changing the locks into locks which could be acquired in this context will lead to other problems such as increased latencies if everything in the chain has IRQ-off locks. This will not solve all the issues as one callback has been noticed which invoked kref_put() and its callback invokes kfree() and this can not be invoked in hardirq context. Some callbacks are required to be invoked in hardirq context even on PREEMPT_RT to work properly. This includes for instance the NO_HZ callback which needs to be able to observe the idle context. The callbacks which require to be run in hardirq have already been marked. Use this information to split the callbacks onto the two lists on PREEMPT_RT: - lazy_list Work items which are not marked with IRQ_WORK_HARD_IRQ will be added to this list. Callbacks on this list will be invoked from a per-CPU thread. The handler here may acquire sleeping locks such as spinlock_t and invoke kfree(). - raised_list Work items which are marked with IRQ_WORK_HARD_IRQ will be added to this list. They will be invoked in hardirq context and must not acquire any sleeping locks. The wake up of the per-CPU thread occurs from irq_work handler/ hardirq context. The thread runs with lowest RT priority to ensure it runs before any SCHED_OTHER tasks do. [bigeasy: melt tglx's irq_work_tick_soft() which splits irq_work_tick() into a hard and soft variant. Collected fixes over time from Steven Rostedt and Mike Galbraith. Move to per-CPU threads instead of softirq as suggested by PeterZ.] Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://lkml.kernel.org/r/20211007092646.uhshe3ut2wkrcfzv@linutronix.de		2021-10-15 11:25:17 +02:00
..
bpf	bpf: Exempt CAP_BPF from checks against bpf_jit_limit	2021-09-28 09:28:37 +02:00
cgroup	bpf, cgroup: Assign cgroup in cgroup_sk_alloc when called from interrupt	2021-09-28 09:29:19 +02:00
configs	drivers/char: remove /dev/kmem for good	2021-05-07 00:26:34 -07:00
debug	kgdb patches for 5.15	2021-09-07 12:08:04 -07:00
dma	dma-mapping fixes for Linux 5.15	2021-09-17 11:54:48 -07:00
entry	entry: rseq: Call rseq_handle_notify_resume() in tracehook_notify_resume()	2021-09-22 10:24:01 -04:00
events	perf/core: fix userpage->time_enabled of inactive events	2021-10-01 13:57:54 +02:00
gcov	Kconfig: Introduce ARCH_WANTS_NO_INSTR and CC_HAS_NO_PROFILE_FN_ATTR	2021-06-22 11:07:18 -07:00
irq	irqchip fixes for 5.15, take #1	2021-09-24 14:11:04 +02:00
kcsan	LKMM updates:	2021-09-02 13:00:15 -07:00
livepatch	sched,livepatch: Use wake_up_if_idle()	2021-10-14 13:09:25 +02:00
locking	locking/rwbase: Take care of ordering guarantee for fastpath reader	2021-09-15 17:49:16 +02:00
power	Merge branches 'pm-pci', 'pm-sleep', 'pm-domains' and 'powercap'	2021-08-30 19:25:42 +02:00
printk	memblock: introduce saner 'memblock_free_ptr()' interface	2021-09-14 13:23:22 -07:00
rcu	sched,rcu: Rework try_invoke_on_locked_down_task()	2021-10-07 13:51:15 +02:00
sched	sched/rt: Annotate the RT balancing logic irqwork as IRQ_WORK_HARD_IRQ	2021-10-15 11:25:16 +02:00
time	posix-cpu-timers: Prevent spuriously armed 0-value itimer	2021-09-23 11:53:51 +02:00
trace	blktrace: Fix uaf in blk_trace access after removing by sysfs	2021-09-24 11:06:15 -06:00
.gitignore	.gitignore: prefix local generated files with a slash	2021-05-02 00:43:35 +09:00
acct.c	kernel/acct.c: use dedicated helper to access rlimit values	2021-09-08 11:50:26 -07:00
async.c	kernel/async.c: remove async_unregister_domain()	2021-05-07 00:26:33 -07:00
audit_fsnotify.c	audit_alloc_mark(): don't open-code ERR_CAST()	2021-02-23 10:25:27 -05:00
audit_tree.c	audit: move put_tree() to avoid trim_trees refcount underflow and UAF	2021-08-24 18:52:36 -04:00
audit_watch.c
audit.c	lsm: separate security_task_getsecid() into subjective and objective variants	2021-03-22 15:23:32 -04:00
audit.h	audit: add header protection to kernel/audit.h	2021-07-19 22:38:24 -04:00
auditfilter.c	lsm: separate security_task_getsecid() into subjective and objective variants	2021-03-22 15:23:32 -04:00
auditsc.c	audit: remove trailing spaces and tabs	2021-06-10 20:59:05 -04:00
backtracetest.c
bounds.c
capability.c
cfi.c	cfi: Use rcu_read_{un}lock_sched_notrace	2021-08-11 13:11:12 -07:00
compat.c	arch: remove compat_alloc_user_space	2021-09-08 15:32:35 -07:00
configs.c
context_tracking.c
cpu_pm.c	PM: cpu: Make notifier chain use a raw_spinlock_t	2021-08-16 18:55:32 +02:00
cpu.c	cpu/hotplug: Add debug printks for hotplug callback failures	2021-08-10 18:31:32 +02:00
crash_core.c	kdump: use vmlinux_build_id to simplify	2021-07-08 11:48:22 -07:00
crash_dump.c
cred.c	ucounts: Increase ucounts reference counter before the security hook	2021-08-23 16:13:04 -05:00
delayacct.c	delayacct: Add sysctl to enable at runtime	2021-05-12 11:43:25 +02:00
dma.c
exec_domain.c
exit.c	sched: Move kprobes cleanup out of finish_task_switch()	2021-10-05 15:52:14 +02:00
extable.c
fail_function.c
fork.c	kernel/sched: Fix sched_fork() access an invalid sched_task_group	2021-10-14 13:09:58 +02:00
freezer.c	sched: Add get_current_state()	2021-06-18 11:43:08 +02:00
futex.c	futex: Remove unused variable 'vpid' in futex_proxy_trylock_atomic()	2021-09-03 23:00:22 +02:00
gen_kheaders.sh	kbuild: clean up ${quiet} checks in shell scripts	2021-05-27 04:01:50 +09:00
groups.c	groups: simplify struct group_info allocation	2021-02-26 09:41:03 -08:00
hung_task.c	Merge branch 'akpm' (patches from Andrew)	2021-07-02 12:08:10 -07:00
iomem.c
irq_work.c	irq_work: Handle some irq_work in a per-CPU thread on PREEMPT_RT	2021-10-15 11:25:17 +02:00
jump_label.c	jump_label: Fix jump_label_text_reserved() vs __init	2021-07-05 10:46:20 +02:00
kallsyms.c	module: add printk formats to add module build ID to stacktraces	2021-07-08 11:48:22 -07:00
kcmp.c
Kconfig.freezer
Kconfig.hz
Kconfig.locks	locking/rwlock: Provide RT variant	2021-08-17 17:50:51 +02:00
Kconfig.preempt	sched: Provide Kconfig support for default dynamic preempt mode	2021-10-05 15:51:56 +02:00
kcov.c
kexec_core.c	Merge branch 'rework/printk_safe-removal' into for-linus	2021-08-30 16:36:10 +02:00
kexec_elf.c
kexec_file.c	kernel: kexec_file: fix error return code of kexec_calculate_store_digests()	2021-05-07 00:26:32 -07:00
kexec_internal.h	kexec: move machine_kexec_post_load() to public interface	2021-02-22 12:33:26 +00:00
kexec.c	kexec: avoid compat_alloc_user_space	2021-09-08 15:32:34 -07:00
kheaders.c
kmod.c	modules: add CONFIG_MODPROBE_PATH	2021-05-07 00:26:33 -07:00
kprobes.c	sched: Move kprobes cleanup out of finish_task_switch()	2021-10-05 15:52:14 +02:00
ksysfs.c
kthread.c	kthread: Move prio/affinite change into the newly created thread	2021-10-05 15:51:58 +02:00
latencytop.c
Makefile	kbuild: update config_data.gz only when the content of .config is changed	2021-05-02 00:43:35 +09:00
module_signature.c
module_signing.c
module-internal.h
module.c	printk: Userspace format indexing support	2021-07-19 11:57:48 +02:00
notifier.c	notifier: Remove atomic_notifier_call_chain_robust()	2021-08-16 18:55:32 +02:00
nsproxy.c	memcg: enable accounting for new namesapces and struct nsproxy	2021-09-03 09:58:12 -07:00
padata.c	padata: Remove repeated verbose license text	2021-08-27 16:30:18 +08:00
panic.c	Merge branch 'rework/printk_safe-removal' into for-linus	2021-08-30 16:36:10 +02:00
params.c	params: lift param_set_uint_minmax to common code	2021-08-16 14:42:22 +02:00
pid_namespace.c	memcg: enable accounting for new namesapces and struct nsproxy	2021-09-03 09:58:12 -07:00
pid.c	kernel/pid.c: implement additional checks upon pidfd_create() parameters	2021-08-10 12:53:07 +02:00
profile.c	profiling: fix shift-out-of-bounds bugs	2021-09-08 11:50:26 -07:00
ptrace.c	sched: Change task_struct::state	2021-06-18 11:43:09 +02:00
range.c
reboot.c	reboot: Add hardware protection power-off	2021-06-21 13:08:36 +01:00
regset.c
relay.c
resource_kunit.c
resource.c	kernel/resource: fix return code check in __request_free_mem_region	2021-05-14 19:41:32 -07:00
rseq.c	KVM: rseq: Update rseq when processing NOTIFY_RESUME on xfer to KVM guest	2021-09-22 10:24:01 -04:00
scftorture.c	scftorture: Avoid NULL pointer exception on early exit	2021-07-27 11:39:30 -07:00
scs.c
seccomp.c	Merge branch 'exit-cleanups-for-v5.15' of git://git.kernel.org/pub/scm/linux/kernel/git/ebiederm/user-namespace	2021-09-01 14:52:05 -07:00
signal.c	Merge branch 'akpm' (patches from Andrew)	2021-09-03 10:08:28 -07:00
smp.c	sched: Simplify wake_up_*idle*()	2021-10-07 13:51:15 +02:00
smpboot.c	smpboot: Replace deprecated CPU-hotplug functions.	2021-08-10 14:57:42 +02:00
smpboot.h
softirq.c	genirq: Change force_irqthreads to a static key	2021-08-10 22:50:07 +02:00
stackleak.c
stacktrace.c
static_call.c	static_call: Fix static_call_text_reserved() vs __init	2021-07-05 10:46:33 +02:00
stop_machine.c	stop_machine: Add caller debug info to queue_stop_cpus_work	2021-03-23 16:01:58 +01:00
sys_ni.c	compat: remove some compat entry points	2021-09-08 15:32:35 -07:00
sys.c	Merge branch 'akpm' (patches from Andrew)	2021-09-08 12:55:35 -07:00
sysctl-test.c	kernel/sysctl-test: Remove some casts which are no-longer required	2021-06-23 16:41:24 -06:00
sysctl.c	Merge branch 'akpm' (patches from Andrew)	2021-09-03 10:08:28 -07:00
task_work.c	kasan: record task_work_add() call stack	2021-04-30 11:20:42 -07:00
taskstats.c
test_kprobes.c
torture.c	torture: Replace deprecated CPU-hotplug functions.	2021-08-10 10:48:07 -07:00
tracepoint.c	tracepoint: Fix kerneldoc comments	2021-08-16 11:39:51 -04:00
tsacct.c
ucount.c	ucounts: add missing data type changes	2021-08-09 15:45:02 -05:00
uid16.c
uid16.h
umh.c	kernel/umh.c: fix some spelling mistakes	2021-05-07 00:26:34 -07:00
up.c	A set of locking related fixes and updates:	2021-05-09 13:07:03 -07:00
user_namespace.c	memcg: enable accounting for new namesapces and struct nsproxy	2021-09-03 09:58:12 -07:00
user-return-notifier.c
user.c	fs/epoll: use a per-cpu counter for user's watches count	2021-09-08 11:50:27 -07:00
usermode_driver.c	Merge branch 'work.namei' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs	2021-07-03 11:41:14 -07:00
utsname_sysctl.c
utsname.c
watch_queue.c
watchdog_hld.c
watchdog.c	kernel: watchdog: modify the explanation related to watchdog thread	2021-06-29 10:53:46 -07:00
workqueue_internal.h	workqueue: Assign a color to barrier work items	2021-08-17 07:49:10 -10:00
workqueue.c	workqueue: Assign a color to barrier work items	2021-08-17 07:49:10 -10:00