mirror of
https://github.com/torvalds/linux.git
synced 2024-11-24 21:21:41 +00:00
f0e1a0643a
Implement a new scheduler class sched_ext (SCX), which allows scheduling
policies to be implemented as BPF programs to achieve the following:
1. Ease of experimentation and exploration: Enabling rapid iteration of new
scheduling policies.
2. Customization: Building application-specific schedulers which implement
policies that are not applicable to general-purpose schedulers.
3. Rapid scheduler deployments: Non-disruptive swap outs of scheduling
policies in production environments.
sched_ext leverages BPF’s struct_ops feature to define a structure which
exports function callbacks and flags to BPF programs that wish to implement
scheduling policies. The struct_ops structure exported by sched_ext is
struct sched_ext_ops, and is conceptually similar to struct sched_class. The
role of sched_ext is to map the complex sched_class callbacks to the more
simple and ergonomic struct sched_ext_ops callbacks.
For more detailed discussion on the motivations and overview, please refer
to the cover letter.
Later patches will also add several example schedulers and documentation.
This patch implements the minimum core framework to enable implementation of
BPF schedulers. Subsequent patches will gradually add functionalities
including safety guarantee mechanisms, nohz and cgroup support.
include/linux/sched/ext.h defines struct sched_ext_ops. With the comment on
top, each operation should be self-explanatory. The followings are worth
noting:
- Both "sched_ext" and its shorthand "scx" are used. If the identifier
already has "sched" in it, "ext" is used; otherwise, "scx".
- In sched_ext_ops, only .name is mandatory. Every operation is optional and
if omitted a simple but functional default behavior is provided.
- A new policy constant SCHED_EXT is added and a task can select sched_ext
by invoking sched_setscheduler(2) with the new policy constant. However,
if the BPF scheduler is not loaded, SCHED_EXT is the same as SCHED_NORMAL
and the task is scheduled by CFS. When the BPF scheduler is loaded, all
tasks which have the SCHED_EXT policy are switched to sched_ext.
- To bridge the workflow imbalance between the scheduler core and
sched_ext_ops callbacks, sched_ext uses simple FIFOs called dispatch
queues (dsq's). By default, there is one global dsq (SCX_DSQ_GLOBAL), and
one local per-CPU dsq (SCX_DSQ_LOCAL). SCX_DSQ_GLOBAL is provided for
convenience and need not be used by a scheduler that doesn't require it.
SCX_DSQ_LOCAL is the per-CPU FIFO that sched_ext pulls from when putting
the next task on the CPU. The BPF scheduler can manage an arbitrary number
of dsq's using scx_bpf_create_dsq() and scx_bpf_destroy_dsq().
- sched_ext guarantees system integrity no matter what the BPF scheduler
does. To enable this, each task's ownership is tracked through
p->scx.ops_state and all tasks are put on scx_tasks list. The disable path
can always recover and revert all tasks back to CFS. See p->scx.ops_state
and scx_tasks.
- A task is not tied to its rq while enqueued. This decouples CPU selection
from queueing and allows sharing a scheduling queue across an arbitrary
subset of CPUs. This adds some complexities as a task may need to be
bounced between rq's right before it starts executing. See
dispatch_to_local_dsq() and move_task_to_local_dsq().
- One complication that arises from the above weak association between task
and rq is that synchronizing with dequeue() gets complicated as dequeue()
may happen anytime while the task is enqueued and the dispatch path might
need to release the rq lock to transfer the task. Solving this requires a
bit of complexity. See the logic around p->scx.sticky_cpu and
p->scx.ops_qseq.
- Both enable and disable paths are a bit complicated. The enable path
switches all tasks without blocking to avoid issues which can arise from
partially switched states (e.g. the switching task itself being starved).
The disable path can't trust the BPF scheduler at all, so it also has to
guarantee forward progress without blocking. See scx_ops_enable() and
scx_ops_disable_workfn().
- When sched_ext is disabled, static_branches are used to shut down the
entry points from hot paths.
v7: - scx_ops_bypass() was incorrectly and unnecessarily trying to grab
scx_ops_enable_mutex which can lead to deadlocks in the disable path.
Fixed.
- Fixed TASK_DEAD handling bug in scx_ops_enable() path which could lead
to use-after-free.
- Consolidated per-cpu variable usages and other cleanups.
v6: - SCX_NR_ONLINE_OPS replaced with SCX_OPI_*_BEGIN/END so that multiple
groups can be expressed. Later CPU hotplug operations are put into
their own group.
- SCX_OPS_DISABLING state is replaced with the new bypass mechanism
which allows temporarily putting the system into simple FIFO
scheduling mode bypassing the BPF scheduler. In addition to the shut
down path, this will also be used to isolate the BPF scheduler across
PM events. Enabling and disabling the bypass mode requires iterating
all runnable tasks. rq->scx.runnable_list addition is moved from the
later watchdog patch.
- ops.prep_enable() is replaced with ops.init_task() and
ops.enable/disable() are now called whenever the task enters and
leaves sched_ext instead of when the task becomes schedulable on
sched_ext and stops being so. A new operation - ops.exit_task() - is
called when the task stops being schedulable on sched_ext.
- scx_bpf_dispatch() can now be called from ops.select_cpu() too. This
removes the need for communicating local dispatch decision made by
ops.select_cpu() to ops.enqueue() via per-task storage.
SCX_KF_SELECT_CPU is added to support the change.
- SCX_TASK_ENQ_LOCAL which told the BPF scheudler that
scx_select_cpu_dfl() wants the task to be dispatched to the local DSQ
was removed. Instead, scx_bpf_select_cpu_dfl() now dispatches directly
if it finds a suitable idle CPU. If such behavior is not desired,
users can use scx_bpf_select_cpu_dfl() which returns the verdict in a
bool out param.
- scx_select_cpu_dfl() was mishandling WAKE_SYNC and could end up
queueing many tasks on a local DSQ which makes tasks to execute in
order while other CPUs stay idle which made some hackbench numbers
really bad. Fixed.
- The current state of sched_ext can now be monitored through files
under /sys/sched_ext instead of /sys/kernel/debug/sched/ext. This is
to enable monitoring on kernels which don't enable debugfs.
- sched_ext wasn't telling BPF that ops.dispatch()'s @prev argument may
be NULL and a BPF scheduler which derefs the pointer without checking
could crash the kernel. Tell BPF. This is currently a bit ugly. A
better way to annotate this is expected in the future.
- scx_exit_info updated to carry pointers to message buffers instead of
embedding them directly. This decouples buffer sizes from API so that
they can be changed without breaking compatibility.
- exit_code added to scx_exit_info. This is used to indicate different
exit conditions on non-error exits and will be used to handle e.g. CPU
hotplugs.
- The patch "sched_ext: Allow BPF schedulers to switch all eligible
tasks into sched_ext" is folded in and the interface is changed so
that partial switching is indicated with a new ops flag
%SCX_OPS_SWITCH_PARTIAL. This makes scx_bpf_switch_all() unnecessasry
and in turn SCX_KF_INIT. ops.init() is now called with
SCX_KF_SLEEPABLE.
- Code reorganized so that only the parts necessary to integrate with
the rest of the kernel are in the header files.
- Changes to reflect the BPF and other kernel changes including the
addition of bpf_sched_ext_ops.cfi_stubs.
v5: - To accommodate 32bit configs, p->scx.ops_state is now atomic_long_t
instead of atomic64_t and scx_dsp_buf_ent.qseq which uses
load_acquire/store_release is now unsigned long instead of u64.
- Fix the bug where bpf_scx_btf_struct_access() was allowing write
access to arbitrary fields.
- Distinguish kfuncs which can be called from any sched_ext ops and from
anywhere. e.g. scx_bpf_pick_idle_cpu() can now be called only from
sched_ext ops.
- Rename "type" to "kind" in scx_exit_info to make it easier to use on
languages in which "type" is a reserved keyword.
- Since
|
||
|---|---|---|
| .. | ||
| bitops | ||
| vdso | ||
| access_ok.h | ||
| agp.h | ||
| archrandom.h | ||
| asm-offsets.h | ||
| asm-prototypes.h | ||
| atomic64.h | ||
| atomic.h | ||
| audit_change_attr.h | ||
| audit_dir_write.h | ||
| audit_read.h | ||
| audit_signal.h | ||
| audit_write.h | ||
| barrier.h | ||
| bitops.h | ||
| bitsperlong.h | ||
| bug.h | ||
| cache.h | ||
| cacheflush.h | ||
| cfi.h | ||
| checksum.h | ||
| cmpxchg-local.h | ||
| cmpxchg.h | ||
| codetag.lds.h | ||
| compat.h | ||
| current.h | ||
| delay.h | ||
| device.h | ||
| div64.h | ||
| dma-mapping.h | ||
| dma.h | ||
| early_ioremap.h | ||
| emergency-restart.h | ||
| error-injection.h | ||
| exec.h | ||
| extable.h | ||
| fixmap.h | ||
| flat.h | ||
| ftrace.h | ||
| futex.h | ||
| getorder.h | ||
| hardirq.h | ||
| hugetlb.h | ||
| hw_irq.h | ||
| hyperv-tlfs.h | ||
| int-ll64.h | ||
| io.h | ||
| ioctl.h | ||
| iomap.h | ||
| irq_regs.h | ||
| irq_work.h | ||
| irq.h | ||
| irqflags.h | ||
| Kbuild | ||
| kdebug.h | ||
| kmap_size.h | ||
| kprobes.h | ||
| kvm_para.h | ||
| kvm_types.h | ||
| linkage.h | ||
| local64.h | ||
| local.h | ||
| logic_io.h | ||
| mcs_spinlock.h | ||
| memory_model.h | ||
| mm_hooks.h | ||
| mmiowb_types.h | ||
| mmiowb.h | ||
| mmu_context.h | ||
| mmu.h | ||
| module.h | ||
| module.lds.h | ||
| mshyperv.h | ||
| msi.h | ||
| nommu_context.h | ||
| numa.h | ||
| param.h | ||
| parport.h | ||
| pci_iomap.h | ||
| pci.h | ||
| percpu.h | ||
| pgalloc.h | ||
| pgtable_uffd.h | ||
| pgtable-nop4d.h | ||
| pgtable-nopmd.h | ||
| pgtable-nopud.h | ||
| preempt.h | ||
| qrwlock_types.h | ||
| qrwlock.h | ||
| qspinlock_types.h | ||
| qspinlock.h | ||
| resource.h | ||
| rwonce.h | ||
| seccomp.h | ||
| sections.h | ||
| serial.h | ||
| set_memory.h | ||
| shmparam.h | ||
| signal.h | ||
| simd.h | ||
| softirq_stack.h | ||
| spinlock_types.h | ||
| spinlock.h | ||
| statfs.h | ||
| string.h | ||
| switch_to.h | ||
| syscall.h | ||
| syscalls.h | ||
| timex.h | ||
| tlb.h | ||
| tlbflush.h | ||
| topology.h | ||
| trace_clock.h | ||
| uaccess.h | ||
| unaligned.h | ||
| user.h | ||
| vermagic.h | ||
| vga.h | ||
| video.h | ||
| vmlinux.lds.h | ||
| word-at-a-time.h | ||
| xor.h | ||