mirror of
https://github.com/torvalds/linux.git
synced 2024-12-04 01:51:34 +00:00
0888f06ac9
Fernando Lopez-Lezcano reported frequent scheduling latencies and audio
xruns starting at the 2.6.18-rt kernel, and those problems persisted all
until current -rt kernels. The latencies were serious and unjustified by
system load, often in the milliseconds range.
After a patient and heroic multi-month effort of Fernando, where he
tested dozens of kernels, tried various configs, boot options,
test-patches of mine and provided latency traces of those incidents, the
following 'smoking gun' trace was captured by him:
_------=> CPU#
/ _-----=> irqs-off
| / _----=> need-resched
|| / _---=> hardirq/softirq
||| / _--=> preempt-depth
|||| /
||||| delay
cmd pid ||||| time | caller
\ / ||||| \ | /
IRQ_19-1479 1D..1 0us : __trace_start_sched_wakeup (try_to_wake_up)
IRQ_19-1479 1D..1 0us : __trace_start_sched_wakeup <<...>-5856> (37 0)
IRQ_19-1479 1D..1 0us : __trace_start_sched_wakeup (c01262ba 0 0)
IRQ_19-1479 1D..1 0us : resched_task (try_to_wake_up)
IRQ_19-1479 1D..1 0us : __spin_unlock_irqrestore (try_to_wake_up)
...
<idle>-0 1...1 11us!: default_idle (cpu_idle)
...
<idle>-0 0Dn.1 602us : smp_apic_timer_interrupt (c0103baf 1 0)
...
<...>-5856 0D..2 618us : __switch_to (__schedule)
<...>-5856 0D..2 618us : __schedule <<idle>-0> (20 162)
<...>-5856 0D..2 619us : __spin_unlock_irq (__schedule)
<...>-5856 0...1 619us : trace_stop_sched_switched (__schedule)
<...>-5856 0D..1 619us : trace_stop_sched_switched <<...>-5856> (37 0)
what is visible in this trace is that CPU#1 ran try_to_wake_up() for
PID:5856, it placed PID:5856 on CPU#0's runqueue and ran resched_task()
for CPU#0. But it decided to not send an IPI that no CPU - due to
TS_POLLING. But CPU#0 never woke up after its NEED_RESCHED bit was set,
and only rescheduled to PID:5856 upon the next lapic timer IRQ. The
result was a 600+ usecs latency and a missed wakeup!
the bug turned out to be an idle-wakeup bug introduced into the mainline
kernel this summer via an optimization in the x86_64 tree:
commit
|
||
|---|---|---|
| .. | ||
| dispatcher | ||
| events | ||
| executer | ||
| hardware | ||
| namespace | ||
| parser | ||
| resources | ||
| sleep | ||
| tables | ||
| utilities | ||
| ac.c | ||
| acpi_memhotplug.c | ||
| asus_acpi.c | ||
| battery.c | ||
| blacklist.c | ||
| bus.c | ||
| button.c | ||
| cm_sbs.c | ||
| container.c | ||
| debug.c | ||
| dock.c | ||
| ec.c | ||
| event.c | ||
| fan.c | ||
| glue.c | ||
| hotkey.c | ||
| i2c_ec.c | ||
| i2c_ec.h | ||
| ibm_acpi.c | ||
| Kconfig | ||
| Makefile | ||
| motherboard.c | ||
| numa.c | ||
| osl.c | ||
| pci_bind.c | ||
| pci_irq.c | ||
| pci_link.c | ||
| pci_root.c | ||
| power.c | ||
| processor_core.c | ||
| processor_idle.c | ||
| processor_perflib.c | ||
| processor_thermal.c | ||
| processor_throttling.c | ||
| sbs.c | ||
| scan.c | ||
| system.c | ||
| tables.c | ||
| thermal.c | ||
| toshiba_acpi.c | ||
| utils.c | ||
| video.c | ||