linux/kernel/rcu
Paul E. McKenney 4a81e8328d rcu: Reduce overhead of cond_resched() checks for RCU
Commit ac1bea8578 (Make cond_resched() report RCU quiescent states)
fixed a problem where a CPU looping in the kernel with but one runnable
task would give RCU CPU stall warnings, even if the in-kernel loop
contained cond_resched() calls.  Unfortunately, in so doing, it introduced
performance regressions in Anton Blanchard's will-it-scale "open1" test.
The problem appears to be not so much the increased cond_resched() path
length as an increase in the rate at which grace periods complete, which
increased per-update grace-period overhead.

This commit takes a different approach to fixing this bug, mainly by
moving the RCU-visible quiescent state from cond_resched() to
rcu_note_context_switch(), and by further reducing the check to a
simple non-zero test of a single per-CPU variable.  However, this
approach requires that the force-quiescent-state processing send
resched IPIs to the offending CPUs.  These will be sent only once
the grace period has reached an age specified by the boot/sysfs
parameter rcutree.jiffies_till_sched_qs, or once the grace period
reaches an age halfway to the point at which RCU CPU stall warnings
will be emitted, whichever comes first.

Reported-by: Dave Hansen <dave.hansen@intel.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Christoph Lameter <cl@gentwo.org>
Cc: Mike Galbraith <umgwanakikbuti@gmail.com>
Cc: Eric Dumazet <eric.dumazet@gmail.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
[ paulmck: Made rcu_momentary_dyntick_idle() as suggested by the
  ktest build robot.  Also fixed smp_mb() comment as noted by
  Oleg Nesterov. ]

Merge with e552592e (Reduce overhead of cond_resched() checks for RCU)

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
2014-06-23 11:19:32 -07:00
..
Makefile rcutorture: Abstract rcu_torture_random() 2014-02-23 09:00:58 -08:00
rcu.h rcu: Ensure kernel/rcu/rcu.h can be sourced/used stand-alone 2014-02-26 06:35:18 -08:00
rcutorture.c torture: Check for multiple concurrent torture tests 2014-05-14 09:46:29 -07:00
srcu.c rcu: Ensure kernel/rcu/rcu.h can be sourced/used stand-alone 2014-02-26 06:35:18 -08:00
tiny_plugin.h rcu: Protect uses of ->jiffies_stall with ACCESS_ONCE() 2014-04-29 08:44:41 -07:00
tiny.c rcu: Ensure kernel/rcu/rcu.h can be sourced/used stand-alone 2014-02-26 06:35:18 -08:00
tree_plugin.h rcu: Reduce overhead of cond_resched() checks for RCU 2014-06-23 11:19:32 -07:00
tree_trace.c rcu: Stop tracking FSF's postal address 2014-02-17 15:01:37 -08:00
tree.c rcu: Reduce overhead of cond_resched() checks for RCU 2014-06-23 11:19:32 -07:00
tree.h rcu: Reduce overhead of cond_resched() checks for RCU 2014-06-23 11:19:32 -07:00
update.c rcu: Reduce overhead of cond_resched() checks for RCU 2014-06-23 11:19:32 -07:00