forked from Minki/linux
locking/qrwlock: Reduce reader/writer to reader lock transfer latency
Currently, a reader will check first to make sure that the writer mode byte is cleared before incrementing the reader count. That waiting is not really necessary. It increases the latency in the reader/writer to reader transition and reduces readers performance. This patch eliminates that waiting. It also has the side effect of reducing the chance of writer lock stealing and improving the fairness of the lock. Using a locking microbenchmark, a 10-threads 5M locking loop of mostly readers (RW ratio = 10,000:1) has the following performance numbers in a Haswell-EX box: Kernel Locking Rate (Kops/s) ------ --------------------- 4.1.1 15,063,081 4.1.1+patch 17,241,552 (+14.4%) Signed-off-by: Waiman Long <Waiman.Long@hp.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Douglas Hatch <doug.hatch@hp.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Scott J Norton <scott.norton@hp.com> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Link: http://lkml.kernel.org/r/1436459543-29126-2-git-send-email-Waiman.Long@hp.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
This commit is contained in:
parent
3b3fdf10a8
commit
ffffeaf318
@ -88,15 +88,11 @@ void queued_read_lock_slowpath(struct qrwlock *lock, u32 cnts)
|
||||
arch_spin_lock(&lock->lock);
|
||||
|
||||
/*
|
||||
* At the head of the wait queue now, wait until the writer state
|
||||
* goes to 0 and then try to increment the reader count and get
|
||||
* the lock. It is possible that an incoming writer may steal the
|
||||
* lock in the interim, so it is necessary to check the writer byte
|
||||
* to make sure that the write lock isn't taken.
|
||||
* At the head of the wait queue now, increment the reader count
|
||||
* and wait until the writer, if it has the lock, has gone away.
|
||||
* At ths stage, it is not possible for a writer to remain in the
|
||||
* waiting state (_QW_WAITING). So there won't be any deadlock.
|
||||
*/
|
||||
while (atomic_read(&lock->cnts) & _QW_WMASK)
|
||||
cpu_relax_lowlatency();
|
||||
|
||||
cnts = atomic_add_return(_QR_BIAS, &lock->cnts) - _QR_BIAS;
|
||||
rspin_until_writer_unlock(lock, cnts);
|
||||
|
||||
|
Loading…
Reference in New Issue
Block a user