Files
linux/include/linux
Will Deacon dcb786493f locking/refcount: Improve performance of generic REFCOUNT_FULL code
Rewrite the generic REFCOUNT_FULL implementation so that the saturation
point is moved to INT_MIN / 2. This allows us to defer the sanity checks
until after the atomic operation, which removes many uses of cmpxchg()
in favour of atomic_fetch_{add,sub}().

Some crude perf results obtained from lkdtm show substantially less
overhead, despite the checking:

 $ perf stat -r 3 -B -- echo {ATOMIC,REFCOUNT}_TIMING >/sys/kernel/debug/provoke-crash/DIRECT

 # arm64
 ATOMIC_TIMING:                                      46.50451 +- 0.00134 seconds time elapsed  ( +-  0.00% )
 REFCOUNT_TIMING (REFCOUNT_FULL, mainline):          77.57522 +- 0.00982 seconds time elapsed  ( +-  0.01% )
 REFCOUNT_TIMING (REFCOUNT_FULL, this series):       48.7181  +- 0.0256  seconds time elapsed  ( +-  0.05% )

 # x86
 ATOMIC_TIMING:                                      31.6225 +- 0.0776 seconds time elapsed  ( +-  0.25% )
 REFCOUNT_TIMING (!REFCOUNT_FULL, mainline/x86 asm): 31.6689 +- 0.0901 seconds time elapsed  ( +-  0.28% )
 REFCOUNT_TIMING (REFCOUNT_FULL, mainline):          53.203  +- 0.138  seconds time elapsed  ( +-  0.26% )
 REFCOUNT_TIMING (REFCOUNT_FULL, this series):       31.7408 +- 0.0486 seconds time elapsed  ( +-  0.15% )

Signed-off-by: Will Deacon <will@kernel.org>
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Reviewed-by: Kees Cook <keescook@chromium.org>
Tested-by: Hanjun Guo <guohanjun@huawei.com>
Tested-by: Jan Glauber <jglauber@marvell.com>
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Cc: Elena Reshetova <elena.reshetova@intel.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: https://lkml.kernel.org/r/20191121115902.2551-6-will@kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2019-11-25 09:15:10 +01:00
..
2019-08-09 09:15:05 +02:00
2019-09-17 10:27:46 -07:00
2019-09-16 09:56:27 -07:00
2019-11-20 09:40:08 +01:00
2019-08-20 22:09:52 +02:00
2019-07-01 08:18:54 -06:00
2019-07-25 15:47:37 +02:00
2019-08-28 21:17:12 -06:00
2019-06-12 20:27:13 +02:00
2019-09-05 19:52:33 -06:00
2019-08-05 21:52:53 +02:00
2019-11-20 09:40:10 +01:00
2019-09-05 11:40:54 +02:00
2019-09-24 15:54:08 -07:00
2019-07-05 21:34:50 +02:00
2019-06-26 13:19:46 -07:00
2019-06-10 13:00:24 +02:00
2019-09-07 04:28:05 -03:00
2019-09-02 11:43:54 +01:00
2019-09-05 12:32:05 +02:00
2019-08-14 15:30:35 +02:00
2019-08-08 09:09:25 +02:00
2019-07-08 19:25:19 -07:00
2019-06-13 09:02:33 -04:00
2019-07-26 17:01:29 -06:00
2019-07-16 19:23:25 -07:00
2019-10-04 18:11:08 -07:00
2019-08-01 21:49:46 +02:00
2019-08-21 00:38:54 +02:00
2019-06-12 11:42:13 +02:00
2019-06-15 12:25:49 +02:00
2019-08-23 16:39:26 +01:00
2019-07-31 19:03:35 +02:00
2019-11-20 09:40:10 +01:00
2019-09-25 17:51:41 -07:00
2019-09-07 21:42:25 +02:00
2019-09-04 13:37:17 +02:00
2019-08-30 07:27:17 -07:00
2019-08-01 20:51:22 +02:00
2019-08-20 17:05:05 +02:00
2019-07-31 19:03:35 +02:00