Mel Gorman cc9a6c8776 cpuset: mm: reduce large amounts of memory barrier related damage v3 ... Commit c0ff7453bb ("cpuset,mm: fix no node to alloc memory when changing cpuset's mems") wins a super prize for the largest number of memory barriers entered into fast paths for one commit. [get|put]_mems_allowed is incredibly heavy with pairs of full memory barriers inserted into a number of hot paths. This was detected while investigating at large page allocator slowdown introduced some time after 2.6.32. The largest portion of this overhead was shown by oprofile to be at an mfence introduced by this commit into the page allocator hot path. For extra style points, the commit introduced the use of yield() in an implementation of what looks like a spinning mutex. This patch replaces the full memory barriers on both read and write sides with a sequence counter with just read barriers on the fast path side. This is much cheaper on some architectures, including x86. The main bulk of the patch is the retry logic if the nodemask changes in a manner that can cause a false failure. While updating the nodemask, a check is made to see if a false failure is a risk. If it is, the sequence number gets bumped and parallel allocators will briefly stall while the nodemask update takes place. In a page fault test microbenchmark, oprofile samples from __alloc_pages_nodemask went from 4.53% of all samples to 1.15%. The actual results were 3.3.0-rc3 3.3.0-rc3 rc3-vanilla nobarrier-v2r1 Clients 1 UserTime 0.07 ( 0.00%) 0.08 (-14.19%) Clients 2 UserTime 0.07 ( 0.00%) 0.07 ( 2.72%) Clients 4 UserTime 0.08 ( 0.00%) 0.07 ( 3.29%) Clients 1 SysTime 0.70 ( 0.00%) 0.65 ( 6.65%) Clients 2 SysTime 0.85 ( 0.00%) 0.82 ( 3.65%) Clients 4 SysTime 1.41 ( 0.00%) 1.41 ( 0.32%) Clients 1 WallTime 0.77 ( 0.00%) 0.74 ( 4.19%) Clients 2 WallTime 0.47 ( 0.00%) 0.45 ( 3.73%) Clients 4 WallTime 0.38 ( 0.00%) 0.37 ( 1.58%) Clients 1 Flt/sec/cpu 497620.28 ( 0.00%) 520294.53 ( 4.56%) Clients 2 Flt/sec/cpu 414639.05 ( 0.00%) 429882.01 ( 3.68%) Clients 4 Flt/sec/cpu 257959.16 ( 0.00%) 258761.48 ( 0.31%) Clients 1 Flt/sec 495161.39 ( 0.00%) 517292.87 ( 4.47%) Clients 2 Flt/sec 820325.95 ( 0.00%) 850289.77 ( 3.65%) Clients 4 Flt/sec 1020068.93 ( 0.00%) 1022674.06 ( 0.26%) MMTests Statistics: duration Sys Time Running Test (seconds) 135.68 132.17 User+Sys Time Running Test (seconds) 164.2 160.13 Total Elapsed Time (seconds) 123.46 120.87 The overall improvement is small but the System CPU time is much improved and roughly in correlation to what oprofile reported (these performance figures are without profiling so skew is expected). The actual number of page faults is noticeably improved. For benchmarks like kernel builds, the overall benefit is marginal but the system CPU time is slightly reduced. To test the actual bug the commit fixed I opened two terminals. The first ran within a cpuset and continually ran a small program that faulted 100M of anonymous data. In a second window, the nodemask of the cpuset was continually randomised in a loop. Without the commit, the program would fail every so often (usually within 10 seconds) and obviously with the commit everything worked fine. With this patch applied, it also worked fine so the fix should be functionally equivalent. Signed-off-by: Mel Gorman <mgorman@suse.de> Cc: Miao Xie <miaox@cn.fujitsu.com> Cc: David Rientjes <rientjes@google.com> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Christoph Lameter <cl@linux.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>		2012-03-21 17:54:59 -07:00
..
acpi	Merge 3.3-rc2 into the driver-core-next branch.	2012-02-02 11:24:44 -08:00
asm-generic	mm: thp: fix pmd_bad() triggering in code paths holding mmap_sem read mode	2012-03-21 17:54:54 -07:00
crypto	crypto: remove the second argument of k[un]map_atomic()	2012-03-20 21:48:16 +08:00
drm	drm/exynos: exynos_drm.h header file fixes	2012-02-15 10:29:12 +09:00
keys	keys: add a "logon" key type	2012-01-17 22:39:40 -06:00
linux	cpuset: mm: reduce large amounts of memory barrier related damage v3	2012-03-21 17:54:59 -07:00
math-emu
media	Merge branch 'v4l_for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-media	2012-01-18 12:53:54 -08:00
misc
mtd	mtd: document that MEMWRITE ioctl is NAND-specific	2012-01-09 18:18:36 +00:00
net	Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next	2012-03-20 21:04:47 -07:00
pcmcia
rdma	Merge branches 'cma', 'cxgb3', 'cxgb4', 'ehca', 'iser', 'mad', 'nes', 'qib', 'srp' and 'srpt' into for-next	2012-03-19 09:50:33 -07:00
rxrpc
scsi	SCSI & usb-storage: add flags for VPD pages and REPORT LUNS	2012-02-08 17:36:41 -08:00
sound	Merge branch 'fix/asoc' into for-linus	2012-01-31 15:13:14 +01:00
target	target: Change target_submit_cmd() to return void	2012-02-07 06:41:04 +00:00
trace	Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial	2012-03-20 21:12:50 -07:00
video	fbdev fixes for 3.3	2012-02-07 15:54:02 -08:00
xen	Merge branch 'for-3.3/drivers' of git://git.kernel.dk/linux-block	2012-01-15 12:48:41 -08:00
Kbuild