linux/arch/x86/mm
Rik van Riel ac03158969 x86/mm/tlb: Make lazy TLB mode lazier
Lazy TLB mode can result in an idle CPU being woken up by a TLB flush,
when all it really needs to do is reload %CR3 at the next context switch,
assuming no page table pages got freed.

Memory ordering is used to prevent race conditions between switch_mm_irqs_off,
which checks whether .tlb_gen changed, and the TLB invalidation code, which
increments .tlb_gen whenever page table entries get invalidated.

The atomic increment in inc_mm_tlb_gen is its own barrier; the context
switch code adds an explicit barrier between reading tlbstate.is_lazy and
next->context.tlb_gen.

Unlike the 2016 version of this patch, CPUs with cpu_tlbstate.is_lazy set
are not removed from the mm_cpumask(mm), since that would prevent the TLB
flush IPIs at page table free time from being sent to all the CPUs
that need them.

This patch reduces total CPU use in the system by about 1-2% for a
memcache workload on two socket systems, and by about 1% for a heavily
multi-process netperf between two systems.

Tested-by: Song Liu <songliubraving@fb.com>
Signed-off-by: Rik van Riel <riel@surriel.com>
Acked-by: Dave Hansen <dave.hansen@intel.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: efault@gmx.de
Cc: kernel-team@fb.com
Cc: luto@kernel.org
Link: http://lkml.kernel.org/r/20180716190337.26133-5-riel@surriel.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-07-17 09:35:33 +02:00
..
amdtopology.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
cpu_entry_area.c x86/pti: Enable global pages for shared areas 2018-04-12 09:05:59 +02:00
debug_pagetables.c x86/efi: Replace efi_pgd with efi_mm.pgd 2018-03-12 11:05:05 +01:00
dump_pagetables.c x86/mm: Stop pretending pgtable_l5_enabled is a variable 2018-05-19 11:56:57 +02:00
extable.c x86/extable: Mark exception handler functions visible 2018-01-14 20:04:16 +01:00
fault.c x86/mm: Clean up the printk()s in show_fault_oops() 2018-06-27 14:08:11 +02:00
highmem_32.c x86/mm: Audit and remove any unnecessary uses of module.h 2016-07-14 13:04:20 +02:00
hugetlbpage.c x86/mm: Prevent non-MAP_FIXED mapping across DEFAULT_MAP_WINDOW border 2017-11-16 11:43:11 +01:00
ident_map.c x86/mm: Stop pretending pgtable_l5_enabled is a variable 2018-05-19 11:56:57 +02:00
init_32.c treewide: use PHYS_ADDR_MAX to avoid type casting ULLONG_MAX 2018-06-15 07:55:25 +09:00
init_64.c Merge branch 'linus' into x86/urgent 2018-06-22 21:20:35 +02:00
init.c mm: fix devmem_is_allowed() for sub-page System RAM intersections 2018-06-15 07:55:23 +09:00
iomap_32.c x86/mm: Do not auto-massage page protections 2018-04-12 09:04:22 +02:00
ioremap.c x86/mm: Do not auto-massage page protections 2018-04-12 09:04:22 +02:00
kasan_init_64.c x86/mm: Stop pretending pgtable_l5_enabled is a variable 2018-05-19 11:56:57 +02:00
kaslr.c x86/mm: Stop pretending pgtable_l5_enabled is a variable 2018-05-19 11:56:57 +02:00
kmmio.c x86/mm: Rename flush_tlb_single() and flush_tlb_one() to __flush_tlb_one_[user|kernel]() 2018-02-15 01:15:52 +01:00
Makefile x86/mm/sme: Disable stack protection for mem_encrypt_identity.c 2018-02-28 15:24:12 +01:00
mem_encrypt_boot.S x86/mm/sme, objtool: Annotate indirect call in sme_encrypt_execute() 2018-02-21 09:05:04 +01:00
mem_encrypt_identity.c x86/mm: Decouple dynamic __PHYSICAL_MASK from AMD SME 2018-06-06 13:38:01 +02:00
mem_encrypt.c dma/direct: Handle force decryption for DMA coherent buffers in common code 2018-03-20 10:01:59 +01:00
mm_internal.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
mmap.c exec: pass stack rlimit into mm layout functions 2018-04-11 10:28:37 -07:00
mmio-mod.c x86: do not use print_symbol() 2018-01-05 15:23:01 +01:00
mpx.c Merge branch 'x86/mpx' into x86/asm, to pick up dependent commits 2017-11-08 10:55:48 +01:00
numa_32.c x86/mm/32: Remove unused node_memmap_size_bytes() & CONFIG_NEED_NODE_MEMMAP_SIZE logic 2018-03-27 08:45:02 +02:00
numa_64.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
numa_emulation.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
numa_internal.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
numa.c x86: Remove pr_fmt duplicate logging prefixes 2018-05-13 21:25:18 +02:00
pageattr-test.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
pageattr.c x86/pti: Fix boot warning from Global-bit setting 2018-04-25 11:02:50 +02:00
pat_internal.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
pat_rbtree.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
pat.c x86/mm: add a function to check if a pfn is UC/UC-/WC 2018-01-16 16:40:09 +01:00
pf_in.c x86/mm: Audit and remove any unnecessary uses of module.h 2016-07-14 13:04:20 +02:00
pf_in.h
pgtable_32.c x86/mm: Rename flush_tlb_single() and flush_tlb_one() to __flush_tlb_one_[user|kernel]() 2018-02-15 01:15:52 +01:00
pgtable.c x86/mm: Add TLB purge to free pmd/pte page interfaces 2018-07-04 21:37:09 +02:00
physaddr.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
physaddr.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
pkeys.c x86/pkeys: Override pkey when moving away from PROT_EXEC 2018-05-14 11:14:45 +02:00
pti.c x86/pti: Disallow global kernel text with RANDSTRUCT 2018-04-25 11:02:51 +02:00
setup_nx.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
srat.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
testmmiotrace.c Annotate hardware config module parameters in arch/x86/mm/ 2017-04-04 16:54:21 +01:00
tlb.c x86/mm/tlb: Make lazy TLB mode lazier 2018-07-17 09:35:33 +02:00