linux/arch/x86/mm
Nadav Amit 3db6d5a5ec x86/mm/tlb: Remove 'struct flush_tlb_info' from the stack
Move flush_tlb_info variables off the stack. This allows to align
flush_tlb_info to cache-line and avoid potentially unnecessary cache
line movements. It also allows to have a fixed virtual-to-physical
translation of the variables, which reduces TLB misses.

Use per-CPU struct for flush_tlb_mm_range() and
flush_tlb_kernel_range(). Add debug assertions to ensure there are
no nested TLB flushes that might overwrite the per-CPU data. For
arch_tlbbatch_flush() use a const struct.

Results when running a microbenchmarks that performs 10^6 MADV_DONTEED
operations and touching a page, in which 3 additional threads run a
busy-wait loop (5 runs, PTI and retpolines are turned off):

			base		off-stack
			----		---------
  avg (usec/op)		1.629		1.570	(-3%)
  stddev		0.014		0.009

Signed-off-by: Nadav Amit <namit@vmware.com>
Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Dave Hansen <dave.hansen@intel.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/20190425230143.7008-1-namit@vmware.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2019-04-26 12:01:45 +02:00
..
amdtopology.c mm: remove include/linux/bootmem.h 2018-10-31 08:54:16 -07:00
cpu_entry_area.c x86/cpu_entry_area: Move percpu_setup_debug_store() to __init section 2019-02-08 14:32:33 +01:00
debug_pagetables.c x86/mm/dump_pagetables: Use DEFINE_SHOW_ATTRIBUTE() 2018-12-18 13:05:54 +01:00
dump_pagetables.c x86/mm: Prevent bogus warnings with "noexec=off" 2019-04-16 09:42:10 +02:00
extable.c x86-64: add warning for non-canonical user access address dereferences 2019-03-04 10:08:28 -08:00
fault.c x86/mm: Remove in_nmi() warning from 64-bit implementation of vmalloc_fault() 2019-04-24 12:21:35 +02:00
highmem_32.c mm: remove include/linux/bootmem.h 2018-10-31 08:54:16 -07:00
hugetlbpage.c x86/compat: Adjust in_compat_syscall() to generic code under !COMPAT 2018-11-01 12:59:25 +01:00
ident_map.c x86/mm: Stop pretending pgtable_l5_enabled is a variable 2018-05-19 11:56:57 +02:00
init_32.c mm, memory_hotplug: add nid parameter to arch_remove_memory 2018-12-28 12:11:49 -08:00
init_64.c mm, memory_hotplug: add nid parameter to arch_remove_memory 2018-12-28 12:11:49 -08:00
init.c mm: make free_reserved_area() return "const char *" 2018-12-28 12:11:48 -08:00
iomap_32.c x86/mm: Do not auto-massage page protections 2018-04-12 09:04:22 +02:00
ioremap.c x86/mm: Prevent bogus warnings with "noexec=off" 2019-04-16 09:42:10 +02:00
kasan_init_64.c memblock: drop memblock_alloc_*_nopanic() variants 2019-03-12 10:04:02 -07:00
kaslr.c Merge branch 'linus' into x86/mm, to pick up dependent fix 2019-04-26 12:01:33 +02:00
kmmio.c x86/mm/kmmio: Make the tracer robust against L1TF 2018-08-08 22:28:34 +02:00
Makefile x86/mm/sme: Disable stack protection for mem_encrypt_identity.c 2018-02-28 15:24:12 +01:00
mem_encrypt_boot.S x86/mm/sme, objtool: Annotate indirect call in sme_encrypt_execute() 2018-02-21 09:05:04 +01:00
mem_encrypt_identity.c x86/mm/mem_encrypt: Fix erroneous sizeof() 2019-01-15 11:41:58 +01:00
mem_encrypt.c dma-direct: merge swiotlb_dma_ops into the dma_direct code 2018-12-13 21:06:17 +01:00
mm_internal.h x86/mm/cpa: Optimize cpa_flush_array() TLB invalidation 2018-12-17 18:54:26 +01:00
mmap.c x86/mm: Don't exceed the valid physical address space 2019-03-28 14:13:51 +01:00
mmio-mod.c x86: do not use print_symbol() 2018-01-05 15:23:01 +01:00
mpx.c x86/mpx: tweak header name 2019-02-18 17:49:30 +01:00
numa_32.c mm: remove include/linux/bootmem.h 2018-10-31 08:54:16 -07:00
numa_64.c mm: remove include/linux/bootmem.h 2018-10-31 08:54:16 -07:00
numa_emulation.c Merge branch 'core/urgent' into x86/urgent, to pick up objtool fix 2018-11-03 23:42:16 +01:00
numa_internal.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
numa.c memblock: drop __memblock_alloc_base() 2019-03-12 10:04:01 -07:00
pageattr-test.c x86/mm/cpa: Simplify the code after making cpa->vaddr invariant 2018-12-17 18:54:25 +01:00
pageattr.c x86/mm: Remove unused variable 'old_pte' 2019-03-06 23:24:53 +01:00
pat_internal.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
pat_rbtree.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
pat.c x86/mm: Fix decoy address handling vs 32-bit builds 2018-12-11 18:28:20 -08:00
pf_in.c x86/mm: Audit and remove any unnecessary uses of module.h 2016-07-14 13:04:20 +02:00
pf_in.h
pgtable_32.c x86/mm: Rename flush_tlb_single() and flush_tlb_one() to __flush_tlb_one_[user|kernel]() 2018-02-15 01:15:52 +01:00
pgtable.c mm: treewide: remove unused address argument from pte_alloc functions 2019-01-04 13:13:47 -08:00
physaddr.c mm: remove include/linux/bootmem.h 2018-10-31 08:54:16 -07:00
physaddr.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
pkeys.c x86/pkeys: Make init_pkru_value static 2018-12-03 19:10:18 +01:00
pti.c x86/mm/pti: Make local symbols static 2019-03-22 13:31:28 +01:00
setup_nx.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
srat.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
testmmiotrace.c Annotate hardware config module parameters in arch/x86/mm/ 2017-04-04 16:54:21 +01:00
tlb.c x86/mm/tlb: Remove 'struct flush_tlb_info' from the stack 2019-04-26 12:01:45 +02:00