mirror of
https://github.com/torvalds/linux.git
synced 2024-12-29 14:21:47 +00:00
6fd166aae7
We can use PCID to retain the TLBs across CR3 switches; including those now part of the user/kernel switch. This increases performance of kernel entry/exit at the cost of more expensive/complicated TLB flushing. Now that we have two address spaces, one for kernel and one for user space, we need two PCIDs per mm. We use the top PCID bit to indicate a user PCID (just like we use the PFN LSB for the PGD). Since we do TLB invalidation from kernel space, the existing code will only invalidate the kernel PCID, we augment that by marking the corresponding user PCID invalid, and upon switching back to userspace, use a flushing CR3 write for the switch. In order to access the user_pcid_flush_mask we use PER_CPU storage, which means the previously established SWAPGS vs CR3 ordering is now mandatory and required. Having to do this memory access does require additional registers, most sites have a functioning stack and we can spill one (RAX), sites without functional stack need to otherwise provide the second scratch register. Note: PCID is generally available on Intel Sandybridge and later CPUs. Note: Up until this point TLB flushing was broken in this series. Based-on-code-from: Dave Hansen <dave.hansen@linux.intel.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Cc: Andy Lutomirski <luto@kernel.org> Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com> Cc: Borislav Petkov <bp@alien8.de> Cc: Brian Gerst <brgerst@gmail.com> Cc: Dave Hansen <dave.hansen@linux.intel.com> Cc: David Laight <David.Laight@aculab.com> Cc: Denys Vlasenko <dvlasenk@redhat.com> Cc: Eduardo Valentin <eduval@amazon.com> Cc: Greg KH <gregkh@linuxfoundation.org> Cc: H. Peter Anvin <hpa@zytor.com> Cc: Josh Poimboeuf <jpoimboe@redhat.com> Cc: Juergen Gross <jgross@suse.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Will Deacon <will.deacon@arm.com> Cc: aliguori@amazon.com Cc: daniel.gruss@iaik.tugraz.at Cc: hughd@google.com Cc: keescook@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
57 lines
1.7 KiB
C
57 lines
1.7 KiB
C
/* SPDX-License-Identifier: GPL-2.0 */
|
|
#ifndef _ASM_X86_PROCESSOR_FLAGS_H
|
|
#define _ASM_X86_PROCESSOR_FLAGS_H
|
|
|
|
#include <uapi/asm/processor-flags.h>
|
|
#include <linux/mem_encrypt.h>
|
|
|
|
#ifdef CONFIG_VM86
|
|
#define X86_VM_MASK X86_EFLAGS_VM
|
|
#else
|
|
#define X86_VM_MASK 0 /* No VM86 support */
|
|
#endif
|
|
|
|
/*
|
|
* CR3's layout varies depending on several things.
|
|
*
|
|
* If CR4.PCIDE is set (64-bit only), then CR3[11:0] is the address space ID.
|
|
* If PAE is enabled, then CR3[11:5] is part of the PDPT address
|
|
* (i.e. it's 32-byte aligned, not page-aligned) and CR3[4:0] is ignored.
|
|
* Otherwise (non-PAE, non-PCID), CR3[3] is PWT, CR3[4] is PCD, and
|
|
* CR3[2:0] and CR3[11:5] are ignored.
|
|
*
|
|
* In all cases, Linux puts zeros in the low ignored bits and in PWT and PCD.
|
|
*
|
|
* CR3[63] is always read as zero. If CR4.PCIDE is set, then CR3[63] may be
|
|
* written as 1 to prevent the write to CR3 from flushing the TLB.
|
|
*
|
|
* On systems with SME, one bit (in a variable position!) is stolen to indicate
|
|
* that the top-level paging structure is encrypted.
|
|
*
|
|
* All of the remaining bits indicate the physical address of the top-level
|
|
* paging structure.
|
|
*
|
|
* CR3_ADDR_MASK is the mask used by read_cr3_pa().
|
|
*/
|
|
#ifdef CONFIG_X86_64
|
|
/* Mask off the address space ID and SME encryption bits. */
|
|
#define CR3_ADDR_MASK __sme_clr(0x7FFFFFFFFFFFF000ull)
|
|
#define CR3_PCID_MASK 0xFFFull
|
|
#define CR3_NOFLUSH BIT_ULL(63)
|
|
|
|
#ifdef CONFIG_PAGE_TABLE_ISOLATION
|
|
# define X86_CR3_PTI_SWITCH_BIT 11
|
|
#endif
|
|
|
|
#else
|
|
/*
|
|
* CR3_ADDR_MASK needs at least bits 31:5 set on PAE systems, and we save
|
|
* a tiny bit of code size by setting all the bits.
|
|
*/
|
|
#define CR3_ADDR_MASK 0xFFFFFFFFull
|
|
#define CR3_PCID_MASK 0ull
|
|
#define CR3_NOFLUSH 0
|
|
#endif
|
|
|
|
#endif /* _ASM_X86_PROCESSOR_FLAGS_H */
|