License cleanup: add SPDX GPL-2.0 license identifier to files with no license
Many source files in the tree are missing licensing information, which
makes it harder for compliance tools to determine the correct license.
By default all files without license information are under the default
license of the kernel, which is GPL version 2.
Update the files which contain no license information with the 'GPL-2.0'
SPDX license identifier. The SPDX identifier is a legally binding
shorthand, which can be used instead of the full boiler plate text.
This patch is based on work done by Thomas Gleixner and Kate Stewart and
Philippe Ombredanne.
How this work was done:
Patches were generated and checked against linux-4.14-rc6 for a subset of
the use cases:
- file had no licensing information it it.
- file was a */uapi/* one with no licensing information in it,
- file was a */uapi/* one with existing licensing information,
Further patches will be generated in subsequent months to fix up cases
where non-standard license headers were used, and references to license
had to be inferred by heuristics based on keywords.
The analysis to determine which SPDX License Identifier to be applied to
a file was done in a spreadsheet of side by side results from of the
output of two independent scanners (ScanCode & Windriver) producing SPDX
tag:value files created by Philippe Ombredanne. Philippe prepared the
base worksheet, and did an initial spot review of a few 1000 files.
The 4.13 kernel was the starting point of the analysis with 60,537 files
assessed. Kate Stewart did a file by file comparison of the scanner
results in the spreadsheet to determine which SPDX license identifier(s)
to be applied to the file. She confirmed any determination that was not
immediately clear with lawyers working with the Linux Foundation.
Criteria used to select files for SPDX license identifier tagging was:
- Files considered eligible had to be source code files.
- Make and config files were included as candidates if they contained >5
lines of source
- File already had some variant of a license header in it (even if <5
lines).
All documentation files were explicitly excluded.
The following heuristics were used to determine which SPDX license
identifiers to apply.
- when both scanners couldn't find any license traces, file was
considered to have no license information in it, and the top level
COPYING file license applied.
For non */uapi/* files that summary was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 11139
and resulted in the first patch in this series.
If that file was a */uapi/* path one, it was "GPL-2.0 WITH
Linux-syscall-note" otherwise it was "GPL-2.0". Results of that was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 WITH Linux-syscall-note 930
and resulted in the second patch in this series.
- if a file had some form of licensing information in it, and was one
of the */uapi/* ones, it was denoted with the Linux-syscall-note if
any GPL family license was found in the file or had no licensing in
it (per prior point). Results summary:
SPDX license identifier # files
---------------------------------------------------|------
GPL-2.0 WITH Linux-syscall-note 270
GPL-2.0+ WITH Linux-syscall-note 169
((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) 21
((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) 17
LGPL-2.1+ WITH Linux-syscall-note 15
GPL-1.0+ WITH Linux-syscall-note 14
((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) 5
LGPL-2.0+ WITH Linux-syscall-note 4
LGPL-2.1 WITH Linux-syscall-note 3
((GPL-2.0 WITH Linux-syscall-note) OR MIT) 3
((GPL-2.0 WITH Linux-syscall-note) AND MIT) 1
and that resulted in the third patch in this series.
- when the two scanners agreed on the detected license(s), that became
the concluded license(s).
- when there was disagreement between the two scanners (one detected a
license but the other didn't, or they both detected different
licenses) a manual inspection of the file occurred.
- In most cases a manual inspection of the information in the file
resulted in a clear resolution of the license that should apply (and
which scanner probably needed to revisit its heuristics).
- When it was not immediately clear, the license identifier was
confirmed with lawyers working with the Linux Foundation.
- If there was any question as to the appropriate license identifier,
the file was flagged for further research and to be revisited later
in time.
In total, over 70 hours of logged manual review was done on the
spreadsheet to determine the SPDX license identifiers to apply to the
source files by Kate, Philippe, Thomas and, in some cases, confirmation
by lawyers working with the Linux Foundation.
Kate also obtained a third independent scan of the 4.13 code base from
FOSSology, and compared selected files where the other two scanners
disagreed against that SPDX file, to see if there was new insights. The
Windriver scanner is based on an older version of FOSSology in part, so
they are related.
Thomas did random spot checks in about 500 files from the spreadsheets
for the uapi headers and agreed with SPDX license identifier in the
files he inspected. For the non-uapi files Thomas did random spot checks
in about 15000 files.
In initial set of patches against 4.14-rc6, 3 files were found to have
copy/paste license identifier errors, and have been fixed to reflect the
correct identifier.
Additionally Philippe spent 10 hours this week doing a detailed manual
inspection and review of the 12,461 patched files from the initial patch
version early this week with:
- a full scancode scan run, collecting the matched texts, detected
license ids and scores
- reviewing anything where there was a license detected (about 500+
files) to ensure that the applied SPDX license was correct
- reviewing anything where there was no detection but the patch license
was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied
SPDX license was correct
This produced a worksheet with 20 files needing minor correction. This
worksheet was then exported into 3 different .csv files for the
different types of files to be modified.
These .csv files were then reviewed by Greg. Thomas wrote a script to
parse the csv files and add the proper SPDX tag to the file, in the
format that the file expected. This script was further refined by Greg
based on the output to detect more types of files automatically and to
distinguish between header and source .c files (which need different
comment types.) Finally Greg ran the script using the .csv files to
generate the patches.
Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Philippe Ombredanne <pombredanne@nexb.com>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2017-11-01 14:07:57 +00:00
|
|
|
/* SPDX-License-Identifier: GPL-2.0 */
|
2005-04-16 22:20:36 +00:00
|
|
|
/*
|
2015-06-08 07:49:11 +00:00
|
|
|
* Copyright (C) 1991,1992 Linus Torvalds
|
2005-04-16 22:20:36 +00:00
|
|
|
*
|
2015-06-08 07:49:11 +00:00
|
|
|
* entry_32.S contains the system-call and low-level fault and trap handling routines.
|
2005-04-16 22:20:36 +00:00
|
|
|
*
|
2015-10-06 00:48:13 +00:00
|
|
|
* Stack layout while running C code:
|
2015-06-08 07:49:11 +00:00
|
|
|
* ptrace needs to have all registers on the stack.
|
|
|
|
* If the order here is changed, it needs to be
|
|
|
|
* updated in fork.c:copy_process(), signal.c:do_signal(),
|
2005-04-16 22:20:36 +00:00
|
|
|
* ptrace.c and ptrace.h
|
|
|
|
*
|
|
|
|
* 0(%esp) - %ebx
|
|
|
|
* 4(%esp) - %ecx
|
|
|
|
* 8(%esp) - %edx
|
2015-06-08 20:35:33 +00:00
|
|
|
* C(%esp) - %esi
|
2005-04-16 22:20:36 +00:00
|
|
|
* 10(%esp) - %edi
|
|
|
|
* 14(%esp) - %ebp
|
|
|
|
* 18(%esp) - %eax
|
|
|
|
* 1C(%esp) - %ds
|
|
|
|
* 20(%esp) - %es
|
2007-02-13 12:26:20 +00:00
|
|
|
* 24(%esp) - %fs
|
2009-02-09 13:17:40 +00:00
|
|
|
* 28(%esp) - %gs saved iff !CONFIG_X86_32_LAZY_GS
|
|
|
|
* 2C(%esp) - orig_eax
|
|
|
|
* 30(%esp) - %eip
|
|
|
|
* 34(%esp) - %cs
|
|
|
|
* 38(%esp) - %eflags
|
|
|
|
* 3C(%esp) - %oldesp
|
|
|
|
* 40(%esp) - %oldss
|
2005-04-16 22:20:36 +00:00
|
|
|
*/
|
|
|
|
|
|
|
|
#include <linux/linkage.h>
|
2012-01-03 19:23:06 +00:00
|
|
|
#include <linux/err.h>
|
2005-04-16 22:20:36 +00:00
|
|
|
#include <asm/thread_info.h>
|
2006-07-03 07:24:43 +00:00
|
|
|
#include <asm/irqflags.h>
|
2005-04-16 22:20:36 +00:00
|
|
|
#include <asm/errno.h>
|
|
|
|
#include <asm/segment.h>
|
|
|
|
#include <asm/smp.h>
|
2006-12-07 01:14:01 +00:00
|
|
|
#include <asm/percpu.h>
|
2008-03-25 19:16:32 +00:00
|
|
|
#include <asm/processor-flags.h>
|
2008-05-02 18:10:09 +00:00
|
|
|
#include <asm/irq_vectors.h>
|
2016-01-26 21:12:04 +00:00
|
|
|
#include <asm/cpufeatures.h>
|
2011-08-25 20:10:33 +00:00
|
|
|
#include <asm/alternative-asm.h>
|
2012-04-20 19:19:50 +00:00
|
|
|
#include <asm/asm.h>
|
2012-09-21 20:58:10 +00:00
|
|
|
#include <asm/smap.h>
|
2016-09-21 21:04:01 +00:00
|
|
|
#include <asm/frame.h>
|
2018-01-11 21:46:28 +00:00
|
|
|
#include <asm/nospec-branch.h>
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2011-03-07 18:10:39 +00:00
|
|
|
.section .entry.text, "ax"
|
|
|
|
|
2006-12-07 01:14:08 +00:00
|
|
|
/*
|
|
|
|
* We use macros for low-level operations which need to be overridden
|
|
|
|
* for paravirtualization. The following will never clobber any registers:
|
|
|
|
* INTERRUPT_RETURN (aka. "iret")
|
|
|
|
* GET_CR0_INTO_EAX (aka. "movl %cr0, %eax")
|
2008-06-25 04:19:26 +00:00
|
|
|
* ENABLE_INTERRUPTS_SYSEXIT (aka "sti; sysexit").
|
2006-12-07 01:14:08 +00:00
|
|
|
*
|
|
|
|
* For DISABLE_INTERRUPTS/ENABLE_INTERRUPTS (aka "cli"/"sti"), you must
|
|
|
|
* specify what registers can be overwritten (CLBR_NONE, CLBR_EAX/EDX/ECX/ANY).
|
|
|
|
* Allowing a register to be clobbered can shrink the paravirt replacement
|
|
|
|
* enough to patch inline, increasing performance.
|
|
|
|
*/
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
#ifdef CONFIG_PREEMPT
|
2015-06-08 07:49:11 +00:00
|
|
|
# define preempt_stop(clobbers) DISABLE_INTERRUPTS(clobbers); TRACE_IRQS_OFF
|
2005-04-16 22:20:36 +00:00
|
|
|
#else
|
2015-06-08 07:49:11 +00:00
|
|
|
# define preempt_stop(clobbers)
|
|
|
|
# define resume_kernel restore_all
|
2005-04-16 22:20:36 +00:00
|
|
|
#endif
|
|
|
|
|
2006-07-03 07:24:43 +00:00
|
|
|
.macro TRACE_IRQS_IRET
|
|
|
|
#ifdef CONFIG_TRACE_IRQFLAGS
|
2015-06-08 07:49:11 +00:00
|
|
|
testl $X86_EFLAGS_IF, PT_EFLAGS(%esp) # interrupts off?
|
|
|
|
jz 1f
|
2006-07-03 07:24:43 +00:00
|
|
|
TRACE_IRQS_ON
|
|
|
|
1:
|
|
|
|
#endif
|
|
|
|
.endm
|
|
|
|
|
2009-02-09 13:17:40 +00:00
|
|
|
/*
|
|
|
|
* User gs save/restore
|
|
|
|
*
|
|
|
|
* %gs is used for userland TLS and kernel only uses it for stack
|
|
|
|
* canary which is required to be at %gs:20 by gcc. Read the comment
|
|
|
|
* at the top of stackprotector.h for more info.
|
|
|
|
*
|
|
|
|
* Local labels 98 and 99 are used.
|
|
|
|
*/
|
|
|
|
#ifdef CONFIG_X86_32_LAZY_GS
|
|
|
|
|
|
|
|
/* unfortunately push/pop can't be no-op */
|
|
|
|
.macro PUSH_GS
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl $0
|
2009-02-09 13:17:40 +00:00
|
|
|
.endm
|
|
|
|
.macro POP_GS pop=0
|
2015-06-08 07:49:11 +00:00
|
|
|
addl $(4 + \pop), %esp
|
2009-02-09 13:17:40 +00:00
|
|
|
.endm
|
|
|
|
.macro POP_GS_EX
|
|
|
|
.endm
|
|
|
|
|
|
|
|
/* all the rest are no-op */
|
|
|
|
.macro PTGS_TO_GS
|
|
|
|
.endm
|
|
|
|
.macro PTGS_TO_GS_EX
|
|
|
|
.endm
|
|
|
|
.macro GS_TO_REG reg
|
|
|
|
.endm
|
|
|
|
.macro REG_TO_PTGS reg
|
|
|
|
.endm
|
|
|
|
.macro SET_KERNEL_GS reg
|
|
|
|
.endm
|
|
|
|
|
|
|
|
#else /* CONFIG_X86_32_LAZY_GS */
|
|
|
|
|
|
|
|
.macro PUSH_GS
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl %gs
|
2009-02-09 13:17:40 +00:00
|
|
|
.endm
|
|
|
|
|
|
|
|
.macro POP_GS pop=0
|
2015-06-08 07:49:11 +00:00
|
|
|
98: popl %gs
|
2009-02-09 13:17:40 +00:00
|
|
|
.if \pop <> 0
|
2015-06-08 20:35:33 +00:00
|
|
|
add $\pop, %esp
|
2009-02-09 13:17:40 +00:00
|
|
|
.endif
|
|
|
|
.endm
|
|
|
|
.macro POP_GS_EX
|
|
|
|
.pushsection .fixup, "ax"
|
2015-06-08 07:49:11 +00:00
|
|
|
99: movl $0, (%esp)
|
|
|
|
jmp 98b
|
2009-02-09 13:17:40 +00:00
|
|
|
.popsection
|
2015-06-08 07:49:11 +00:00
|
|
|
_ASM_EXTABLE(98b, 99b)
|
2009-02-09 13:17:40 +00:00
|
|
|
.endm
|
|
|
|
|
|
|
|
.macro PTGS_TO_GS
|
2015-06-08 07:49:11 +00:00
|
|
|
98: mov PT_GS(%esp), %gs
|
2009-02-09 13:17:40 +00:00
|
|
|
.endm
|
|
|
|
.macro PTGS_TO_GS_EX
|
|
|
|
.pushsection .fixup, "ax"
|
2015-06-08 07:49:11 +00:00
|
|
|
99: movl $0, PT_GS(%esp)
|
|
|
|
jmp 98b
|
2009-02-09 13:17:40 +00:00
|
|
|
.popsection
|
2015-06-08 07:49:11 +00:00
|
|
|
_ASM_EXTABLE(98b, 99b)
|
2009-02-09 13:17:40 +00:00
|
|
|
.endm
|
|
|
|
|
|
|
|
.macro GS_TO_REG reg
|
2015-06-08 07:49:11 +00:00
|
|
|
movl %gs, \reg
|
2009-02-09 13:17:40 +00:00
|
|
|
.endm
|
|
|
|
.macro REG_TO_PTGS reg
|
2015-06-08 07:49:11 +00:00
|
|
|
movl \reg, PT_GS(%esp)
|
2009-02-09 13:17:40 +00:00
|
|
|
.endm
|
|
|
|
.macro SET_KERNEL_GS reg
|
2015-06-08 07:49:11 +00:00
|
|
|
movl $(__KERNEL_STACK_CANARY), \reg
|
|
|
|
movl \reg, %gs
|
2009-02-09 13:17:40 +00:00
|
|
|
.endm
|
|
|
|
|
2015-06-08 07:49:11 +00:00
|
|
|
#endif /* CONFIG_X86_32_LAZY_GS */
|
2009-02-09 13:17:40 +00:00
|
|
|
|
2015-10-06 00:48:14 +00:00
|
|
|
.macro SAVE_ALL pt_regs_ax=%eax
|
2009-02-09 13:17:40 +00:00
|
|
|
cld
|
2009-02-09 13:17:40 +00:00
|
|
|
PUSH_GS
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl %fs
|
|
|
|
pushl %es
|
|
|
|
pushl %ds
|
2015-10-06 00:48:14 +00:00
|
|
|
pushl \pt_regs_ax
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl %ebp
|
|
|
|
pushl %edi
|
|
|
|
pushl %esi
|
|
|
|
pushl %edx
|
|
|
|
pushl %ecx
|
|
|
|
pushl %ebx
|
|
|
|
movl $(__USER_DS), %edx
|
|
|
|
movl %edx, %ds
|
|
|
|
movl %edx, %es
|
|
|
|
movl $(__KERNEL_PERCPU), %edx
|
|
|
|
movl %edx, %fs
|
2009-02-09 13:17:40 +00:00
|
|
|
SET_KERNEL_GS %edx
|
2009-02-09 13:17:40 +00:00
|
|
|
.endm
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2016-10-20 16:34:40 +00:00
|
|
|
/*
|
|
|
|
* This is a sneaky trick to help the unwinder find pt_regs on the stack. The
|
|
|
|
* frame pointer is replaced with an encoded pointer to pt_regs. The encoding
|
x86/unwind: Use MSB for frame pointer encoding on 32-bit
On x86-32, Tetsuo Handa and Fengguang Wu reported unwinder warnings
like:
WARNING: kernel stack regs at f60bb9c8 in swapper:1 has bad 'bp' value 0ba00000
And also there were some stack dumps with a bunch of unreliable '?'
symbols after an apic_timer_interrupt symbol, meaning the unwinder got
confused when it tried to read the regs.
The cause of those issues is that, with GCC 4.8 (and possibly older),
there are cases where GCC misaligns the stack pointer in a leaf function
for no apparent reason:
c124a388 <acpi_rs_move_data>:
c124a388: 55 push %ebp
c124a389: 89 e5 mov %esp,%ebp
c124a38b: 57 push %edi
c124a38c: 56 push %esi
c124a38d: 89 d6 mov %edx,%esi
c124a38f: 53 push %ebx
c124a390: 31 db xor %ebx,%ebx
c124a392: 83 ec 03 sub $0x3,%esp
...
c124a3e3: 83 c4 03 add $0x3,%esp
c124a3e6: 5b pop %ebx
c124a3e7: 5e pop %esi
c124a3e8: 5f pop %edi
c124a3e9: 5d pop %ebp
c124a3ea: c3 ret
If an interrupt occurs in such a function, the regs on the stack will be
unaligned, which breaks the frame pointer encoding assumption. So on
32-bit, use the MSB instead of the LSB to encode the regs.
This isn't an issue on 64-bit, because interrupts align the stack before
writing to it.
Reported-and-tested-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Reported-and-tested-by: Fengguang Wu <fengguang.wu@intel.com>
Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Byungchul Park <byungchul.park@lge.com>
Cc: LKP <lkp@01.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/279a26996a482ca716605c7dbc7f2db9d8d91e81.1507597785.git.jpoimboe@redhat.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-10-10 01:20:03 +00:00
|
|
|
* is just clearing the MSB, which makes it an invalid stack address and is also
|
2016-10-20 16:34:40 +00:00
|
|
|
* a signal to the unwinder that it's a pt_regs pointer in disguise.
|
|
|
|
*
|
|
|
|
* NOTE: This macro must be used *after* SAVE_ALL because it corrupts the
|
|
|
|
* original rbp.
|
|
|
|
*/
|
|
|
|
.macro ENCODE_FRAME_POINTER
|
|
|
|
#ifdef CONFIG_FRAME_POINTER
|
|
|
|
mov %esp, %ebp
|
x86/unwind: Use MSB for frame pointer encoding on 32-bit
On x86-32, Tetsuo Handa and Fengguang Wu reported unwinder warnings
like:
WARNING: kernel stack regs at f60bb9c8 in swapper:1 has bad 'bp' value 0ba00000
And also there were some stack dumps with a bunch of unreliable '?'
symbols after an apic_timer_interrupt symbol, meaning the unwinder got
confused when it tried to read the regs.
The cause of those issues is that, with GCC 4.8 (and possibly older),
there are cases where GCC misaligns the stack pointer in a leaf function
for no apparent reason:
c124a388 <acpi_rs_move_data>:
c124a388: 55 push %ebp
c124a389: 89 e5 mov %esp,%ebp
c124a38b: 57 push %edi
c124a38c: 56 push %esi
c124a38d: 89 d6 mov %edx,%esi
c124a38f: 53 push %ebx
c124a390: 31 db xor %ebx,%ebx
c124a392: 83 ec 03 sub $0x3,%esp
...
c124a3e3: 83 c4 03 add $0x3,%esp
c124a3e6: 5b pop %ebx
c124a3e7: 5e pop %esi
c124a3e8: 5f pop %edi
c124a3e9: 5d pop %ebp
c124a3ea: c3 ret
If an interrupt occurs in such a function, the regs on the stack will be
unaligned, which breaks the frame pointer encoding assumption. So on
32-bit, use the MSB instead of the LSB to encode the regs.
This isn't an issue on 64-bit, because interrupts align the stack before
writing to it.
Reported-and-tested-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Reported-and-tested-by: Fengguang Wu <fengguang.wu@intel.com>
Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Byungchul Park <byungchul.park@lge.com>
Cc: LKP <lkp@01.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/279a26996a482ca716605c7dbc7f2db9d8d91e81.1507597785.git.jpoimboe@redhat.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-10-10 01:20:03 +00:00
|
|
|
andl $0x7fffffff, %ebp
|
2016-10-20 16:34:40 +00:00
|
|
|
#endif
|
|
|
|
.endm
|
|
|
|
|
2009-02-09 13:17:40 +00:00
|
|
|
.macro RESTORE_INT_REGS
|
2015-06-08 07:49:11 +00:00
|
|
|
popl %ebx
|
|
|
|
popl %ecx
|
|
|
|
popl %edx
|
|
|
|
popl %esi
|
|
|
|
popl %edi
|
|
|
|
popl %ebp
|
|
|
|
popl %eax
|
2009-02-09 13:17:40 +00:00
|
|
|
.endm
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2009-02-09 13:17:40 +00:00
|
|
|
.macro RESTORE_REGS pop=0
|
2009-02-09 13:17:40 +00:00
|
|
|
RESTORE_INT_REGS
|
2015-06-08 07:49:11 +00:00
|
|
|
1: popl %ds
|
|
|
|
2: popl %es
|
|
|
|
3: popl %fs
|
2009-02-09 13:17:40 +00:00
|
|
|
POP_GS \pop
|
2009-02-09 13:17:40 +00:00
|
|
|
.pushsection .fixup, "ax"
|
2015-06-08 07:49:11 +00:00
|
|
|
4: movl $0, (%esp)
|
|
|
|
jmp 1b
|
|
|
|
5: movl $0, (%esp)
|
|
|
|
jmp 2b
|
|
|
|
6: movl $0, (%esp)
|
|
|
|
jmp 3b
|
[PATCH] i386: Use %gs as the PDA base-segment in the kernel
This patch is the meat of the PDA change. This patch makes several related
changes:
1: Most significantly, %gs is now used in the kernel. This means that on
entry, the old value of %gs is saved away, and it is reloaded with
__KERNEL_PDA.
2: entry.S constructs the stack in the shape of struct pt_regs, and this
is passed around the kernel so that the process's saved register
state can be accessed.
Unfortunately struct pt_regs doesn't currently have space for %gs
(or %fs). This patch extends pt_regs to add space for gs (no space
is allocated for %fs, since it won't be used, and it would just
complicate the code in entry.S to work around the space).
3: Because %gs is now saved on the stack like %ds, %es and the integer
registers, there are a number of places where it no longer needs to
be handled specially; namely context switch, and saving/restoring the
register state in a signal context.
4: And since kernel threads run in kernel space and call normal kernel
code, they need to be created with their %gs == __KERNEL_PDA.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Andi Kleen <ak@suse.de>
Cc: Chuck Ebbert <76306.1226@compuserve.com>
Cc: Zachary Amsden <zach@vmware.com>
Cc: Jan Beulich <jbeulich@novell.com>
Cc: Andi Kleen <ak@suse.de>
Signed-off-by: Andrew Morton <akpm@osdl.org>
2006-12-07 01:14:02 +00:00
|
|
|
.popsection
|
2015-06-08 07:49:11 +00:00
|
|
|
_ASM_EXTABLE(1b, 4b)
|
|
|
|
_ASM_EXTABLE(2b, 5b)
|
|
|
|
_ASM_EXTABLE(3b, 6b)
|
2009-02-09 13:17:40 +00:00
|
|
|
POP_GS_EX
|
2009-02-09 13:17:40 +00:00
|
|
|
.endm
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2016-08-13 16:38:19 +00:00
|
|
|
/*
|
|
|
|
* %eax: prev task
|
|
|
|
* %edx: next task
|
|
|
|
*/
|
|
|
|
ENTRY(__switch_to_asm)
|
|
|
|
/*
|
|
|
|
* Save callee-saved registers
|
|
|
|
* This must match the order in struct inactive_task_frame
|
|
|
|
*/
|
|
|
|
pushl %ebp
|
|
|
|
pushl %ebx
|
|
|
|
pushl %edi
|
|
|
|
pushl %esi
|
|
|
|
|
|
|
|
/* switch stack */
|
|
|
|
movl %esp, TASK_threadsp(%eax)
|
|
|
|
movl TASK_threadsp(%edx), %esp
|
|
|
|
|
|
|
|
#ifdef CONFIG_CC_STACKPROTECTOR
|
|
|
|
movl TASK_stack_canary(%edx), %ebx
|
|
|
|
movl %ebx, PER_CPU_VAR(stack_canary)+stack_canary_offset
|
|
|
|
#endif
|
|
|
|
|
2018-01-12 17:49:25 +00:00
|
|
|
#ifdef CONFIG_RETPOLINE
|
|
|
|
/*
|
|
|
|
* When switching from a shallower to a deeper call stack
|
|
|
|
* the RSB may either underflow or use entries populated
|
|
|
|
* with userspace addresses. On CPUs where those concerns
|
|
|
|
* exist, overwrite the RSB with entries which capture
|
|
|
|
* speculative execution to prevent attack.
|
|
|
|
*/
|
2018-01-27 16:24:33 +00:00
|
|
|
/* Clobbers %ebx */
|
|
|
|
FILL_RETURN_BUFFER RSB_CLEAR_LOOPS, X86_FEATURE_RSB_CTXSW
|
2018-01-12 17:49:25 +00:00
|
|
|
#endif
|
|
|
|
|
2016-08-13 16:38:19 +00:00
|
|
|
/* restore callee-saved registers */
|
|
|
|
popl %esi
|
|
|
|
popl %edi
|
|
|
|
popl %ebx
|
|
|
|
popl %ebp
|
|
|
|
|
|
|
|
jmp __switch_to
|
|
|
|
END(__switch_to_asm)
|
|
|
|
|
2017-05-23 15:37:29 +00:00
|
|
|
/*
|
|
|
|
* The unwinder expects the last frame on the stack to always be at the same
|
|
|
|
* offset from the end of the page, which allows it to validate the stack.
|
|
|
|
* Calling schedule_tail() directly would break that convention because its an
|
|
|
|
* asmlinkage function so its argument has to be pushed on the stack. This
|
|
|
|
* wrapper creates a proper "end of stack" frame header before the call.
|
|
|
|
*/
|
|
|
|
ENTRY(schedule_tail_wrapper)
|
|
|
|
FRAME_BEGIN
|
|
|
|
|
|
|
|
pushl %eax
|
|
|
|
call schedule_tail
|
|
|
|
popl %eax
|
|
|
|
|
|
|
|
FRAME_END
|
|
|
|
ret
|
|
|
|
ENDPROC(schedule_tail_wrapper)
|
2016-08-13 16:38:19 +00:00
|
|
|
/*
|
|
|
|
* A newly forked process directly context switches into this address.
|
|
|
|
*
|
|
|
|
* eax: prev task we switched from
|
2016-08-13 16:38:20 +00:00
|
|
|
* ebx: kernel thread func (NULL for user thread)
|
|
|
|
* edi: kernel thread arg
|
2016-08-13 16:38:19 +00:00
|
|
|
*/
|
2005-04-16 22:20:36 +00:00
|
|
|
ENTRY(ret_from_fork)
|
2017-05-23 15:37:29 +00:00
|
|
|
call schedule_tail_wrapper
|
2015-10-06 00:48:13 +00:00
|
|
|
|
2016-08-13 16:38:20 +00:00
|
|
|
testl %ebx, %ebx
|
|
|
|
jnz 1f /* kernel threads are uncommon */
|
|
|
|
|
|
|
|
2:
|
2015-10-06 00:48:13 +00:00
|
|
|
/* When we fork, we trace the syscall return in the child, too. */
|
2017-05-23 15:37:29 +00:00
|
|
|
movl %esp, %eax
|
2015-10-06 00:48:13 +00:00
|
|
|
call syscall_return_slowpath
|
|
|
|
jmp restore_all
|
|
|
|
|
2016-08-13 16:38:20 +00:00
|
|
|
/* kernel thread */
|
|
|
|
1: movl %edi, %eax
|
2018-01-11 21:46:28 +00:00
|
|
|
CALL_NOSPEC %ebx
|
2015-10-06 00:48:13 +00:00
|
|
|
/*
|
2016-08-13 16:38:20 +00:00
|
|
|
* A kernel thread is allowed to return here after successfully
|
|
|
|
* calling do_execve(). Exit to userspace to complete the execve()
|
|
|
|
* syscall.
|
2015-10-06 00:48:13 +00:00
|
|
|
*/
|
2016-08-13 16:38:20 +00:00
|
|
|
movl $0, PT_EAX(%esp)
|
|
|
|
jmp 2b
|
|
|
|
END(ret_from_fork)
|
2012-08-02 19:05:11 +00:00
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/*
|
|
|
|
* Return to user mode is not as complex as all this looks,
|
|
|
|
* but we want the default path for a system call return to
|
|
|
|
* go as quickly as possible which is why some of this is
|
|
|
|
* less clear than it otherwise should be.
|
|
|
|
*/
|
|
|
|
|
|
|
|
# userspace resumption stub bypassing syscall exit tracing
|
|
|
|
ALIGN
|
|
|
|
ret_from_exception:
|
2006-12-07 01:14:08 +00:00
|
|
|
preempt_stop(CLBR_ANY)
|
2005-04-16 22:20:36 +00:00
|
|
|
ret_from_intr:
|
2012-03-22 20:39:25 +00:00
|
|
|
#ifdef CONFIG_VM86
|
2015-06-08 07:49:11 +00:00
|
|
|
movl PT_EFLAGS(%esp), %eax # mix EFLAGS and CS
|
|
|
|
movb PT_CS(%esp), %al
|
|
|
|
andl $(X86_EFLAGS_VM | SEGMENT_RPL_MASK), %eax
|
2012-03-22 20:39:25 +00:00
|
|
|
#else
|
|
|
|
/*
|
2012-08-02 19:05:11 +00:00
|
|
|
* We can be coming here from child spawned by kernel_thread().
|
2012-03-22 20:39:25 +00:00
|
|
|
*/
|
2015-06-08 07:49:11 +00:00
|
|
|
movl PT_CS(%esp), %eax
|
|
|
|
andl $SEGMENT_RPL_MASK, %eax
|
2012-03-22 20:39:25 +00:00
|
|
|
#endif
|
2015-06-08 07:49:11 +00:00
|
|
|
cmpl $USER_RPL, %eax
|
|
|
|
jb resume_kernel # not returning to v8086 or userspace
|
[PATCH] i386: Use %gs as the PDA base-segment in the kernel
This patch is the meat of the PDA change. This patch makes several related
changes:
1: Most significantly, %gs is now used in the kernel. This means that on
entry, the old value of %gs is saved away, and it is reloaded with
__KERNEL_PDA.
2: entry.S constructs the stack in the shape of struct pt_regs, and this
is passed around the kernel so that the process's saved register
state can be accessed.
Unfortunately struct pt_regs doesn't currently have space for %gs
(or %fs). This patch extends pt_regs to add space for gs (no space
is allocated for %fs, since it won't be used, and it would just
complicate the code in entry.S to work around the space).
3: Because %gs is now saved on the stack like %ds, %es and the integer
registers, there are a number of places where it no longer needs to
be handled specially; namely context switch, and saving/restoring the
register state in a signal context.
4: And since kernel threads run in kernel space and call normal kernel
code, they need to be created with their %gs == __KERNEL_PDA.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Andi Kleen <ak@suse.de>
Cc: Chuck Ebbert <76306.1226@compuserve.com>
Cc: Zachary Amsden <zach@vmware.com>
Cc: Jan Beulich <jbeulich@novell.com>
Cc: Andi Kleen <ak@suse.de>
Signed-off-by: Andrew Morton <akpm@osdl.org>
2006-12-07 01:14:02 +00:00
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
ENTRY(resume_userspace)
|
2015-07-31 21:41:09 +00:00
|
|
|
DISABLE_INTERRUPTS(CLBR_ANY)
|
2008-06-06 08:14:08 +00:00
|
|
|
TRACE_IRQS_OFF
|
2015-07-31 21:41:09 +00:00
|
|
|
movl %esp, %eax
|
|
|
|
call prepare_exit_to_usermode
|
2015-06-08 07:49:11 +00:00
|
|
|
jmp restore_all
|
2007-02-13 12:26:24 +00:00
|
|
|
END(ret_from_exception)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
#ifdef CONFIG_PREEMPT
|
|
|
|
ENTRY(resume_kernel)
|
2006-12-07 01:14:08 +00:00
|
|
|
DISABLE_INTERRUPTS(CLBR_ANY)
|
2016-09-21 21:03:59 +00:00
|
|
|
.Lneed_resched:
|
2015-06-08 07:49:11 +00:00
|
|
|
cmpl $0, PER_CPU_VAR(__preempt_count)
|
|
|
|
jnz restore_all
|
|
|
|
testl $X86_EFLAGS_IF, PT_EFLAGS(%esp) # interrupts off (exception path) ?
|
|
|
|
jz restore_all
|
|
|
|
call preempt_schedule_irq
|
2016-09-21 21:03:59 +00:00
|
|
|
jmp .Lneed_resched
|
2007-02-13 12:26:24 +00:00
|
|
|
END(resume_kernel)
|
2005-04-16 22:20:36 +00:00
|
|
|
#endif
|
|
|
|
|
2016-03-10 03:00:30 +00:00
|
|
|
GLOBAL(__begin_SYSENTER_singlestep_region)
|
|
|
|
/*
|
|
|
|
* All code from here through __end_SYSENTER_singlestep_region is subject
|
|
|
|
* to being single-stepped if a user program sets TF and executes SYSENTER.
|
|
|
|
* There is absolutely nothing that we can do to prevent this from happening
|
|
|
|
* (thanks Intel!). To keep our handling of this situation as simple as
|
|
|
|
* possible, we handle TF just like AC and NT, except that our #DB handler
|
|
|
|
* will ignore all of the single-step traps generated in this range.
|
|
|
|
*/
|
|
|
|
|
|
|
|
#ifdef CONFIG_XEN
|
|
|
|
/*
|
|
|
|
* Xen doesn't set %esp to be precisely what the normal SYSENTER
|
|
|
|
* entry point expects, so fix it up before using the normal path.
|
|
|
|
*/
|
|
|
|
ENTRY(xen_sysenter_target)
|
|
|
|
addl $5*4, %esp /* remove xen-provided frame */
|
2016-09-21 21:03:59 +00:00
|
|
|
jmp .Lsysenter_past_esp
|
2016-03-10 03:00:30 +00:00
|
|
|
#endif
|
|
|
|
|
2016-03-10 03:00:35 +00:00
|
|
|
/*
|
|
|
|
* 32-bit SYSENTER entry.
|
|
|
|
*
|
|
|
|
* 32-bit system calls through the vDSO's __kernel_vsyscall enter here
|
|
|
|
* if X86_FEATURE_SEP is available. This is the preferred system call
|
|
|
|
* entry on 32-bit systems.
|
|
|
|
*
|
|
|
|
* The SYSENTER instruction, in principle, should *only* occur in the
|
|
|
|
* vDSO. In practice, a small number of Android devices were shipped
|
|
|
|
* with a copy of Bionic that inlined a SYSENTER instruction. This
|
|
|
|
* never happened in any of Google's Bionic versions -- it only happened
|
|
|
|
* in a narrow range of Intel-provided versions.
|
|
|
|
*
|
|
|
|
* SYSENTER loads SS, ESP, CS, and EIP from previously programmed MSRs.
|
|
|
|
* IF and VM in RFLAGS are cleared (IOW: interrupts are off).
|
|
|
|
* SYSENTER does not save anything on the stack,
|
|
|
|
* and does not save old EIP (!!!), ESP, or EFLAGS.
|
|
|
|
*
|
|
|
|
* To avoid losing track of EFLAGS.VM (and thus potentially corrupting
|
|
|
|
* user and/or vm86 state), we explicitly disable the SYSENTER
|
|
|
|
* instruction in vm86 mode by reprogramming the MSRs.
|
|
|
|
*
|
|
|
|
* Arguments:
|
|
|
|
* eax system call number
|
|
|
|
* ebx arg1
|
|
|
|
* ecx arg2
|
|
|
|
* edx arg3
|
|
|
|
* esi arg4
|
|
|
|
* edi arg5
|
|
|
|
* ebp user stack
|
|
|
|
* 0(%ebp) arg6
|
|
|
|
*/
|
2015-06-08 06:33:56 +00:00
|
|
|
ENTRY(entry_SYSENTER_32)
|
2015-06-08 07:49:11 +00:00
|
|
|
movl TSS_sysenter_sp0(%esp), %esp
|
2016-09-21 21:03:59 +00:00
|
|
|
.Lsysenter_past_esp:
|
2015-10-06 00:48:15 +00:00
|
|
|
pushl $__USER_DS /* pt_regs->ss */
|
2015-12-17 07:18:48 +00:00
|
|
|
pushl %ebp /* pt_regs->sp (stashed in bp) */
|
2015-10-06 00:48:15 +00:00
|
|
|
pushfl /* pt_regs->flags (except IF = 0) */
|
|
|
|
orl $X86_EFLAGS_IF, (%esp) /* Fix IF */
|
|
|
|
pushl $__USER_CS /* pt_regs->cs */
|
|
|
|
pushl $0 /* pt_regs->ip = 0 (placeholder) */
|
|
|
|
pushl %eax /* pt_regs->orig_ax */
|
|
|
|
SAVE_ALL pt_regs_ax=$-ENOSYS /* save rest */
|
|
|
|
|
2016-03-10 03:00:26 +00:00
|
|
|
/*
|
2016-03-10 03:00:30 +00:00
|
|
|
* SYSENTER doesn't filter flags, so we need to clear NT, AC
|
|
|
|
* and TF ourselves. To save a few cycles, we can check whether
|
2016-03-10 03:00:26 +00:00
|
|
|
* either was set instead of doing an unconditional popfq.
|
|
|
|
* This needs to happen before enabling interrupts so that
|
|
|
|
* we don't get preempted with NT set.
|
|
|
|
*
|
2016-03-10 03:00:30 +00:00
|
|
|
* If TF is set, we will single-step all the way to here -- do_debug
|
|
|
|
* will ignore all the traps. (Yes, this is slow, but so is
|
|
|
|
* single-stepping in general. This allows us to avoid having
|
|
|
|
* a more complicated code to handle the case where a user program
|
|
|
|
* forces us to single-step through the SYSENTER entry code.)
|
|
|
|
*
|
2016-03-10 03:00:26 +00:00
|
|
|
* NB.: .Lsysenter_fix_flags is a label with the code under it moved
|
|
|
|
* out-of-line as an optimization: NT is unlikely to be set in the
|
|
|
|
* majority of the cases and instead of polluting the I$ unnecessarily,
|
|
|
|
* we're keeping that code behind a branch which will predict as
|
|
|
|
* not-taken and therefore its instructions won't be fetched.
|
|
|
|
*/
|
2016-03-10 03:00:30 +00:00
|
|
|
testl $X86_EFLAGS_NT|X86_EFLAGS_AC|X86_EFLAGS_TF, PT_EFLAGS(%esp)
|
2016-03-10 03:00:26 +00:00
|
|
|
jnz .Lsysenter_fix_flags
|
|
|
|
.Lsysenter_flags_fixed:
|
|
|
|
|
2006-07-03 07:24:43 +00:00
|
|
|
/*
|
2015-10-06 00:48:15 +00:00
|
|
|
* User mode is traced as though IRQs are on, and SYSENTER
|
|
|
|
* turned them off.
|
[PATCH] vdso: randomize the i386 vDSO by moving it into a vma
Move the i386 VDSO down into a vma and thus randomize it.
Besides the security implications, this feature also helps debuggers, which
can COW a vma-backed VDSO just like a normal DSO and can thus do
single-stepping and other debugging features.
It's good for hypervisors (Xen, VMWare) too, which typically live in the same
high-mapped address space as the VDSO, hence whenever the VDSO is used, they
get lots of guest pagefaults and have to fix such guest accesses up - which
slows things down instead of speeding things up (the primary purpose of the
VDSO).
There's a new CONFIG_COMPAT_VDSO (default=y) option, which provides support
for older glibcs that still rely on a prelinked high-mapped VDSO. Newer
distributions (using glibc 2.3.3 or later) can turn this option off. Turning
it off is also recommended for security reasons: attackers cannot use the
predictable high-mapped VDSO page as syscall trampoline anymore.
There is a new vdso=[0|1] boot option as well, and a runtime
/proc/sys/vm/vdso_enabled sysctl switch, that allows the VDSO to be turned
on/off.
(This version of the VDSO-randomization patch also has working ELF
coredumping, the previous patch crashed in the coredumping code.)
This code is a combined work of the exec-shield VDSO randomization
code and Gerd Hoffmann's hypervisor-centric VDSO patch. Rusty Russell
started this patch and i completed it.
[akpm@osdl.org: cleanups]
[akpm@osdl.org: compile fix]
[akpm@osdl.org: compile fix 2]
[akpm@osdl.org: compile fix 3]
[akpm@osdl.org: revernt MAXMEM change]
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Arjan van de Ven <arjan@infradead.org>
Cc: Gerd Hoffmann <kraxel@suse.de>
Cc: Rusty Russell <rusty@rustcorp.com.au>
Cc: Zachary Amsden <zach@vmware.com>
Cc: Andi Kleen <ak@muc.de>
Cc: Jan Beulich <jbeulich@novell.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 09:53:50 +00:00
|
|
|
*/
|
2006-07-03 07:24:43 +00:00
|
|
|
TRACE_IRQS_OFF
|
2015-10-06 00:48:15 +00:00
|
|
|
|
|
|
|
movl %esp, %eax
|
|
|
|
call do_fast_syscall_32
|
2015-11-19 21:55:45 +00:00
|
|
|
/* XEN PV guests always use IRET path */
|
|
|
|
ALTERNATIVE "testl %eax, %eax; jz .Lsyscall_32_done", \
|
|
|
|
"jmp .Lsyscall_32_done", X86_FEATURE_XENPV
|
2015-10-06 00:48:15 +00:00
|
|
|
|
|
|
|
/* Opportunistic SYSEXIT */
|
|
|
|
TRACE_IRQS_ON /* User mode traces as IRQs on. */
|
|
|
|
movl PT_EIP(%esp), %edx /* pt_regs->ip */
|
|
|
|
movl PT_OLDESP(%esp), %ecx /* pt_regs->sp */
|
2015-10-16 22:42:55 +00:00
|
|
|
1: mov PT_FS(%esp), %fs
|
|
|
|
PTGS_TO_GS
|
2015-10-06 00:48:15 +00:00
|
|
|
popl %ebx /* pt_regs->bx */
|
|
|
|
addl $2*4, %esp /* skip pt_regs->cx and pt_regs->dx */
|
|
|
|
popl %esi /* pt_regs->si */
|
|
|
|
popl %edi /* pt_regs->di */
|
|
|
|
popl %ebp /* pt_regs->bp */
|
|
|
|
popl %eax /* pt_regs->ax */
|
|
|
|
|
2016-03-10 03:00:27 +00:00
|
|
|
/*
|
|
|
|
* Restore all flags except IF. (We restore IF separately because
|
|
|
|
* STI gives a one-instruction window in which we won't be interrupted,
|
|
|
|
* whereas POPF does not.)
|
|
|
|
*/
|
|
|
|
addl $PT_EFLAGS-PT_DS, %esp /* point esp at pt_regs->flags */
|
|
|
|
btr $X86_EFLAGS_IF_BIT, (%esp)
|
|
|
|
popfl
|
|
|
|
|
2015-10-06 00:48:15 +00:00
|
|
|
/*
|
|
|
|
* Return back to the vDSO, which will pop ecx and edx.
|
|
|
|
* Don't bother with DS and ES (they already contain __USER_DS).
|
|
|
|
*/
|
2015-11-19 21:55:46 +00:00
|
|
|
sti
|
|
|
|
sysexit
|
2008-06-24 11:16:52 +00:00
|
|
|
|
2015-06-08 07:49:11 +00:00
|
|
|
.pushsection .fixup, "ax"
|
|
|
|
2: movl $0, PT_FS(%esp)
|
|
|
|
jmp 1b
|
[PATCH] i386: Use %gs as the PDA base-segment in the kernel
This patch is the meat of the PDA change. This patch makes several related
changes:
1: Most significantly, %gs is now used in the kernel. This means that on
entry, the old value of %gs is saved away, and it is reloaded with
__KERNEL_PDA.
2: entry.S constructs the stack in the shape of struct pt_regs, and this
is passed around the kernel so that the process's saved register
state can be accessed.
Unfortunately struct pt_regs doesn't currently have space for %gs
(or %fs). This patch extends pt_regs to add space for gs (no space
is allocated for %fs, since it won't be used, and it would just
complicate the code in entry.S to work around the space).
3: Because %gs is now saved on the stack like %ds, %es and the integer
registers, there are a number of places where it no longer needs to
be handled specially; namely context switch, and saving/restoring the
register state in a signal context.
4: And since kernel threads run in kernel space and call normal kernel
code, they need to be created with their %gs == __KERNEL_PDA.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Andi Kleen <ak@suse.de>
Cc: Chuck Ebbert <76306.1226@compuserve.com>
Cc: Zachary Amsden <zach@vmware.com>
Cc: Jan Beulich <jbeulich@novell.com>
Cc: Andi Kleen <ak@suse.de>
Signed-off-by: Andrew Morton <akpm@osdl.org>
2006-12-07 01:14:02 +00:00
|
|
|
.popsection
|
2015-06-08 07:49:11 +00:00
|
|
|
_ASM_EXTABLE(1b, 2b)
|
2009-02-09 13:17:40 +00:00
|
|
|
PTGS_TO_GS_EX
|
2016-03-10 03:00:26 +00:00
|
|
|
|
|
|
|
.Lsysenter_fix_flags:
|
|
|
|
pushl $X86_EFLAGS_FIXED
|
|
|
|
popfl
|
|
|
|
jmp .Lsysenter_flags_fixed
|
2016-03-10 03:00:30 +00:00
|
|
|
GLOBAL(__end_SYSENTER_singlestep_region)
|
2015-06-08 06:33:56 +00:00
|
|
|
ENDPROC(entry_SYSENTER_32)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2016-03-10 03:00:35 +00:00
|
|
|
/*
|
|
|
|
* 32-bit legacy system call entry.
|
|
|
|
*
|
|
|
|
* 32-bit x86 Linux system calls traditionally used the INT $0x80
|
|
|
|
* instruction. INT $0x80 lands here.
|
|
|
|
*
|
|
|
|
* This entry point can be used by any 32-bit perform system calls.
|
|
|
|
* Instances of INT $0x80 can be found inline in various programs and
|
|
|
|
* libraries. It is also used by the vDSO's __kernel_vsyscall
|
|
|
|
* fallback for hardware that doesn't support a faster entry method.
|
|
|
|
* Restarted 32-bit system calls also fall back to INT $0x80
|
|
|
|
* regardless of what instruction was originally used to do the system
|
|
|
|
* call. (64-bit programs can use INT $0x80 as well, but they can
|
|
|
|
* only run on 64-bit kernels and therefore land in
|
|
|
|
* entry_INT80_compat.)
|
|
|
|
*
|
|
|
|
* This is considered a slow path. It is not used by most libc
|
|
|
|
* implementations on modern hardware except during process startup.
|
|
|
|
*
|
|
|
|
* Arguments:
|
|
|
|
* eax system call number
|
|
|
|
* ebx arg1
|
|
|
|
* ecx arg2
|
|
|
|
* edx arg3
|
|
|
|
* esi arg4
|
|
|
|
* edi arg5
|
|
|
|
* ebp arg6
|
|
|
|
*/
|
2015-06-08 06:42:03 +00:00
|
|
|
ENTRY(entry_INT80_32)
|
2012-09-21 20:58:10 +00:00
|
|
|
ASM_CLAC
|
2015-10-06 00:48:14 +00:00
|
|
|
pushl %eax /* pt_regs->orig_ax */
|
2015-10-06 00:48:15 +00:00
|
|
|
SAVE_ALL pt_regs_ax=$-ENOSYS /* save rest */
|
2015-10-06 00:48:14 +00:00
|
|
|
|
|
|
|
/*
|
2016-03-09 21:24:32 +00:00
|
|
|
* User mode is traced as though IRQs are on, and the interrupt gate
|
|
|
|
* turned them off.
|
2015-10-06 00:48:14 +00:00
|
|
|
*/
|
2016-03-09 21:24:32 +00:00
|
|
|
TRACE_IRQS_OFF
|
2015-10-06 00:48:14 +00:00
|
|
|
|
|
|
|
movl %esp, %eax
|
2016-03-09 21:24:32 +00:00
|
|
|
call do_int80_syscall_32
|
2015-10-06 00:48:15 +00:00
|
|
|
.Lsyscall_32_done:
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
restore_all:
|
i386: fix return to 16-bit stack from NMI handler
Returning to a task with a 16-bit stack requires special care: the iret
instruction does not restore the high word of esp in that case. The
espfix code fixes this, but currently is not invoked on NMIs. This means
that a running task gets the upper word of esp clobbered due intervening
NMIs. To reproduce, compile and run the following program with the nmi
watchdog enabled (nmi_watchdog=2 on the command line). Using gdb you can
see that the high bits of esp contain garbage, while the low bits are
still correct.
This patch puts the espfix code back into the NMI code path.
The patch is slightly complicated due to the irqtrace infrastructure not
being NMI-safe. The NMI return path cannot call TRACE_IRQS_IRET.
Otherwise, the tail of the normal iret-code is correct for the nmi code
path too. To be able to share this code-path, the TRACE_IRQS_IRET was
move up a bit. The espfix code exists after the TRACE_IRQS_IRET, but
this code explicitly disables interrupts. This short interrupts-off
section is now not traced anymore. The return-to-kernel path now always
includes the preliminary test to decide if the espfix code should be
called. This is never the case, but doing it this way keeps the patch as
simple as possible and the few extra instructions should not affect
timing in any significant way.
#define _GNU_SOURCE
#include <stdio.h>
#include <sys/types.h>
#include <sys/mman.h>
#include <unistd.h>
#include <sys/syscall.h>
#include <asm/ldt.h>
int modify_ldt(int func, void *ptr, unsigned long bytecount)
{
return syscall(SYS_modify_ldt, func, ptr, bytecount);
}
/* this is assumed to be usable */
#define SEGBASEADDR 0x10000
#define SEGLIMIT 0x20000
/* 16-bit segment */
struct user_desc desc = {
.entry_number = 0,
.base_addr = SEGBASEADDR,
.limit = SEGLIMIT,
.seg_32bit = 0,
.contents = 0, /* ??? */
.read_exec_only = 0,
.limit_in_pages = 0,
.seg_not_present = 0,
.useable = 1
};
int main(void)
{
setvbuf(stdout, NULL, _IONBF, 0);
/* map a 64 kb segment */
char *pointer = mmap((void *)SEGBASEADDR, SEGLIMIT+1,
PROT_EXEC|PROT_READ|PROT_WRITE,
MAP_SHARED|MAP_ANONYMOUS, -1, 0);
if (pointer == NULL) {
printf("could not map space\n");
return 0;
}
/* write ldt, new mode */
int err = modify_ldt(0x11, &desc, sizeof(desc));
if (err) {
printf("error modifying ldt: %i\n", err);
return 0;
}
for (int i=0; i<1000; i++) {
asm volatile (
"pusha\n\t"
"mov %ss, %eax\n\t" /* preserve ss:esp */
"mov %esp, %ebp\n\t"
"push $7\n\t" /* index 0, ldt, user mode */
"push $65536-4096\n\t" /* esp */
"lss (%esp), %esp\n\t" /* switch to new stack */
"push %eax\n\t" /* save old ss:esp on new stack */
"push %ebp\n\t"
"add $17*65536, %esp\n\t" /* set high bits */
"mov %esp, %edx\n\t"
"mov $10000000, %ecx\n\t" /* wait... */
"1: loop 1b\n\t" /* ... a bit */
"cmp %esp, %edx\n\t"
"je 1f\n\t"
"ud2\n\t" /* esp changed inexplicably! */
"1:\n\t"
"sub $17*65536, %esp\n\t" /* restore high bits */
"lss (%esp), %esp\n\t" /* restore old ss:esp */
"popa\n\t");
printf("\rx%ix", i);
}
return 0;
}
Signed-off-by: Alexander van Heukelum <heukelum@fastmail.fm>
Acked-by: Stas Sergeev <stsp@aknet.ru>
Signed-off-by: H. Peter Anvin <hpa@zytor.com>
2009-06-17 22:35:57 +00:00
|
|
|
TRACE_IRQS_IRET
|
2016-09-21 21:03:59 +00:00
|
|
|
.Lrestore_all_notrace:
|
2014-05-04 17:36:22 +00:00
|
|
|
#ifdef CONFIG_X86_ESPFIX32
|
2016-09-21 21:03:59 +00:00
|
|
|
ALTERNATIVE "jmp .Lrestore_nocheck", "", X86_BUG_ESPFIX
|
x86/entry/32: Introduce and use X86_BUG_ESPFIX instead of paravirt_enabled
x86_64 has very clean espfix handling on paravirt: espfix64 is set
up in native_iret, so paravirt systems that override iret bypass
espfix64 automatically. This is robust and straightforward.
x86_32 is messier. espfix is set up before the IRET paravirt patch
point, so it can't be directly conditionalized on whether we use
native_iret. We also can't easily move it into native_iret without
regressing performance due to a bizarre consideration. Specifically,
on 64-bit kernels, the logic is:
if (regs->ss & 0x4)
setup_espfix;
On 32-bit kernels, the logic is:
if ((regs->ss & 0x4) && (regs->cs & 0x3) == 3 &&
(regs->flags & X86_EFLAGS_VM) == 0)
setup_espfix;
The performance of setup_espfix itself is essentially irrelevant, but
the comparison happens on every IRET so its performance matters. On
x86_64, there's no need for any registers except flags to implement
the comparison, so we fold the whole thing into native_iret. On
x86_32, we don't do that because we need a free register to
implement the comparison efficiently. We therefore do espfix setup
before restoring registers on x86_32.
This patch gets rid of the explicit paravirt_enabled check by
introducing X86_BUG_ESPFIX on 32-bit systems and using an ALTERNATIVE
to skip espfix on paravirt systems where iret != native_iret. This is
also messy, but it's at least in line with other things we do.
This improves espfix performance by removing a branch, but no one
cares. More importantly, it removes a paravirt_enabled user, which is
good because paravirt_enabled is ill-defined and is going away.
Signed-off-by: Andy Lutomirski <luto@kernel.org>
Reviewed-by: Borislav Petkov <bp@suse.de>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Luis R. Rodriguez <mcgrof@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: boris.ostrovsky@oracle.com
Cc: david.vrabel@citrix.com
Cc: konrad.wilk@oracle.com
Cc: lguest@lists.ozlabs.org
Cc: xen-devel@lists.xensource.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2016-02-29 23:50:19 +00:00
|
|
|
|
2015-06-08 07:49:11 +00:00
|
|
|
movl PT_EFLAGS(%esp), %eax # mix EFLAGS, SS and CS
|
|
|
|
/*
|
|
|
|
* Warning: PT_OLDSS(%esp) contains the wrong/random values if we
|
|
|
|
* are returning to the kernel.
|
|
|
|
* See comments in process.c:copy_thread() for details.
|
|
|
|
*/
|
|
|
|
movb PT_OLDSS(%esp), %ah
|
|
|
|
movb PT_CS(%esp), %al
|
|
|
|
andl $(X86_EFLAGS_VM | (SEGMENT_TI_MASK << 8) | SEGMENT_RPL_MASK), %eax
|
|
|
|
cmpl $((SEGMENT_LDT << 8) | USER_RPL), %eax
|
2016-09-21 21:03:59 +00:00
|
|
|
je .Lldt_ss # returning to user-space with LDT SS
|
2014-05-04 17:36:22 +00:00
|
|
|
#endif
|
2016-09-21 21:03:59 +00:00
|
|
|
.Lrestore_nocheck:
|
2015-06-08 07:49:11 +00:00
|
|
|
RESTORE_REGS 4 # skip orig_eax/error_code
|
2016-09-21 21:03:59 +00:00
|
|
|
.Lirq_return:
|
membarrier/x86: Provide core serializing command
There are two places where core serialization is needed by membarrier:
1) When returning from the membarrier IPI,
2) After scheduler updates curr to a thread with a different mm, before
going back to user-space, since the curr->mm is used by membarrier to
check whether it needs to send an IPI to that CPU.
x86-32 uses IRET as return from interrupt, and both IRET and SYSEXIT to go
back to user-space. The IRET instruction is core serializing, but not
SYSEXIT.
x86-64 uses IRET as return from interrupt, which takes care of the IPI.
However, it can return to user-space through either SYSRETL (compat
code), SYSRETQ, or IRET. Given that SYSRET{L,Q} is not core serializing,
we rely instead on write_cr3() performed by switch_mm() to provide core
serialization after changing the current mm, and deal with the special
case of kthread -> uthread (temporarily keeping current mm into
active_mm) by adding a sync_core() in that specific case.
Use the new sync_core_before_usermode() to guarantee this.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Acked-by: Thomas Gleixner <tglx@linutronix.de>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Andrea Parri <parri.andrea@gmail.com>
Cc: Andrew Hunter <ahh@google.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Avi Kivity <avi@scylladb.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Dave Watson <davejwatson@fb.com>
Cc: David Sehr <sehr@google.com>
Cc: Greg Hackmann <ghackmann@google.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Maged Michael <maged.michael@gmail.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Russell King <linux@armlinux.org.uk>
Cc: Will Deacon <will.deacon@arm.com>
Cc: linux-api@vger.kernel.org
Cc: linux-arch@vger.kernel.org
Link: http://lkml.kernel.org/r/20180129202020.8515-10-mathieu.desnoyers@efficios.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-29 20:20:18 +00:00
|
|
|
/*
|
|
|
|
* ARCH_HAS_MEMBARRIER_SYNC_CORE rely on IRET core serialization
|
|
|
|
* when returning from IPI handler and when returning from
|
|
|
|
* scheduler to user-space.
|
|
|
|
*/
|
2008-02-09 22:24:08 +00:00
|
|
|
INTERRUPT_RETURN
|
2016-09-21 21:03:59 +00:00
|
|
|
|
2015-06-08 07:49:11 +00:00
|
|
|
.section .fixup, "ax"
|
|
|
|
ENTRY(iret_exc )
|
|
|
|
pushl $0 # no error code
|
|
|
|
pushl $do_iret_error
|
2016-09-21 21:04:00 +00:00
|
|
|
jmp common_exception
|
2005-04-16 22:20:36 +00:00
|
|
|
.previous
|
2016-09-21 21:03:59 +00:00
|
|
|
_ASM_EXTABLE(.Lirq_return, iret_exc)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2014-05-04 17:36:22 +00:00
|
|
|
#ifdef CONFIG_X86_ESPFIX32
|
2016-09-21 21:03:59 +00:00
|
|
|
.Lldt_ss:
|
i386: fix/simplify espfix stack switching, move it into assembly
The espfix code triggers if we have a protected mode userspace
application with a 16-bit stack. On returning to userspace, with iret,
the CPU doesn't restore the high word of the stack pointer. This is an
"official" bug, and the work-around used in the kernel is to temporarily
switch to a 32-bit stack segment/pointer pair where the high word of the
pointer is equal to the high word of the userspace stackpointer.
The current implementation uses THREAD_SIZE to determine the cut-off,
but there is no good reason not to use the more natural 64kb... However,
implementing this by simply substituting THREAD_SIZE with 65536 in
patch_espfix_desc crashed the test application. patch_espfix_desc tries
to do what is described above, but gets it subtly wrong if the userspace
stack pointer is just below a multiple of THREAD_SIZE: an overflow
occurs to bit 13... With a bit of luck, when the kernelspace
stackpointer is just below a 64kb-boundary, the overflow then ripples
trough to bit 16 and userspace will see its stack pointer changed by
65536.
This patch moves all espfix code into entry_32.S. Selecting a 16-bit
cut-off simplifies the code. The game with changing the limit dynamically
is removed too. It complicates matters and I see no value in it. Changing
only the top 16-bit word of ESP is one instruction and it also implies
that only two bytes of the ESPFIX GDT entry need to be changed and this
can be implemented in just a handful simple to understand instructions.
As a side effect, the operation to compute the original ESP from the
ESPFIX ESP and the GDT entry simplifies a bit too, and the remaining
three instructions have been expanded inline in entry_32.S.
impact: can now reliably run userspace with ESP=xxxxfffc on 16-bit
stack segment
Signed-off-by: Alexander van Heukelum <heukelum@fastmail.fm>
Acked-by: Stas Sergeev <stsp@aknet.ru>
Signed-off-by: H. Peter Anvin <hpa@zytor.com>
2009-06-17 22:35:58 +00:00
|
|
|
/*
|
|
|
|
* Setup and switch to ESPFIX stack
|
|
|
|
*
|
|
|
|
* We're returning to userspace with a 16 bit stack. The CPU will not
|
|
|
|
* restore the high word of ESP for us on executing iret... This is an
|
|
|
|
* "official" bug of all the x86-compatible CPUs, which we can work
|
|
|
|
* around to make dosemu and wine happy. We do this by preloading the
|
|
|
|
* high word of ESP with the high word of the userspace ESP while
|
|
|
|
* compensating for the offset by changing to the ESPFIX segment with
|
|
|
|
* a base address that matches for the difference.
|
|
|
|
*/
|
2010-07-31 16:48:23 +00:00
|
|
|
#define GDT_ESPFIX_SS PER_CPU_VAR(gdt_page) + (GDT_ENTRY_ESPFIX_SS * 8)
|
2015-06-08 07:49:11 +00:00
|
|
|
mov %esp, %edx /* load kernel esp */
|
|
|
|
mov PT_OLDESP(%esp), %eax /* load userspace esp */
|
|
|
|
mov %dx, %ax /* eax: new kernel esp */
|
2015-06-08 20:35:33 +00:00
|
|
|
sub %eax, %edx /* offset (low word is 0) */
|
|
|
|
shr $16, %edx
|
2015-06-08 07:49:11 +00:00
|
|
|
mov %dl, GDT_ESPFIX_SS + 4 /* bits 16..23 */
|
|
|
|
mov %dh, GDT_ESPFIX_SS + 7 /* bits 24..31 */
|
|
|
|
pushl $__ESPFIX_SS
|
|
|
|
pushl %eax /* new kernel esp */
|
|
|
|
/*
|
|
|
|
* Disable interrupts, but do not irqtrace this section: we
|
i386: fix return to 16-bit stack from NMI handler
Returning to a task with a 16-bit stack requires special care: the iret
instruction does not restore the high word of esp in that case. The
espfix code fixes this, but currently is not invoked on NMIs. This means
that a running task gets the upper word of esp clobbered due intervening
NMIs. To reproduce, compile and run the following program with the nmi
watchdog enabled (nmi_watchdog=2 on the command line). Using gdb you can
see that the high bits of esp contain garbage, while the low bits are
still correct.
This patch puts the espfix code back into the NMI code path.
The patch is slightly complicated due to the irqtrace infrastructure not
being NMI-safe. The NMI return path cannot call TRACE_IRQS_IRET.
Otherwise, the tail of the normal iret-code is correct for the nmi code
path too. To be able to share this code-path, the TRACE_IRQS_IRET was
move up a bit. The espfix code exists after the TRACE_IRQS_IRET, but
this code explicitly disables interrupts. This short interrupts-off
section is now not traced anymore. The return-to-kernel path now always
includes the preliminary test to decide if the espfix code should be
called. This is never the case, but doing it this way keeps the patch as
simple as possible and the few extra instructions should not affect
timing in any significant way.
#define _GNU_SOURCE
#include <stdio.h>
#include <sys/types.h>
#include <sys/mman.h>
#include <unistd.h>
#include <sys/syscall.h>
#include <asm/ldt.h>
int modify_ldt(int func, void *ptr, unsigned long bytecount)
{
return syscall(SYS_modify_ldt, func, ptr, bytecount);
}
/* this is assumed to be usable */
#define SEGBASEADDR 0x10000
#define SEGLIMIT 0x20000
/* 16-bit segment */
struct user_desc desc = {
.entry_number = 0,
.base_addr = SEGBASEADDR,
.limit = SEGLIMIT,
.seg_32bit = 0,
.contents = 0, /* ??? */
.read_exec_only = 0,
.limit_in_pages = 0,
.seg_not_present = 0,
.useable = 1
};
int main(void)
{
setvbuf(stdout, NULL, _IONBF, 0);
/* map a 64 kb segment */
char *pointer = mmap((void *)SEGBASEADDR, SEGLIMIT+1,
PROT_EXEC|PROT_READ|PROT_WRITE,
MAP_SHARED|MAP_ANONYMOUS, -1, 0);
if (pointer == NULL) {
printf("could not map space\n");
return 0;
}
/* write ldt, new mode */
int err = modify_ldt(0x11, &desc, sizeof(desc));
if (err) {
printf("error modifying ldt: %i\n", err);
return 0;
}
for (int i=0; i<1000; i++) {
asm volatile (
"pusha\n\t"
"mov %ss, %eax\n\t" /* preserve ss:esp */
"mov %esp, %ebp\n\t"
"push $7\n\t" /* index 0, ldt, user mode */
"push $65536-4096\n\t" /* esp */
"lss (%esp), %esp\n\t" /* switch to new stack */
"push %eax\n\t" /* save old ss:esp on new stack */
"push %ebp\n\t"
"add $17*65536, %esp\n\t" /* set high bits */
"mov %esp, %edx\n\t"
"mov $10000000, %ecx\n\t" /* wait... */
"1: loop 1b\n\t" /* ... a bit */
"cmp %esp, %edx\n\t"
"je 1f\n\t"
"ud2\n\t" /* esp changed inexplicably! */
"1:\n\t"
"sub $17*65536, %esp\n\t" /* restore high bits */
"lss (%esp), %esp\n\t" /* restore old ss:esp */
"popa\n\t");
printf("\rx%ix", i);
}
return 0;
}
Signed-off-by: Alexander van Heukelum <heukelum@fastmail.fm>
Acked-by: Stas Sergeev <stsp@aknet.ru>
Signed-off-by: H. Peter Anvin <hpa@zytor.com>
2009-06-17 22:35:57 +00:00
|
|
|
* will soon execute iret and the tracer was already set to
|
2015-06-08 07:49:11 +00:00
|
|
|
* the irqstate after the IRET:
|
|
|
|
*/
|
2017-02-03 08:58:03 +00:00
|
|
|
DISABLE_INTERRUPTS(CLBR_ANY)
|
2015-06-08 07:49:11 +00:00
|
|
|
lss (%esp), %esp /* switch to espfix segment */
|
2016-09-21 21:03:59 +00:00
|
|
|
jmp .Lrestore_nocheck
|
2014-05-04 17:36:22 +00:00
|
|
|
#endif
|
2015-06-08 06:42:03 +00:00
|
|
|
ENDPROC(entry_INT80_32)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2009-02-09 13:17:40 +00:00
|
|
|
.macro FIXUP_ESPFIX_STACK
|
i386: fix/simplify espfix stack switching, move it into assembly
The espfix code triggers if we have a protected mode userspace
application with a 16-bit stack. On returning to userspace, with iret,
the CPU doesn't restore the high word of the stack pointer. This is an
"official" bug, and the work-around used in the kernel is to temporarily
switch to a 32-bit stack segment/pointer pair where the high word of the
pointer is equal to the high word of the userspace stackpointer.
The current implementation uses THREAD_SIZE to determine the cut-off,
but there is no good reason not to use the more natural 64kb... However,
implementing this by simply substituting THREAD_SIZE with 65536 in
patch_espfix_desc crashed the test application. patch_espfix_desc tries
to do what is described above, but gets it subtly wrong if the userspace
stack pointer is just below a multiple of THREAD_SIZE: an overflow
occurs to bit 13... With a bit of luck, when the kernelspace
stackpointer is just below a 64kb-boundary, the overflow then ripples
trough to bit 16 and userspace will see its stack pointer changed by
65536.
This patch moves all espfix code into entry_32.S. Selecting a 16-bit
cut-off simplifies the code. The game with changing the limit dynamically
is removed too. It complicates matters and I see no value in it. Changing
only the top 16-bit word of ESP is one instruction and it also implies
that only two bytes of the ESPFIX GDT entry need to be changed and this
can be implemented in just a handful simple to understand instructions.
As a side effect, the operation to compute the original ESP from the
ESPFIX ESP and the GDT entry simplifies a bit too, and the remaining
three instructions have been expanded inline in entry_32.S.
impact: can now reliably run userspace with ESP=xxxxfffc on 16-bit
stack segment
Signed-off-by: Alexander van Heukelum <heukelum@fastmail.fm>
Acked-by: Stas Sergeev <stsp@aknet.ru>
Signed-off-by: H. Peter Anvin <hpa@zytor.com>
2009-06-17 22:35:58 +00:00
|
|
|
/*
|
|
|
|
* Switch back for ESPFIX stack to the normal zerobased stack
|
|
|
|
*
|
|
|
|
* We can't call C functions using the ESPFIX stack. This code reads
|
|
|
|
* the high word of the segment base from the GDT and swiches to the
|
|
|
|
* normal stack and adjusts ESP with the matching offset.
|
|
|
|
*/
|
2014-05-04 17:36:22 +00:00
|
|
|
#ifdef CONFIG_X86_ESPFIX32
|
i386: fix/simplify espfix stack switching, move it into assembly
The espfix code triggers if we have a protected mode userspace
application with a 16-bit stack. On returning to userspace, with iret,
the CPU doesn't restore the high word of the stack pointer. This is an
"official" bug, and the work-around used in the kernel is to temporarily
switch to a 32-bit stack segment/pointer pair where the high word of the
pointer is equal to the high word of the userspace stackpointer.
The current implementation uses THREAD_SIZE to determine the cut-off,
but there is no good reason not to use the more natural 64kb... However,
implementing this by simply substituting THREAD_SIZE with 65536 in
patch_espfix_desc crashed the test application. patch_espfix_desc tries
to do what is described above, but gets it subtly wrong if the userspace
stack pointer is just below a multiple of THREAD_SIZE: an overflow
occurs to bit 13... With a bit of luck, when the kernelspace
stackpointer is just below a 64kb-boundary, the overflow then ripples
trough to bit 16 and userspace will see its stack pointer changed by
65536.
This patch moves all espfix code into entry_32.S. Selecting a 16-bit
cut-off simplifies the code. The game with changing the limit dynamically
is removed too. It complicates matters and I see no value in it. Changing
only the top 16-bit word of ESP is one instruction and it also implies
that only two bytes of the ESPFIX GDT entry need to be changed and this
can be implemented in just a handful simple to understand instructions.
As a side effect, the operation to compute the original ESP from the
ESPFIX ESP and the GDT entry simplifies a bit too, and the remaining
three instructions have been expanded inline in entry_32.S.
impact: can now reliably run userspace with ESP=xxxxfffc on 16-bit
stack segment
Signed-off-by: Alexander van Heukelum <heukelum@fastmail.fm>
Acked-by: Stas Sergeev <stsp@aknet.ru>
Signed-off-by: H. Peter Anvin <hpa@zytor.com>
2009-06-17 22:35:58 +00:00
|
|
|
/* fixup the stack */
|
2015-06-08 07:49:11 +00:00
|
|
|
mov GDT_ESPFIX_SS + 4, %al /* bits 16..23 */
|
|
|
|
mov GDT_ESPFIX_SS + 7, %ah /* bits 24..31 */
|
2015-06-08 20:35:33 +00:00
|
|
|
shl $16, %eax
|
2015-06-08 07:49:11 +00:00
|
|
|
addl %esp, %eax /* the adjusted stack pointer */
|
|
|
|
pushl $__KERNEL_DS
|
|
|
|
pushl %eax
|
|
|
|
lss (%esp), %esp /* switch to the normal stack segment */
|
2014-05-04 17:36:22 +00:00
|
|
|
#endif
|
2009-02-09 13:17:40 +00:00
|
|
|
.endm
|
|
|
|
.macro UNWIND_ESPFIX_STACK
|
2014-05-04 17:36:22 +00:00
|
|
|
#ifdef CONFIG_X86_ESPFIX32
|
2015-06-08 07:49:11 +00:00
|
|
|
movl %ss, %eax
|
2009-02-09 13:17:40 +00:00
|
|
|
/* see if on espfix stack */
|
2015-06-08 07:49:11 +00:00
|
|
|
cmpw $__ESPFIX_SS, %ax
|
|
|
|
jne 27f
|
|
|
|
movl $__KERNEL_DS, %eax
|
|
|
|
movl %eax, %ds
|
|
|
|
movl %eax, %es
|
2009-02-09 13:17:40 +00:00
|
|
|
/* switch to normal stack */
|
|
|
|
FIXUP_ESPFIX_STACK
|
|
|
|
27:
|
2014-05-04 17:36:22 +00:00
|
|
|
#endif
|
2009-02-09 13:17:40 +00:00
|
|
|
.endm
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
/*
|
2015-04-03 19:49:13 +00:00
|
|
|
* Build the entry stubs with some assembler magic.
|
|
|
|
* We pack 1 stub into every 8-byte block.
|
2005-04-16 22:20:36 +00:00
|
|
|
*/
|
2015-04-03 19:49:13 +00:00
|
|
|
.align 8
|
2005-04-16 22:20:36 +00:00
|
|
|
ENTRY(irq_entries_start)
|
2015-04-03 19:49:13 +00:00
|
|
|
vector=FIRST_EXTERNAL_VECTOR
|
|
|
|
.rept (FIRST_SYSTEM_VECTOR - FIRST_EXTERNAL_VECTOR)
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl $(~vector+0x80) /* Note: always in signed byte range */
|
2015-04-03 19:49:13 +00:00
|
|
|
vector=vector+1
|
|
|
|
jmp common_interrupt
|
|
|
|
.align 8
|
|
|
|
.endr
|
2007-02-13 12:26:24 +00:00
|
|
|
END(irq_entries_start)
|
|
|
|
|
2006-07-03 07:24:43 +00:00
|
|
|
/*
|
|
|
|
* the CPU automatically disables interrupts when executing an IRQ vector,
|
|
|
|
* so IRQ-flags tracing has to follow that:
|
|
|
|
*/
|
2008-11-11 21:24:58 +00:00
|
|
|
.p2align CONFIG_X86_L1_CACHE_SHIFT
|
2005-04-16 22:20:36 +00:00
|
|
|
common_interrupt:
|
2012-09-21 20:58:10 +00:00
|
|
|
ASM_CLAC
|
2015-06-08 07:49:11 +00:00
|
|
|
addl $-0x80, (%esp) /* Adjust vector into the [-256, -1] range */
|
2005-04-16 22:20:36 +00:00
|
|
|
SAVE_ALL
|
2016-10-20 16:34:40 +00:00
|
|
|
ENCODE_FRAME_POINTER
|
2006-07-03 07:24:43 +00:00
|
|
|
TRACE_IRQS_OFF
|
2015-06-08 07:49:11 +00:00
|
|
|
movl %esp, %eax
|
|
|
|
call do_IRQ
|
|
|
|
jmp ret_from_intr
|
2007-02-13 12:26:24 +00:00
|
|
|
ENDPROC(common_interrupt)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2009-01-21 08:26:06 +00:00
|
|
|
#define BUILD_INTERRUPT3(name, nr, fn) \
|
2005-04-16 22:20:36 +00:00
|
|
|
ENTRY(name) \
|
2012-09-21 20:58:10 +00:00
|
|
|
ASM_CLAC; \
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl $~(nr); \
|
2006-06-26 11:57:44 +00:00
|
|
|
SAVE_ALL; \
|
2016-10-20 16:34:40 +00:00
|
|
|
ENCODE_FRAME_POINTER; \
|
2006-07-03 07:24:43 +00:00
|
|
|
TRACE_IRQS_OFF \
|
2015-06-08 07:49:11 +00:00
|
|
|
movl %esp, %eax; \
|
|
|
|
call fn; \
|
|
|
|
jmp ret_from_intr; \
|
2007-02-13 12:26:24 +00:00
|
|
|
ENDPROC(name)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2015-06-08 07:49:11 +00:00
|
|
|
#define BUILD_INTERRUPT(name, nr) \
|
|
|
|
BUILD_INTERRUPT3(name, nr, smp_##name); \
|
2009-01-21 08:26:06 +00:00
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/* The include is where all of the SMP etc. interrupts come from */
|
2009-01-28 18:34:09 +00:00
|
|
|
#include <asm/entry_arch.h>
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
ENTRY(coprocessor_error)
|
2012-09-21 20:58:10 +00:00
|
|
|
ASM_CLAC
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl $0
|
|
|
|
pushl $do_coprocessor_error
|
2016-09-21 21:04:00 +00:00
|
|
|
jmp common_exception
|
2007-02-13 12:26:24 +00:00
|
|
|
END(coprocessor_error)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
ENTRY(simd_coprocessor_error)
|
2012-09-21 20:58:10 +00:00
|
|
|
ASM_CLAC
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl $0
|
2010-03-21 13:00:43 +00:00
|
|
|
#ifdef CONFIG_X86_INVD_BUG
|
|
|
|
/* AMD 486 bug: invd from userspace calls exception 19 instead of #GP */
|
2015-06-08 07:49:11 +00:00
|
|
|
ALTERNATIVE "pushl $do_general_protection", \
|
|
|
|
"pushl $do_simd_coprocessor_error", \
|
2015-01-18 11:35:55 +00:00
|
|
|
X86_FEATURE_XMM
|
2010-03-21 13:00:43 +00:00
|
|
|
#else
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl $do_simd_coprocessor_error
|
2010-03-21 13:00:43 +00:00
|
|
|
#endif
|
2016-09-21 21:04:00 +00:00
|
|
|
jmp common_exception
|
2007-02-13 12:26:24 +00:00
|
|
|
END(simd_coprocessor_error)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
ENTRY(device_not_available)
|
2012-09-21 20:58:10 +00:00
|
|
|
ASM_CLAC
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl $-1 # mark this as an int
|
|
|
|
pushl $do_device_not_available
|
2016-09-21 21:04:00 +00:00
|
|
|
jmp common_exception
|
2007-02-13 12:26:24 +00:00
|
|
|
END(device_not_available)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2006-12-07 01:14:07 +00:00
|
|
|
#ifdef CONFIG_PARAVIRT
|
|
|
|
ENTRY(native_iret)
|
2008-02-09 22:24:08 +00:00
|
|
|
iret
|
2012-04-20 19:19:50 +00:00
|
|
|
_ASM_EXTABLE(native_iret, iret_exc)
|
2007-02-13 12:26:24 +00:00
|
|
|
END(native_iret)
|
2006-12-07 01:14:07 +00:00
|
|
|
#endif
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
ENTRY(overflow)
|
2012-09-21 20:58:10 +00:00
|
|
|
ASM_CLAC
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl $0
|
|
|
|
pushl $do_overflow
|
2016-09-21 21:04:00 +00:00
|
|
|
jmp common_exception
|
2007-02-13 12:26:24 +00:00
|
|
|
END(overflow)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
ENTRY(bounds)
|
2012-09-21 20:58:10 +00:00
|
|
|
ASM_CLAC
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl $0
|
|
|
|
pushl $do_bounds
|
2016-09-21 21:04:00 +00:00
|
|
|
jmp common_exception
|
2007-02-13 12:26:24 +00:00
|
|
|
END(bounds)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
ENTRY(invalid_op)
|
2012-09-21 20:58:10 +00:00
|
|
|
ASM_CLAC
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl $0
|
|
|
|
pushl $do_invalid_op
|
2016-09-21 21:04:00 +00:00
|
|
|
jmp common_exception
|
2007-02-13 12:26:24 +00:00
|
|
|
END(invalid_op)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
ENTRY(coprocessor_segment_overrun)
|
2012-09-21 20:58:10 +00:00
|
|
|
ASM_CLAC
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl $0
|
|
|
|
pushl $do_coprocessor_segment_overrun
|
2016-09-21 21:04:00 +00:00
|
|
|
jmp common_exception
|
2007-02-13 12:26:24 +00:00
|
|
|
END(coprocessor_segment_overrun)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
ENTRY(invalid_TSS)
|
2012-09-21 20:58:10 +00:00
|
|
|
ASM_CLAC
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl $do_invalid_TSS
|
2016-09-21 21:04:00 +00:00
|
|
|
jmp common_exception
|
2007-02-13 12:26:24 +00:00
|
|
|
END(invalid_TSS)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
ENTRY(segment_not_present)
|
2012-09-21 20:58:10 +00:00
|
|
|
ASM_CLAC
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl $do_segment_not_present
|
2016-09-21 21:04:00 +00:00
|
|
|
jmp common_exception
|
2007-02-13 12:26:24 +00:00
|
|
|
END(segment_not_present)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
ENTRY(stack_segment)
|
2012-09-21 20:58:10 +00:00
|
|
|
ASM_CLAC
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl $do_stack_segment
|
2016-09-21 21:04:00 +00:00
|
|
|
jmp common_exception
|
2007-02-13 12:26:24 +00:00
|
|
|
END(stack_segment)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
ENTRY(alignment_check)
|
2012-09-21 20:58:10 +00:00
|
|
|
ASM_CLAC
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl $do_alignment_check
|
2016-09-21 21:04:00 +00:00
|
|
|
jmp common_exception
|
2007-02-13 12:26:24 +00:00
|
|
|
END(alignment_check)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
[PATCH] x86: error_code is not safe for kprobes
This patch moves the entry.S:error_entry to .kprobes.text section,
since code marked unsafe for kprobes jumps directly to entry.S::error_entry,
that must be marked unsafe as well.
This patch also moves all the ".previous.text" asm directives to ".previous"
for kprobes section.
AK: Following a similar i386 patch from Chuck Ebbert
AK: Also merged Jeremy's fix in.
+From: Jeremy Fitzhardinge <jeremy@goop.org>
KPROBE_ENTRY does a .section .kprobes.text, and expects its users to
do a .previous at the end of the function.
Unfortunately, if any code within the function switches sections, for
example .fixup, then the .previous ends up putting all subsequent code
into .fixup. Worse, any subsequent .fixup code gets intermingled with
the code its supposed to be fixing (which is also in .fixup). It's
surprising this didn't cause more havok.
The fix is to use .pushsection/.popsection, so this stuff nests
properly. A further cleanup would be to get rid of all
.section/.previous pairs, since they're inherently fragile.
+From: Chuck Ebbert <76306.1226@compuserve.com>
Because code marked unsafe for kprobes jumps directly to
entry.S::error_code, that must be marked unsafe as well.
The easiest way to do that is to move the page fault entry
point to just before error_code and let it inherit the same
section.
Also moved all the ".previous" asm directives for kprobes
sections to column 1 and removed ".text" from them.
Signed-off-by: Chuck Ebbert <76306.1226@compuserve.com>
Signed-off-by: Andi Kleen <ak@suse.de>
2006-09-26 08:52:34 +00:00
|
|
|
ENTRY(divide_error)
|
2012-09-21 20:58:10 +00:00
|
|
|
ASM_CLAC
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl $0 # no error code
|
|
|
|
pushl $do_divide_error
|
2016-09-21 21:04:00 +00:00
|
|
|
jmp common_exception
|
2007-02-13 12:26:24 +00:00
|
|
|
END(divide_error)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
#ifdef CONFIG_X86_MCE
|
|
|
|
ENTRY(machine_check)
|
2012-09-21 20:58:10 +00:00
|
|
|
ASM_CLAC
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl $0
|
|
|
|
pushl machine_check_vector
|
2016-09-21 21:04:00 +00:00
|
|
|
jmp common_exception
|
2007-02-13 12:26:24 +00:00
|
|
|
END(machine_check)
|
2005-04-16 22:20:36 +00:00
|
|
|
#endif
|
|
|
|
|
|
|
|
ENTRY(spurious_interrupt_bug)
|
2012-09-21 20:58:10 +00:00
|
|
|
ASM_CLAC
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl $0
|
|
|
|
pushl $do_spurious_interrupt_bug
|
2016-09-21 21:04:00 +00:00
|
|
|
jmp common_exception
|
2007-02-13 12:26:24 +00:00
|
|
|
END(spurious_interrupt_bug)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
xen: Core Xen implementation
This patch is a rollup of all the core pieces of the Xen
implementation, including:
- booting and setup
- pagetable setup
- privileged instructions
- segmentation
- interrupt flags
- upcalls
- multicall batching
BOOTING AND SETUP
The vmlinux image is decorated with ELF notes which tell the Xen
domain builder what the kernel's requirements are; the domain builder
then constructs the address space accordingly and starts the kernel.
Xen has its own entrypoint for the kernel (contained in an ELF note).
The ELF notes are set up by xen-head.S, which is included into head.S.
In principle it could be linked separately, but it seems to provoke
lots of binutils bugs.
Because the domain builder starts the kernel in a fairly sane state
(32-bit protected mode, paging enabled, flat segments set up), there's
not a lot of setup needed before starting the kernel proper. The main
steps are:
1. Install the Xen paravirt_ops, which is simply a matter of a
structure assignment.
2. Set init_mm to use the Xen-supplied pagetables (analogous to the
head.S generated pagetables in a native boot).
3. Reserve address space for Xen, since it takes a chunk at the top
of the address space for its own use.
4. Call start_kernel()
PAGETABLE SETUP
Once we hit the main kernel boot sequence, it will end up calling back
via paravirt_ops to set up various pieces of Xen specific state. One
of the critical things which requires a bit of extra care is the
construction of the initial init_mm pagetable. Because Xen places
tight constraints on pagetables (an active pagetable must always be
valid, and must always be mapped read-only to the guest domain), we
need to be careful when constructing the new pagetable to keep these
constraints in mind. It turns out that the easiest way to do this is
use the initial Xen-provided pagetable as a template, and then just
insert new mappings for memory where a mapping doesn't already exist.
This means that during pagetable setup, it uses a special version of
xen_set_pte which ignores any attempt to remap a read-only page as
read-write (since Xen will map its own initial pagetable as RO), but
lets other changes to the ptes happen, so that things like NX are set
properly.
PRIVILEGED INSTRUCTIONS AND SEGMENTATION
When the kernel runs under Xen, it runs in ring 1 rather than ring 0.
This means that it is more privileged than user-mode in ring 3, but it
still can't run privileged instructions directly. Non-performance
critical instructions are dealt with by taking a privilege exception
and trapping into the hypervisor and emulating the instruction, but
more performance-critical instructions have their own specific
paravirt_ops. In many cases we can avoid having to do any hypercalls
for these instructions, or the Xen implementation is quite different
from the normal native version.
The privileged instructions fall into the broad classes of:
Segmentation: setting up the GDT and the GDT entries, LDT,
TLS and so on. Xen doesn't allow the GDT to be directly
modified; all GDT updates are done via hypercalls where the new
entries can be validated. This is important because Xen uses
segment limits to prevent the guest kernel from damaging the
hypervisor itself.
Traps and exceptions: Xen uses a special format for trap entrypoints,
so when the kernel wants to set an IDT entry, it needs to be
converted to the form Xen expects. Xen sets int 0x80 up specially
so that the trap goes straight from userspace into the guest kernel
without going via the hypervisor. sysenter isn't supported.
Kernel stack: The esp0 entry is extracted from the tss and provided to
Xen.
TLB operations: the various TLB calls are mapped into corresponding
Xen hypercalls.
Control registers: all the control registers are privileged. The most
important is cr3, which points to the base of the current pagetable,
and we handle it specially.
Another instruction we treat specially is CPUID, even though its not
privileged. We want to control what CPU features are visible to the
rest of the kernel, and so CPUID ends up going into a paravirt_op.
Xen implements this mainly to disable the ACPI and APIC subsystems.
INTERRUPT FLAGS
Xen maintains its own separate flag for masking events, which is
contained within the per-cpu vcpu_info structure. Because the guest
kernel runs in ring 1 and not 0, the IF flag in EFLAGS is completely
ignored (and must be, because even if a guest domain disables
interrupts for itself, it can't disable them overall).
(A note on terminology: "events" and interrupts are effectively
synonymous. However, rather than using an "enable flag", Xen uses a
"mask flag", which blocks event delivery when it is non-zero.)
There are paravirt_ops for each of cli/sti/save_fl/restore_fl, which
are implemented to manage the Xen event mask state. The only thing
worth noting is that when events are unmasked, we need to explicitly
see if there's a pending event and call into the hypervisor to make
sure it gets delivered.
UPCALLS
Xen needs a couple of upcall (or callback) functions to be implemented
by each guest. One is the event upcalls, which is how events
(interrupts, effectively) are delivered to the guests. The other is
the failsafe callback, which is used to report errors in either
reloading a segment register, or caused by iret. These are
implemented in i386/kernel/entry.S so they can jump into the normal
iret_exc path when necessary.
MULTICALL BATCHING
Xen provides a multicall mechanism, which allows multiple hypercalls
to be issued at once in order to mitigate the cost of trapping into
the hypervisor. This is particularly useful for context switches,
since the 4-5 hypercalls they would normally need (reload cr3, update
TLS, maybe update LDT) can be reduced to one. This patch implements a
generic batching mechanism for hypercalls, which gets used in many
places in the Xen code.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Chris Wright <chrisw@sous-sol.org>
Cc: Ian Pratt <ian.pratt@xensource.com>
Cc: Christian Limpach <Christian.Limpach@cl.cam.ac.uk>
Cc: Adrian Bunk <bunk@stusta.de>
2007-07-18 01:37:04 +00:00
|
|
|
#ifdef CONFIG_XEN
|
|
|
|
ENTRY(xen_hypervisor_callback)
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl $-1 /* orig_ax = -1 => not a system call */
|
xen: Core Xen implementation
This patch is a rollup of all the core pieces of the Xen
implementation, including:
- booting and setup
- pagetable setup
- privileged instructions
- segmentation
- interrupt flags
- upcalls
- multicall batching
BOOTING AND SETUP
The vmlinux image is decorated with ELF notes which tell the Xen
domain builder what the kernel's requirements are; the domain builder
then constructs the address space accordingly and starts the kernel.
Xen has its own entrypoint for the kernel (contained in an ELF note).
The ELF notes are set up by xen-head.S, which is included into head.S.
In principle it could be linked separately, but it seems to provoke
lots of binutils bugs.
Because the domain builder starts the kernel in a fairly sane state
(32-bit protected mode, paging enabled, flat segments set up), there's
not a lot of setup needed before starting the kernel proper. The main
steps are:
1. Install the Xen paravirt_ops, which is simply a matter of a
structure assignment.
2. Set init_mm to use the Xen-supplied pagetables (analogous to the
head.S generated pagetables in a native boot).
3. Reserve address space for Xen, since it takes a chunk at the top
of the address space for its own use.
4. Call start_kernel()
PAGETABLE SETUP
Once we hit the main kernel boot sequence, it will end up calling back
via paravirt_ops to set up various pieces of Xen specific state. One
of the critical things which requires a bit of extra care is the
construction of the initial init_mm pagetable. Because Xen places
tight constraints on pagetables (an active pagetable must always be
valid, and must always be mapped read-only to the guest domain), we
need to be careful when constructing the new pagetable to keep these
constraints in mind. It turns out that the easiest way to do this is
use the initial Xen-provided pagetable as a template, and then just
insert new mappings for memory where a mapping doesn't already exist.
This means that during pagetable setup, it uses a special version of
xen_set_pte which ignores any attempt to remap a read-only page as
read-write (since Xen will map its own initial pagetable as RO), but
lets other changes to the ptes happen, so that things like NX are set
properly.
PRIVILEGED INSTRUCTIONS AND SEGMENTATION
When the kernel runs under Xen, it runs in ring 1 rather than ring 0.
This means that it is more privileged than user-mode in ring 3, but it
still can't run privileged instructions directly. Non-performance
critical instructions are dealt with by taking a privilege exception
and trapping into the hypervisor and emulating the instruction, but
more performance-critical instructions have their own specific
paravirt_ops. In many cases we can avoid having to do any hypercalls
for these instructions, or the Xen implementation is quite different
from the normal native version.
The privileged instructions fall into the broad classes of:
Segmentation: setting up the GDT and the GDT entries, LDT,
TLS and so on. Xen doesn't allow the GDT to be directly
modified; all GDT updates are done via hypercalls where the new
entries can be validated. This is important because Xen uses
segment limits to prevent the guest kernel from damaging the
hypervisor itself.
Traps and exceptions: Xen uses a special format for trap entrypoints,
so when the kernel wants to set an IDT entry, it needs to be
converted to the form Xen expects. Xen sets int 0x80 up specially
so that the trap goes straight from userspace into the guest kernel
without going via the hypervisor. sysenter isn't supported.
Kernel stack: The esp0 entry is extracted from the tss and provided to
Xen.
TLB operations: the various TLB calls are mapped into corresponding
Xen hypercalls.
Control registers: all the control registers are privileged. The most
important is cr3, which points to the base of the current pagetable,
and we handle it specially.
Another instruction we treat specially is CPUID, even though its not
privileged. We want to control what CPU features are visible to the
rest of the kernel, and so CPUID ends up going into a paravirt_op.
Xen implements this mainly to disable the ACPI and APIC subsystems.
INTERRUPT FLAGS
Xen maintains its own separate flag for masking events, which is
contained within the per-cpu vcpu_info structure. Because the guest
kernel runs in ring 1 and not 0, the IF flag in EFLAGS is completely
ignored (and must be, because even if a guest domain disables
interrupts for itself, it can't disable them overall).
(A note on terminology: "events" and interrupts are effectively
synonymous. However, rather than using an "enable flag", Xen uses a
"mask flag", which blocks event delivery when it is non-zero.)
There are paravirt_ops for each of cli/sti/save_fl/restore_fl, which
are implemented to manage the Xen event mask state. The only thing
worth noting is that when events are unmasked, we need to explicitly
see if there's a pending event and call into the hypervisor to make
sure it gets delivered.
UPCALLS
Xen needs a couple of upcall (or callback) functions to be implemented
by each guest. One is the event upcalls, which is how events
(interrupts, effectively) are delivered to the guests. The other is
the failsafe callback, which is used to report errors in either
reloading a segment register, or caused by iret. These are
implemented in i386/kernel/entry.S so they can jump into the normal
iret_exc path when necessary.
MULTICALL BATCHING
Xen provides a multicall mechanism, which allows multiple hypercalls
to be issued at once in order to mitigate the cost of trapping into
the hypervisor. This is particularly useful for context switches,
since the 4-5 hypercalls they would normally need (reload cr3, update
TLS, maybe update LDT) can be reduced to one. This patch implements a
generic batching mechanism for hypercalls, which gets used in many
places in the Xen code.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Chris Wright <chrisw@sous-sol.org>
Cc: Ian Pratt <ian.pratt@xensource.com>
Cc: Christian Limpach <Christian.Limpach@cl.cam.ac.uk>
Cc: Adrian Bunk <bunk@stusta.de>
2007-07-18 01:37:04 +00:00
|
|
|
SAVE_ALL
|
2016-10-20 16:34:40 +00:00
|
|
|
ENCODE_FRAME_POINTER
|
xen: Core Xen implementation
This patch is a rollup of all the core pieces of the Xen
implementation, including:
- booting and setup
- pagetable setup
- privileged instructions
- segmentation
- interrupt flags
- upcalls
- multicall batching
BOOTING AND SETUP
The vmlinux image is decorated with ELF notes which tell the Xen
domain builder what the kernel's requirements are; the domain builder
then constructs the address space accordingly and starts the kernel.
Xen has its own entrypoint for the kernel (contained in an ELF note).
The ELF notes are set up by xen-head.S, which is included into head.S.
In principle it could be linked separately, but it seems to provoke
lots of binutils bugs.
Because the domain builder starts the kernel in a fairly sane state
(32-bit protected mode, paging enabled, flat segments set up), there's
not a lot of setup needed before starting the kernel proper. The main
steps are:
1. Install the Xen paravirt_ops, which is simply a matter of a
structure assignment.
2. Set init_mm to use the Xen-supplied pagetables (analogous to the
head.S generated pagetables in a native boot).
3. Reserve address space for Xen, since it takes a chunk at the top
of the address space for its own use.
4. Call start_kernel()
PAGETABLE SETUP
Once we hit the main kernel boot sequence, it will end up calling back
via paravirt_ops to set up various pieces of Xen specific state. One
of the critical things which requires a bit of extra care is the
construction of the initial init_mm pagetable. Because Xen places
tight constraints on pagetables (an active pagetable must always be
valid, and must always be mapped read-only to the guest domain), we
need to be careful when constructing the new pagetable to keep these
constraints in mind. It turns out that the easiest way to do this is
use the initial Xen-provided pagetable as a template, and then just
insert new mappings for memory where a mapping doesn't already exist.
This means that during pagetable setup, it uses a special version of
xen_set_pte which ignores any attempt to remap a read-only page as
read-write (since Xen will map its own initial pagetable as RO), but
lets other changes to the ptes happen, so that things like NX are set
properly.
PRIVILEGED INSTRUCTIONS AND SEGMENTATION
When the kernel runs under Xen, it runs in ring 1 rather than ring 0.
This means that it is more privileged than user-mode in ring 3, but it
still can't run privileged instructions directly. Non-performance
critical instructions are dealt with by taking a privilege exception
and trapping into the hypervisor and emulating the instruction, but
more performance-critical instructions have their own specific
paravirt_ops. In many cases we can avoid having to do any hypercalls
for these instructions, or the Xen implementation is quite different
from the normal native version.
The privileged instructions fall into the broad classes of:
Segmentation: setting up the GDT and the GDT entries, LDT,
TLS and so on. Xen doesn't allow the GDT to be directly
modified; all GDT updates are done via hypercalls where the new
entries can be validated. This is important because Xen uses
segment limits to prevent the guest kernel from damaging the
hypervisor itself.
Traps and exceptions: Xen uses a special format for trap entrypoints,
so when the kernel wants to set an IDT entry, it needs to be
converted to the form Xen expects. Xen sets int 0x80 up specially
so that the trap goes straight from userspace into the guest kernel
without going via the hypervisor. sysenter isn't supported.
Kernel stack: The esp0 entry is extracted from the tss and provided to
Xen.
TLB operations: the various TLB calls are mapped into corresponding
Xen hypercalls.
Control registers: all the control registers are privileged. The most
important is cr3, which points to the base of the current pagetable,
and we handle it specially.
Another instruction we treat specially is CPUID, even though its not
privileged. We want to control what CPU features are visible to the
rest of the kernel, and so CPUID ends up going into a paravirt_op.
Xen implements this mainly to disable the ACPI and APIC subsystems.
INTERRUPT FLAGS
Xen maintains its own separate flag for masking events, which is
contained within the per-cpu vcpu_info structure. Because the guest
kernel runs in ring 1 and not 0, the IF flag in EFLAGS is completely
ignored (and must be, because even if a guest domain disables
interrupts for itself, it can't disable them overall).
(A note on terminology: "events" and interrupts are effectively
synonymous. However, rather than using an "enable flag", Xen uses a
"mask flag", which blocks event delivery when it is non-zero.)
There are paravirt_ops for each of cli/sti/save_fl/restore_fl, which
are implemented to manage the Xen event mask state. The only thing
worth noting is that when events are unmasked, we need to explicitly
see if there's a pending event and call into the hypervisor to make
sure it gets delivered.
UPCALLS
Xen needs a couple of upcall (or callback) functions to be implemented
by each guest. One is the event upcalls, which is how events
(interrupts, effectively) are delivered to the guests. The other is
the failsafe callback, which is used to report errors in either
reloading a segment register, or caused by iret. These are
implemented in i386/kernel/entry.S so they can jump into the normal
iret_exc path when necessary.
MULTICALL BATCHING
Xen provides a multicall mechanism, which allows multiple hypercalls
to be issued at once in order to mitigate the cost of trapping into
the hypervisor. This is particularly useful for context switches,
since the 4-5 hypercalls they would normally need (reload cr3, update
TLS, maybe update LDT) can be reduced to one. This patch implements a
generic batching mechanism for hypercalls, which gets used in many
places in the Xen code.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Chris Wright <chrisw@sous-sol.org>
Cc: Ian Pratt <ian.pratt@xensource.com>
Cc: Christian Limpach <Christian.Limpach@cl.cam.ac.uk>
Cc: Adrian Bunk <bunk@stusta.de>
2007-07-18 01:37:04 +00:00
|
|
|
TRACE_IRQS_OFF
|
xen: use iret directly when possible
Most of the time we can simply use the iret instruction to exit the
kernel, rather than having to use the iret hypercall - the only
exception is if we're returning into vm86 mode, or from delivering an
NMI (which we don't support yet).
When running native, iret has the behaviour of testing for a pending
interrupt atomically with re-enabling interrupts. Unfortunately
there's no way to do this with Xen, so there's a window in which we
could get a recursive exception after enabling events but before
actually returning to userspace.
This causes a problem: if the nested interrupt causes one of the
task's TIF_WORK_MASK flags to be set, they will not be checked again
before returning to userspace. This means that pending work may be
left pending indefinitely, until the process enters and leaves the
kernel again. The net effect is that a pending signal or reschedule
event could be delayed for an unbounded amount of time.
To deal with this, the xen event upcall handler checks to see if the
EIP is within the critical section of the iret code, after events
are (potentially) enabled up to the iret itself. If its within this
range, it calls the iret critical section fixup, which adjusts the
stack to deal with any unrestored registers, and then shifts the
stack frame up to replace the previous invocation.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
2007-07-18 01:37:07 +00:00
|
|
|
|
2015-06-08 07:49:11 +00:00
|
|
|
/*
|
|
|
|
* Check to see if we got the event in the critical
|
|
|
|
* region in xen_iret_direct, after we've reenabled
|
|
|
|
* events and checked for pending events. This simulates
|
|
|
|
* iret instruction's behaviour where it delivers a
|
|
|
|
* pending interrupt when enabling interrupts:
|
|
|
|
*/
|
|
|
|
movl PT_EIP(%esp), %eax
|
|
|
|
cmpl $xen_iret_start_crit, %eax
|
|
|
|
jb 1f
|
|
|
|
cmpl $xen_iret_end_crit, %eax
|
|
|
|
jae 1f
|
xen: use iret directly when possible
Most of the time we can simply use the iret instruction to exit the
kernel, rather than having to use the iret hypercall - the only
exception is if we're returning into vm86 mode, or from delivering an
NMI (which we don't support yet).
When running native, iret has the behaviour of testing for a pending
interrupt atomically with re-enabling interrupts. Unfortunately
there's no way to do this with Xen, so there's a window in which we
could get a recursive exception after enabling events but before
actually returning to userspace.
This causes a problem: if the nested interrupt causes one of the
task's TIF_WORK_MASK flags to be set, they will not be checked again
before returning to userspace. This means that pending work may be
left pending indefinitely, until the process enters and leaves the
kernel again. The net effect is that a pending signal or reschedule
event could be delayed for an unbounded amount of time.
To deal with this, the xen event upcall handler checks to see if the
EIP is within the critical section of the iret code, after events
are (potentially) enabled up to the iret itself. If its within this
range, it calls the iret critical section fixup, which adjusts the
stack to deal with any unrestored registers, and then shifts the
stack frame up to replace the previous invocation.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
2007-07-18 01:37:07 +00:00
|
|
|
|
2015-06-08 07:49:11 +00:00
|
|
|
jmp xen_iret_crit_fixup
|
2008-03-17 23:37:17 +00:00
|
|
|
|
|
|
|
ENTRY(xen_do_upcall)
|
2015-06-08 07:49:11 +00:00
|
|
|
1: mov %esp, %eax
|
|
|
|
call xen_evtchn_do_upcall
|
2015-02-19 15:23:17 +00:00
|
|
|
#ifndef CONFIG_PREEMPT
|
2015-06-08 07:49:11 +00:00
|
|
|
call xen_maybe_preempt_hcall
|
2015-02-19 15:23:17 +00:00
|
|
|
#endif
|
2015-06-08 07:49:11 +00:00
|
|
|
jmp ret_from_intr
|
xen: Core Xen implementation
This patch is a rollup of all the core pieces of the Xen
implementation, including:
- booting and setup
- pagetable setup
- privileged instructions
- segmentation
- interrupt flags
- upcalls
- multicall batching
BOOTING AND SETUP
The vmlinux image is decorated with ELF notes which tell the Xen
domain builder what the kernel's requirements are; the domain builder
then constructs the address space accordingly and starts the kernel.
Xen has its own entrypoint for the kernel (contained in an ELF note).
The ELF notes are set up by xen-head.S, which is included into head.S.
In principle it could be linked separately, but it seems to provoke
lots of binutils bugs.
Because the domain builder starts the kernel in a fairly sane state
(32-bit protected mode, paging enabled, flat segments set up), there's
not a lot of setup needed before starting the kernel proper. The main
steps are:
1. Install the Xen paravirt_ops, which is simply a matter of a
structure assignment.
2. Set init_mm to use the Xen-supplied pagetables (analogous to the
head.S generated pagetables in a native boot).
3. Reserve address space for Xen, since it takes a chunk at the top
of the address space for its own use.
4. Call start_kernel()
PAGETABLE SETUP
Once we hit the main kernel boot sequence, it will end up calling back
via paravirt_ops to set up various pieces of Xen specific state. One
of the critical things which requires a bit of extra care is the
construction of the initial init_mm pagetable. Because Xen places
tight constraints on pagetables (an active pagetable must always be
valid, and must always be mapped read-only to the guest domain), we
need to be careful when constructing the new pagetable to keep these
constraints in mind. It turns out that the easiest way to do this is
use the initial Xen-provided pagetable as a template, and then just
insert new mappings for memory where a mapping doesn't already exist.
This means that during pagetable setup, it uses a special version of
xen_set_pte which ignores any attempt to remap a read-only page as
read-write (since Xen will map its own initial pagetable as RO), but
lets other changes to the ptes happen, so that things like NX are set
properly.
PRIVILEGED INSTRUCTIONS AND SEGMENTATION
When the kernel runs under Xen, it runs in ring 1 rather than ring 0.
This means that it is more privileged than user-mode in ring 3, but it
still can't run privileged instructions directly. Non-performance
critical instructions are dealt with by taking a privilege exception
and trapping into the hypervisor and emulating the instruction, but
more performance-critical instructions have their own specific
paravirt_ops. In many cases we can avoid having to do any hypercalls
for these instructions, or the Xen implementation is quite different
from the normal native version.
The privileged instructions fall into the broad classes of:
Segmentation: setting up the GDT and the GDT entries, LDT,
TLS and so on. Xen doesn't allow the GDT to be directly
modified; all GDT updates are done via hypercalls where the new
entries can be validated. This is important because Xen uses
segment limits to prevent the guest kernel from damaging the
hypervisor itself.
Traps and exceptions: Xen uses a special format for trap entrypoints,
so when the kernel wants to set an IDT entry, it needs to be
converted to the form Xen expects. Xen sets int 0x80 up specially
so that the trap goes straight from userspace into the guest kernel
without going via the hypervisor. sysenter isn't supported.
Kernel stack: The esp0 entry is extracted from the tss and provided to
Xen.
TLB operations: the various TLB calls are mapped into corresponding
Xen hypercalls.
Control registers: all the control registers are privileged. The most
important is cr3, which points to the base of the current pagetable,
and we handle it specially.
Another instruction we treat specially is CPUID, even though its not
privileged. We want to control what CPU features are visible to the
rest of the kernel, and so CPUID ends up going into a paravirt_op.
Xen implements this mainly to disable the ACPI and APIC subsystems.
INTERRUPT FLAGS
Xen maintains its own separate flag for masking events, which is
contained within the per-cpu vcpu_info structure. Because the guest
kernel runs in ring 1 and not 0, the IF flag in EFLAGS is completely
ignored (and must be, because even if a guest domain disables
interrupts for itself, it can't disable them overall).
(A note on terminology: "events" and interrupts are effectively
synonymous. However, rather than using an "enable flag", Xen uses a
"mask flag", which blocks event delivery when it is non-zero.)
There are paravirt_ops for each of cli/sti/save_fl/restore_fl, which
are implemented to manage the Xen event mask state. The only thing
worth noting is that when events are unmasked, we need to explicitly
see if there's a pending event and call into the hypervisor to make
sure it gets delivered.
UPCALLS
Xen needs a couple of upcall (or callback) functions to be implemented
by each guest. One is the event upcalls, which is how events
(interrupts, effectively) are delivered to the guests. The other is
the failsafe callback, which is used to report errors in either
reloading a segment register, or caused by iret. These are
implemented in i386/kernel/entry.S so they can jump into the normal
iret_exc path when necessary.
MULTICALL BATCHING
Xen provides a multicall mechanism, which allows multiple hypercalls
to be issued at once in order to mitigate the cost of trapping into
the hypervisor. This is particularly useful for context switches,
since the 4-5 hypercalls they would normally need (reload cr3, update
TLS, maybe update LDT) can be reduced to one. This patch implements a
generic batching mechanism for hypercalls, which gets used in many
places in the Xen code.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Chris Wright <chrisw@sous-sol.org>
Cc: Ian Pratt <ian.pratt@xensource.com>
Cc: Christian Limpach <Christian.Limpach@cl.cam.ac.uk>
Cc: Adrian Bunk <bunk@stusta.de>
2007-07-18 01:37:04 +00:00
|
|
|
ENDPROC(xen_hypervisor_callback)
|
|
|
|
|
2015-06-08 07:49:11 +00:00
|
|
|
/*
|
|
|
|
* Hypervisor uses this for application faults while it executes.
|
|
|
|
* We get here for two reasons:
|
|
|
|
* 1. Fault while reloading DS, ES, FS or GS
|
|
|
|
* 2. Fault while executing IRET
|
|
|
|
* Category 1 we fix up by reattempting the load, and zeroing the segment
|
|
|
|
* register if the load fails.
|
|
|
|
* Category 2 we fix up by jumping to do_iret_error. We cannot use the
|
|
|
|
* normal Linux return path in this case because if we use the IRET hypercall
|
|
|
|
* to pop the stack frame we end up in an infinite loop of failsafe callbacks.
|
|
|
|
* We distinguish between categories by maintaining a status value in EAX.
|
|
|
|
*/
|
xen: Core Xen implementation
This patch is a rollup of all the core pieces of the Xen
implementation, including:
- booting and setup
- pagetable setup
- privileged instructions
- segmentation
- interrupt flags
- upcalls
- multicall batching
BOOTING AND SETUP
The vmlinux image is decorated with ELF notes which tell the Xen
domain builder what the kernel's requirements are; the domain builder
then constructs the address space accordingly and starts the kernel.
Xen has its own entrypoint for the kernel (contained in an ELF note).
The ELF notes are set up by xen-head.S, which is included into head.S.
In principle it could be linked separately, but it seems to provoke
lots of binutils bugs.
Because the domain builder starts the kernel in a fairly sane state
(32-bit protected mode, paging enabled, flat segments set up), there's
not a lot of setup needed before starting the kernel proper. The main
steps are:
1. Install the Xen paravirt_ops, which is simply a matter of a
structure assignment.
2. Set init_mm to use the Xen-supplied pagetables (analogous to the
head.S generated pagetables in a native boot).
3. Reserve address space for Xen, since it takes a chunk at the top
of the address space for its own use.
4. Call start_kernel()
PAGETABLE SETUP
Once we hit the main kernel boot sequence, it will end up calling back
via paravirt_ops to set up various pieces of Xen specific state. One
of the critical things which requires a bit of extra care is the
construction of the initial init_mm pagetable. Because Xen places
tight constraints on pagetables (an active pagetable must always be
valid, and must always be mapped read-only to the guest domain), we
need to be careful when constructing the new pagetable to keep these
constraints in mind. It turns out that the easiest way to do this is
use the initial Xen-provided pagetable as a template, and then just
insert new mappings for memory where a mapping doesn't already exist.
This means that during pagetable setup, it uses a special version of
xen_set_pte which ignores any attempt to remap a read-only page as
read-write (since Xen will map its own initial pagetable as RO), but
lets other changes to the ptes happen, so that things like NX are set
properly.
PRIVILEGED INSTRUCTIONS AND SEGMENTATION
When the kernel runs under Xen, it runs in ring 1 rather than ring 0.
This means that it is more privileged than user-mode in ring 3, but it
still can't run privileged instructions directly. Non-performance
critical instructions are dealt with by taking a privilege exception
and trapping into the hypervisor and emulating the instruction, but
more performance-critical instructions have their own specific
paravirt_ops. In many cases we can avoid having to do any hypercalls
for these instructions, or the Xen implementation is quite different
from the normal native version.
The privileged instructions fall into the broad classes of:
Segmentation: setting up the GDT and the GDT entries, LDT,
TLS and so on. Xen doesn't allow the GDT to be directly
modified; all GDT updates are done via hypercalls where the new
entries can be validated. This is important because Xen uses
segment limits to prevent the guest kernel from damaging the
hypervisor itself.
Traps and exceptions: Xen uses a special format for trap entrypoints,
so when the kernel wants to set an IDT entry, it needs to be
converted to the form Xen expects. Xen sets int 0x80 up specially
so that the trap goes straight from userspace into the guest kernel
without going via the hypervisor. sysenter isn't supported.
Kernel stack: The esp0 entry is extracted from the tss and provided to
Xen.
TLB operations: the various TLB calls are mapped into corresponding
Xen hypercalls.
Control registers: all the control registers are privileged. The most
important is cr3, which points to the base of the current pagetable,
and we handle it specially.
Another instruction we treat specially is CPUID, even though its not
privileged. We want to control what CPU features are visible to the
rest of the kernel, and so CPUID ends up going into a paravirt_op.
Xen implements this mainly to disable the ACPI and APIC subsystems.
INTERRUPT FLAGS
Xen maintains its own separate flag for masking events, which is
contained within the per-cpu vcpu_info structure. Because the guest
kernel runs in ring 1 and not 0, the IF flag in EFLAGS is completely
ignored (and must be, because even if a guest domain disables
interrupts for itself, it can't disable them overall).
(A note on terminology: "events" and interrupts are effectively
synonymous. However, rather than using an "enable flag", Xen uses a
"mask flag", which blocks event delivery when it is non-zero.)
There are paravirt_ops for each of cli/sti/save_fl/restore_fl, which
are implemented to manage the Xen event mask state. The only thing
worth noting is that when events are unmasked, we need to explicitly
see if there's a pending event and call into the hypervisor to make
sure it gets delivered.
UPCALLS
Xen needs a couple of upcall (or callback) functions to be implemented
by each guest. One is the event upcalls, which is how events
(interrupts, effectively) are delivered to the guests. The other is
the failsafe callback, which is used to report errors in either
reloading a segment register, or caused by iret. These are
implemented in i386/kernel/entry.S so they can jump into the normal
iret_exc path when necessary.
MULTICALL BATCHING
Xen provides a multicall mechanism, which allows multiple hypercalls
to be issued at once in order to mitigate the cost of trapping into
the hypervisor. This is particularly useful for context switches,
since the 4-5 hypercalls they would normally need (reload cr3, update
TLS, maybe update LDT) can be reduced to one. This patch implements a
generic batching mechanism for hypercalls, which gets used in many
places in the Xen code.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Chris Wright <chrisw@sous-sol.org>
Cc: Ian Pratt <ian.pratt@xensource.com>
Cc: Christian Limpach <Christian.Limpach@cl.cam.ac.uk>
Cc: Adrian Bunk <bunk@stusta.de>
2007-07-18 01:37:04 +00:00
|
|
|
ENTRY(xen_failsafe_callback)
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl %eax
|
|
|
|
movl $1, %eax
|
|
|
|
1: mov 4(%esp), %ds
|
|
|
|
2: mov 8(%esp), %es
|
|
|
|
3: mov 12(%esp), %fs
|
|
|
|
4: mov 16(%esp), %gs
|
2012-10-19 16:29:07 +00:00
|
|
|
/* EAX == 0 => Category 1 (Bad segment)
|
|
|
|
EAX != 0 => Category 2 (Bad IRET) */
|
2015-06-08 07:49:11 +00:00
|
|
|
testl %eax, %eax
|
|
|
|
popl %eax
|
|
|
|
lea 16(%esp), %esp
|
|
|
|
jz 5f
|
|
|
|
jmp iret_exc
|
|
|
|
5: pushl $-1 /* orig_ax = -1 => not a system call */
|
xen: Core Xen implementation
This patch is a rollup of all the core pieces of the Xen
implementation, including:
- booting and setup
- pagetable setup
- privileged instructions
- segmentation
- interrupt flags
- upcalls
- multicall batching
BOOTING AND SETUP
The vmlinux image is decorated with ELF notes which tell the Xen
domain builder what the kernel's requirements are; the domain builder
then constructs the address space accordingly and starts the kernel.
Xen has its own entrypoint for the kernel (contained in an ELF note).
The ELF notes are set up by xen-head.S, which is included into head.S.
In principle it could be linked separately, but it seems to provoke
lots of binutils bugs.
Because the domain builder starts the kernel in a fairly sane state
(32-bit protected mode, paging enabled, flat segments set up), there's
not a lot of setup needed before starting the kernel proper. The main
steps are:
1. Install the Xen paravirt_ops, which is simply a matter of a
structure assignment.
2. Set init_mm to use the Xen-supplied pagetables (analogous to the
head.S generated pagetables in a native boot).
3. Reserve address space for Xen, since it takes a chunk at the top
of the address space for its own use.
4. Call start_kernel()
PAGETABLE SETUP
Once we hit the main kernel boot sequence, it will end up calling back
via paravirt_ops to set up various pieces of Xen specific state. One
of the critical things which requires a bit of extra care is the
construction of the initial init_mm pagetable. Because Xen places
tight constraints on pagetables (an active pagetable must always be
valid, and must always be mapped read-only to the guest domain), we
need to be careful when constructing the new pagetable to keep these
constraints in mind. It turns out that the easiest way to do this is
use the initial Xen-provided pagetable as a template, and then just
insert new mappings for memory where a mapping doesn't already exist.
This means that during pagetable setup, it uses a special version of
xen_set_pte which ignores any attempt to remap a read-only page as
read-write (since Xen will map its own initial pagetable as RO), but
lets other changes to the ptes happen, so that things like NX are set
properly.
PRIVILEGED INSTRUCTIONS AND SEGMENTATION
When the kernel runs under Xen, it runs in ring 1 rather than ring 0.
This means that it is more privileged than user-mode in ring 3, but it
still can't run privileged instructions directly. Non-performance
critical instructions are dealt with by taking a privilege exception
and trapping into the hypervisor and emulating the instruction, but
more performance-critical instructions have their own specific
paravirt_ops. In many cases we can avoid having to do any hypercalls
for these instructions, or the Xen implementation is quite different
from the normal native version.
The privileged instructions fall into the broad classes of:
Segmentation: setting up the GDT and the GDT entries, LDT,
TLS and so on. Xen doesn't allow the GDT to be directly
modified; all GDT updates are done via hypercalls where the new
entries can be validated. This is important because Xen uses
segment limits to prevent the guest kernel from damaging the
hypervisor itself.
Traps and exceptions: Xen uses a special format for trap entrypoints,
so when the kernel wants to set an IDT entry, it needs to be
converted to the form Xen expects. Xen sets int 0x80 up specially
so that the trap goes straight from userspace into the guest kernel
without going via the hypervisor. sysenter isn't supported.
Kernel stack: The esp0 entry is extracted from the tss and provided to
Xen.
TLB operations: the various TLB calls are mapped into corresponding
Xen hypercalls.
Control registers: all the control registers are privileged. The most
important is cr3, which points to the base of the current pagetable,
and we handle it specially.
Another instruction we treat specially is CPUID, even though its not
privileged. We want to control what CPU features are visible to the
rest of the kernel, and so CPUID ends up going into a paravirt_op.
Xen implements this mainly to disable the ACPI and APIC subsystems.
INTERRUPT FLAGS
Xen maintains its own separate flag for masking events, which is
contained within the per-cpu vcpu_info structure. Because the guest
kernel runs in ring 1 and not 0, the IF flag in EFLAGS is completely
ignored (and must be, because even if a guest domain disables
interrupts for itself, it can't disable them overall).
(A note on terminology: "events" and interrupts are effectively
synonymous. However, rather than using an "enable flag", Xen uses a
"mask flag", which blocks event delivery when it is non-zero.)
There are paravirt_ops for each of cli/sti/save_fl/restore_fl, which
are implemented to manage the Xen event mask state. The only thing
worth noting is that when events are unmasked, we need to explicitly
see if there's a pending event and call into the hypervisor to make
sure it gets delivered.
UPCALLS
Xen needs a couple of upcall (or callback) functions to be implemented
by each guest. One is the event upcalls, which is how events
(interrupts, effectively) are delivered to the guests. The other is
the failsafe callback, which is used to report errors in either
reloading a segment register, or caused by iret. These are
implemented in i386/kernel/entry.S so they can jump into the normal
iret_exc path when necessary.
MULTICALL BATCHING
Xen provides a multicall mechanism, which allows multiple hypercalls
to be issued at once in order to mitigate the cost of trapping into
the hypervisor. This is particularly useful for context switches,
since the 4-5 hypercalls they would normally need (reload cr3, update
TLS, maybe update LDT) can be reduced to one. This patch implements a
generic batching mechanism for hypercalls, which gets used in many
places in the Xen code.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Chris Wright <chrisw@sous-sol.org>
Cc: Ian Pratt <ian.pratt@xensource.com>
Cc: Christian Limpach <Christian.Limpach@cl.cam.ac.uk>
Cc: Adrian Bunk <bunk@stusta.de>
2007-07-18 01:37:04 +00:00
|
|
|
SAVE_ALL
|
2016-10-20 16:34:40 +00:00
|
|
|
ENCODE_FRAME_POINTER
|
2015-06-08 07:49:11 +00:00
|
|
|
jmp ret_from_exception
|
|
|
|
|
|
|
|
.section .fixup, "ax"
|
|
|
|
6: xorl %eax, %eax
|
|
|
|
movl %eax, 4(%esp)
|
|
|
|
jmp 1b
|
|
|
|
7: xorl %eax, %eax
|
|
|
|
movl %eax, 8(%esp)
|
|
|
|
jmp 2b
|
|
|
|
8: xorl %eax, %eax
|
|
|
|
movl %eax, 12(%esp)
|
|
|
|
jmp 3b
|
|
|
|
9: xorl %eax, %eax
|
|
|
|
movl %eax, 16(%esp)
|
|
|
|
jmp 4b
|
xen: Core Xen implementation
This patch is a rollup of all the core pieces of the Xen
implementation, including:
- booting and setup
- pagetable setup
- privileged instructions
- segmentation
- interrupt flags
- upcalls
- multicall batching
BOOTING AND SETUP
The vmlinux image is decorated with ELF notes which tell the Xen
domain builder what the kernel's requirements are; the domain builder
then constructs the address space accordingly and starts the kernel.
Xen has its own entrypoint for the kernel (contained in an ELF note).
The ELF notes are set up by xen-head.S, which is included into head.S.
In principle it could be linked separately, but it seems to provoke
lots of binutils bugs.
Because the domain builder starts the kernel in a fairly sane state
(32-bit protected mode, paging enabled, flat segments set up), there's
not a lot of setup needed before starting the kernel proper. The main
steps are:
1. Install the Xen paravirt_ops, which is simply a matter of a
structure assignment.
2. Set init_mm to use the Xen-supplied pagetables (analogous to the
head.S generated pagetables in a native boot).
3. Reserve address space for Xen, since it takes a chunk at the top
of the address space for its own use.
4. Call start_kernel()
PAGETABLE SETUP
Once we hit the main kernel boot sequence, it will end up calling back
via paravirt_ops to set up various pieces of Xen specific state. One
of the critical things which requires a bit of extra care is the
construction of the initial init_mm pagetable. Because Xen places
tight constraints on pagetables (an active pagetable must always be
valid, and must always be mapped read-only to the guest domain), we
need to be careful when constructing the new pagetable to keep these
constraints in mind. It turns out that the easiest way to do this is
use the initial Xen-provided pagetable as a template, and then just
insert new mappings for memory where a mapping doesn't already exist.
This means that during pagetable setup, it uses a special version of
xen_set_pte which ignores any attempt to remap a read-only page as
read-write (since Xen will map its own initial pagetable as RO), but
lets other changes to the ptes happen, so that things like NX are set
properly.
PRIVILEGED INSTRUCTIONS AND SEGMENTATION
When the kernel runs under Xen, it runs in ring 1 rather than ring 0.
This means that it is more privileged than user-mode in ring 3, but it
still can't run privileged instructions directly. Non-performance
critical instructions are dealt with by taking a privilege exception
and trapping into the hypervisor and emulating the instruction, but
more performance-critical instructions have their own specific
paravirt_ops. In many cases we can avoid having to do any hypercalls
for these instructions, or the Xen implementation is quite different
from the normal native version.
The privileged instructions fall into the broad classes of:
Segmentation: setting up the GDT and the GDT entries, LDT,
TLS and so on. Xen doesn't allow the GDT to be directly
modified; all GDT updates are done via hypercalls where the new
entries can be validated. This is important because Xen uses
segment limits to prevent the guest kernel from damaging the
hypervisor itself.
Traps and exceptions: Xen uses a special format for trap entrypoints,
so when the kernel wants to set an IDT entry, it needs to be
converted to the form Xen expects. Xen sets int 0x80 up specially
so that the trap goes straight from userspace into the guest kernel
without going via the hypervisor. sysenter isn't supported.
Kernel stack: The esp0 entry is extracted from the tss and provided to
Xen.
TLB operations: the various TLB calls are mapped into corresponding
Xen hypercalls.
Control registers: all the control registers are privileged. The most
important is cr3, which points to the base of the current pagetable,
and we handle it specially.
Another instruction we treat specially is CPUID, even though its not
privileged. We want to control what CPU features are visible to the
rest of the kernel, and so CPUID ends up going into a paravirt_op.
Xen implements this mainly to disable the ACPI and APIC subsystems.
INTERRUPT FLAGS
Xen maintains its own separate flag for masking events, which is
contained within the per-cpu vcpu_info structure. Because the guest
kernel runs in ring 1 and not 0, the IF flag in EFLAGS is completely
ignored (and must be, because even if a guest domain disables
interrupts for itself, it can't disable them overall).
(A note on terminology: "events" and interrupts are effectively
synonymous. However, rather than using an "enable flag", Xen uses a
"mask flag", which blocks event delivery when it is non-zero.)
There are paravirt_ops for each of cli/sti/save_fl/restore_fl, which
are implemented to manage the Xen event mask state. The only thing
worth noting is that when events are unmasked, we need to explicitly
see if there's a pending event and call into the hypervisor to make
sure it gets delivered.
UPCALLS
Xen needs a couple of upcall (or callback) functions to be implemented
by each guest. One is the event upcalls, which is how events
(interrupts, effectively) are delivered to the guests. The other is
the failsafe callback, which is used to report errors in either
reloading a segment register, or caused by iret. These are
implemented in i386/kernel/entry.S so they can jump into the normal
iret_exc path when necessary.
MULTICALL BATCHING
Xen provides a multicall mechanism, which allows multiple hypercalls
to be issued at once in order to mitigate the cost of trapping into
the hypervisor. This is particularly useful for context switches,
since the 4-5 hypercalls they would normally need (reload cr3, update
TLS, maybe update LDT) can be reduced to one. This patch implements a
generic batching mechanism for hypercalls, which gets used in many
places in the Xen code.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Chris Wright <chrisw@sous-sol.org>
Cc: Ian Pratt <ian.pratt@xensource.com>
Cc: Christian Limpach <Christian.Limpach@cl.cam.ac.uk>
Cc: Adrian Bunk <bunk@stusta.de>
2007-07-18 01:37:04 +00:00
|
|
|
.previous
|
2015-06-08 07:49:11 +00:00
|
|
|
_ASM_EXTABLE(1b, 6b)
|
|
|
|
_ASM_EXTABLE(2b, 7b)
|
|
|
|
_ASM_EXTABLE(3b, 8b)
|
|
|
|
_ASM_EXTABLE(4b, 9b)
|
xen: Core Xen implementation
This patch is a rollup of all the core pieces of the Xen
implementation, including:
- booting and setup
- pagetable setup
- privileged instructions
- segmentation
- interrupt flags
- upcalls
- multicall batching
BOOTING AND SETUP
The vmlinux image is decorated with ELF notes which tell the Xen
domain builder what the kernel's requirements are; the domain builder
then constructs the address space accordingly and starts the kernel.
Xen has its own entrypoint for the kernel (contained in an ELF note).
The ELF notes are set up by xen-head.S, which is included into head.S.
In principle it could be linked separately, but it seems to provoke
lots of binutils bugs.
Because the domain builder starts the kernel in a fairly sane state
(32-bit protected mode, paging enabled, flat segments set up), there's
not a lot of setup needed before starting the kernel proper. The main
steps are:
1. Install the Xen paravirt_ops, which is simply a matter of a
structure assignment.
2. Set init_mm to use the Xen-supplied pagetables (analogous to the
head.S generated pagetables in a native boot).
3. Reserve address space for Xen, since it takes a chunk at the top
of the address space for its own use.
4. Call start_kernel()
PAGETABLE SETUP
Once we hit the main kernel boot sequence, it will end up calling back
via paravirt_ops to set up various pieces of Xen specific state. One
of the critical things which requires a bit of extra care is the
construction of the initial init_mm pagetable. Because Xen places
tight constraints on pagetables (an active pagetable must always be
valid, and must always be mapped read-only to the guest domain), we
need to be careful when constructing the new pagetable to keep these
constraints in mind. It turns out that the easiest way to do this is
use the initial Xen-provided pagetable as a template, and then just
insert new mappings for memory where a mapping doesn't already exist.
This means that during pagetable setup, it uses a special version of
xen_set_pte which ignores any attempt to remap a read-only page as
read-write (since Xen will map its own initial pagetable as RO), but
lets other changes to the ptes happen, so that things like NX are set
properly.
PRIVILEGED INSTRUCTIONS AND SEGMENTATION
When the kernel runs under Xen, it runs in ring 1 rather than ring 0.
This means that it is more privileged than user-mode in ring 3, but it
still can't run privileged instructions directly. Non-performance
critical instructions are dealt with by taking a privilege exception
and trapping into the hypervisor and emulating the instruction, but
more performance-critical instructions have their own specific
paravirt_ops. In many cases we can avoid having to do any hypercalls
for these instructions, or the Xen implementation is quite different
from the normal native version.
The privileged instructions fall into the broad classes of:
Segmentation: setting up the GDT and the GDT entries, LDT,
TLS and so on. Xen doesn't allow the GDT to be directly
modified; all GDT updates are done via hypercalls where the new
entries can be validated. This is important because Xen uses
segment limits to prevent the guest kernel from damaging the
hypervisor itself.
Traps and exceptions: Xen uses a special format for trap entrypoints,
so when the kernel wants to set an IDT entry, it needs to be
converted to the form Xen expects. Xen sets int 0x80 up specially
so that the trap goes straight from userspace into the guest kernel
without going via the hypervisor. sysenter isn't supported.
Kernel stack: The esp0 entry is extracted from the tss and provided to
Xen.
TLB operations: the various TLB calls are mapped into corresponding
Xen hypercalls.
Control registers: all the control registers are privileged. The most
important is cr3, which points to the base of the current pagetable,
and we handle it specially.
Another instruction we treat specially is CPUID, even though its not
privileged. We want to control what CPU features are visible to the
rest of the kernel, and so CPUID ends up going into a paravirt_op.
Xen implements this mainly to disable the ACPI and APIC subsystems.
INTERRUPT FLAGS
Xen maintains its own separate flag for masking events, which is
contained within the per-cpu vcpu_info structure. Because the guest
kernel runs in ring 1 and not 0, the IF flag in EFLAGS is completely
ignored (and must be, because even if a guest domain disables
interrupts for itself, it can't disable them overall).
(A note on terminology: "events" and interrupts are effectively
synonymous. However, rather than using an "enable flag", Xen uses a
"mask flag", which blocks event delivery when it is non-zero.)
There are paravirt_ops for each of cli/sti/save_fl/restore_fl, which
are implemented to manage the Xen event mask state. The only thing
worth noting is that when events are unmasked, we need to explicitly
see if there's a pending event and call into the hypervisor to make
sure it gets delivered.
UPCALLS
Xen needs a couple of upcall (or callback) functions to be implemented
by each guest. One is the event upcalls, which is how events
(interrupts, effectively) are delivered to the guests. The other is
the failsafe callback, which is used to report errors in either
reloading a segment register, or caused by iret. These are
implemented in i386/kernel/entry.S so they can jump into the normal
iret_exc path when necessary.
MULTICALL BATCHING
Xen provides a multicall mechanism, which allows multiple hypercalls
to be issued at once in order to mitigate the cost of trapping into
the hypervisor. This is particularly useful for context switches,
since the 4-5 hypercalls they would normally need (reload cr3, update
TLS, maybe update LDT) can be reduced to one. This patch implements a
generic batching mechanism for hypercalls, which gets used in many
places in the Xen code.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Chris Wright <chrisw@sous-sol.org>
Cc: Ian Pratt <ian.pratt@xensource.com>
Cc: Christian Limpach <Christian.Limpach@cl.cam.ac.uk>
Cc: Adrian Bunk <bunk@stusta.de>
2007-07-18 01:37:04 +00:00
|
|
|
ENDPROC(xen_failsafe_callback)
|
|
|
|
|
2013-02-04 01:22:39 +00:00
|
|
|
BUILD_INTERRUPT3(xen_hvm_callback_vector, HYPERVISOR_CALLBACK_VECTOR,
|
2017-08-28 06:47:31 +00:00
|
|
|
xen_evtchn_do_upcall)
|
2010-05-14 11:40:51 +00:00
|
|
|
|
2015-06-08 07:49:11 +00:00
|
|
|
#endif /* CONFIG_XEN */
|
2013-02-04 01:22:39 +00:00
|
|
|
|
|
|
|
#if IS_ENABLED(CONFIG_HYPERV)
|
|
|
|
|
|
|
|
BUILD_INTERRUPT3(hyperv_callback_vector, HYPERVISOR_CALLBACK_VECTOR,
|
2017-08-28 06:47:31 +00:00
|
|
|
hyperv_vector_handler)
|
2013-02-04 01:22:39 +00:00
|
|
|
|
2018-01-24 13:23:33 +00:00
|
|
|
BUILD_INTERRUPT3(hyperv_reenlightenment_vector, HYPERV_REENLIGHTENMENT_VECTOR,
|
|
|
|
hyperv_reenlightenment_intr)
|
|
|
|
|
2013-02-04 01:22:39 +00:00
|
|
|
#endif /* CONFIG_HYPERV */
|
xen: Core Xen implementation
This patch is a rollup of all the core pieces of the Xen
implementation, including:
- booting and setup
- pagetable setup
- privileged instructions
- segmentation
- interrupt flags
- upcalls
- multicall batching
BOOTING AND SETUP
The vmlinux image is decorated with ELF notes which tell the Xen
domain builder what the kernel's requirements are; the domain builder
then constructs the address space accordingly and starts the kernel.
Xen has its own entrypoint for the kernel (contained in an ELF note).
The ELF notes are set up by xen-head.S, which is included into head.S.
In principle it could be linked separately, but it seems to provoke
lots of binutils bugs.
Because the domain builder starts the kernel in a fairly sane state
(32-bit protected mode, paging enabled, flat segments set up), there's
not a lot of setup needed before starting the kernel proper. The main
steps are:
1. Install the Xen paravirt_ops, which is simply a matter of a
structure assignment.
2. Set init_mm to use the Xen-supplied pagetables (analogous to the
head.S generated pagetables in a native boot).
3. Reserve address space for Xen, since it takes a chunk at the top
of the address space for its own use.
4. Call start_kernel()
PAGETABLE SETUP
Once we hit the main kernel boot sequence, it will end up calling back
via paravirt_ops to set up various pieces of Xen specific state. One
of the critical things which requires a bit of extra care is the
construction of the initial init_mm pagetable. Because Xen places
tight constraints on pagetables (an active pagetable must always be
valid, and must always be mapped read-only to the guest domain), we
need to be careful when constructing the new pagetable to keep these
constraints in mind. It turns out that the easiest way to do this is
use the initial Xen-provided pagetable as a template, and then just
insert new mappings for memory where a mapping doesn't already exist.
This means that during pagetable setup, it uses a special version of
xen_set_pte which ignores any attempt to remap a read-only page as
read-write (since Xen will map its own initial pagetable as RO), but
lets other changes to the ptes happen, so that things like NX are set
properly.
PRIVILEGED INSTRUCTIONS AND SEGMENTATION
When the kernel runs under Xen, it runs in ring 1 rather than ring 0.
This means that it is more privileged than user-mode in ring 3, but it
still can't run privileged instructions directly. Non-performance
critical instructions are dealt with by taking a privilege exception
and trapping into the hypervisor and emulating the instruction, but
more performance-critical instructions have their own specific
paravirt_ops. In many cases we can avoid having to do any hypercalls
for these instructions, or the Xen implementation is quite different
from the normal native version.
The privileged instructions fall into the broad classes of:
Segmentation: setting up the GDT and the GDT entries, LDT,
TLS and so on. Xen doesn't allow the GDT to be directly
modified; all GDT updates are done via hypercalls where the new
entries can be validated. This is important because Xen uses
segment limits to prevent the guest kernel from damaging the
hypervisor itself.
Traps and exceptions: Xen uses a special format for trap entrypoints,
so when the kernel wants to set an IDT entry, it needs to be
converted to the form Xen expects. Xen sets int 0x80 up specially
so that the trap goes straight from userspace into the guest kernel
without going via the hypervisor. sysenter isn't supported.
Kernel stack: The esp0 entry is extracted from the tss and provided to
Xen.
TLB operations: the various TLB calls are mapped into corresponding
Xen hypercalls.
Control registers: all the control registers are privileged. The most
important is cr3, which points to the base of the current pagetable,
and we handle it specially.
Another instruction we treat specially is CPUID, even though its not
privileged. We want to control what CPU features are visible to the
rest of the kernel, and so CPUID ends up going into a paravirt_op.
Xen implements this mainly to disable the ACPI and APIC subsystems.
INTERRUPT FLAGS
Xen maintains its own separate flag for masking events, which is
contained within the per-cpu vcpu_info structure. Because the guest
kernel runs in ring 1 and not 0, the IF flag in EFLAGS is completely
ignored (and must be, because even if a guest domain disables
interrupts for itself, it can't disable them overall).
(A note on terminology: "events" and interrupts are effectively
synonymous. However, rather than using an "enable flag", Xen uses a
"mask flag", which blocks event delivery when it is non-zero.)
There are paravirt_ops for each of cli/sti/save_fl/restore_fl, which
are implemented to manage the Xen event mask state. The only thing
worth noting is that when events are unmasked, we need to explicitly
see if there's a pending event and call into the hypervisor to make
sure it gets delivered.
UPCALLS
Xen needs a couple of upcall (or callback) functions to be implemented
by each guest. One is the event upcalls, which is how events
(interrupts, effectively) are delivered to the guests. The other is
the failsafe callback, which is used to report errors in either
reloading a segment register, or caused by iret. These are
implemented in i386/kernel/entry.S so they can jump into the normal
iret_exc path when necessary.
MULTICALL BATCHING
Xen provides a multicall mechanism, which allows multiple hypercalls
to be issued at once in order to mitigate the cost of trapping into
the hypervisor. This is particularly useful for context switches,
since the 4-5 hypercalls they would normally need (reload cr3, update
TLS, maybe update LDT) can be reduced to one. This patch implements a
generic batching mechanism for hypercalls, which gets used in many
places in the Xen code.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
Signed-off-by: Chris Wright <chrisw@sous-sol.org>
Cc: Ian Pratt <ian.pratt@xensource.com>
Cc: Christian Limpach <Christian.Limpach@cl.cam.ac.uk>
Cc: Adrian Bunk <bunk@stusta.de>
2007-07-18 01:37:04 +00:00
|
|
|
|
2008-11-24 14:38:45 +00:00
|
|
|
ENTRY(page_fault)
|
2012-09-21 20:58:10 +00:00
|
|
|
ASM_CLAC
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl $do_page_fault
|
2008-11-24 14:38:45 +00:00
|
|
|
ALIGN
|
2016-09-21 21:04:00 +00:00
|
|
|
jmp common_exception
|
|
|
|
END(page_fault)
|
|
|
|
|
|
|
|
common_exception:
|
2009-02-09 13:17:40 +00:00
|
|
|
/* the function address is in %gs's slot on the stack */
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl %fs
|
|
|
|
pushl %es
|
|
|
|
pushl %ds
|
|
|
|
pushl %eax
|
|
|
|
pushl %ebp
|
|
|
|
pushl %edi
|
|
|
|
pushl %esi
|
|
|
|
pushl %edx
|
|
|
|
pushl %ecx
|
|
|
|
pushl %ebx
|
2016-10-20 16:34:40 +00:00
|
|
|
ENCODE_FRAME_POINTER
|
2008-11-24 14:38:45 +00:00
|
|
|
cld
|
2015-06-08 07:49:11 +00:00
|
|
|
movl $(__KERNEL_PERCPU), %ecx
|
|
|
|
movl %ecx, %fs
|
2008-11-24 14:38:45 +00:00
|
|
|
UNWIND_ESPFIX_STACK
|
2009-02-09 13:17:40 +00:00
|
|
|
GS_TO_REG %ecx
|
2015-06-08 07:49:11 +00:00
|
|
|
movl PT_GS(%esp), %edi # get the function address
|
|
|
|
movl PT_ORIG_EAX(%esp), %edx # get the error code
|
|
|
|
movl $-1, PT_ORIG_EAX(%esp) # no syscall to restart
|
2009-02-09 13:17:40 +00:00
|
|
|
REG_TO_PTGS %ecx
|
|
|
|
SET_KERNEL_GS %ecx
|
2015-06-08 07:49:11 +00:00
|
|
|
movl $(__USER_DS), %ecx
|
|
|
|
movl %ecx, %ds
|
|
|
|
movl %ecx, %es
|
2008-11-24 14:38:45 +00:00
|
|
|
TRACE_IRQS_OFF
|
2015-06-08 07:49:11 +00:00
|
|
|
movl %esp, %eax # pt_regs pointer
|
2018-01-11 21:46:28 +00:00
|
|
|
CALL_NOSPEC %edi
|
2015-06-08 07:49:11 +00:00
|
|
|
jmp ret_from_exception
|
2016-09-21 21:04:00 +00:00
|
|
|
END(common_exception)
|
2008-11-24 14:38:45 +00:00
|
|
|
|
|
|
|
ENTRY(debug)
|
x86/entry/32: Simplify and fix up the SYSENTER stack #DB/NMI fixup
Right after SYSENTER, we can get a #DB or NMI. On x86_32, there's no IST,
so the exception handler is invoked on the temporary SYSENTER stack.
Because the SYSENTER stack is very small, we have a fixup to switch
off the stack quickly when this happens. The old fixup had several issues:
1. It checked the interrupt frame's CS and EIP. This wasn't
obviously correct on Xen or if vm86 mode was in use [1].
2. In the NMI handler, it did some frightening digging into the
stack frame. I'm not convinced this digging was correct.
3. The fixup didn't switch stacks and then switch back. Instead, it
synthesized a brand new stack frame that would redirect the IRET
back to the SYSENTER code. That frame was highly questionable.
For one thing, if NMI nested inside #DB, we would effectively
abort the #DB prologue, which was probably safe but was
frightening. For another, the code used PUSHFL to write the
FLAGS portion of the frame, which was simply bogus -- by the time
PUSHFL was called, at least TF, NT, VM, and all of the arithmetic
flags were clobbered.
Simplify this considerably. Instead of looking at the saved frame
to see where we came from, check the hardware ESP register against
the SYSENTER stack directly. Malicious user code cannot spoof the
kernel ESP register, and by moving the check after SAVE_ALL, we can
use normal PER_CPU accesses to find all the relevant addresses.
With this patch applied, the improved syscall_nt_32 test finally
passes on 32-bit kernels.
[1] It isn't obviously correct, but it is nonetheless safe from vm86
shenanigans as far as I can tell. A user can't point EIP at
entry_SYSENTER_32 while in vm86 mode because entry_SYSENTER_32,
like all kernel addresses, is greater than 0xffff and would thus
violate the CS segment limit.
Signed-off-by: Andy Lutomirski <luto@kernel.org>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/b2cdbc037031c07ecf2c40a96069318aec0e7971.1457578375.git.luto@kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2016-03-10 03:00:32 +00:00
|
|
|
/*
|
|
|
|
* #DB can happen at the first instruction of
|
|
|
|
* entry_SYSENTER_32 or in Xen's SYSENTER prologue. If this
|
|
|
|
* happens, then we will be running on a very small stack. We
|
|
|
|
* need to detect this condition and switch to the thread
|
|
|
|
* stack before calling any C code at all.
|
|
|
|
*
|
|
|
|
* If you edit this code, keep in mind that NMIs can happen in here.
|
|
|
|
*/
|
2012-09-21 20:58:10 +00:00
|
|
|
ASM_CLAC
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl $-1 # mark this as an int
|
2008-11-24 14:38:45 +00:00
|
|
|
SAVE_ALL
|
2016-10-20 16:34:40 +00:00
|
|
|
ENCODE_FRAME_POINTER
|
2015-06-08 07:49:11 +00:00
|
|
|
xorl %edx, %edx # error code 0
|
|
|
|
movl %esp, %eax # pt_regs pointer
|
x86/entry/32: Simplify and fix up the SYSENTER stack #DB/NMI fixup
Right after SYSENTER, we can get a #DB or NMI. On x86_32, there's no IST,
so the exception handler is invoked on the temporary SYSENTER stack.
Because the SYSENTER stack is very small, we have a fixup to switch
off the stack quickly when this happens. The old fixup had several issues:
1. It checked the interrupt frame's CS and EIP. This wasn't
obviously correct on Xen or if vm86 mode was in use [1].
2. In the NMI handler, it did some frightening digging into the
stack frame. I'm not convinced this digging was correct.
3. The fixup didn't switch stacks and then switch back. Instead, it
synthesized a brand new stack frame that would redirect the IRET
back to the SYSENTER code. That frame was highly questionable.
For one thing, if NMI nested inside #DB, we would effectively
abort the #DB prologue, which was probably safe but was
frightening. For another, the code used PUSHFL to write the
FLAGS portion of the frame, which was simply bogus -- by the time
PUSHFL was called, at least TF, NT, VM, and all of the arithmetic
flags were clobbered.
Simplify this considerably. Instead of looking at the saved frame
to see where we came from, check the hardware ESP register against
the SYSENTER stack directly. Malicious user code cannot spoof the
kernel ESP register, and by moving the check after SAVE_ALL, we can
use normal PER_CPU accesses to find all the relevant addresses.
With this patch applied, the improved syscall_nt_32 test finally
passes on 32-bit kernels.
[1] It isn't obviously correct, but it is nonetheless safe from vm86
shenanigans as far as I can tell. A user can't point EIP at
entry_SYSENTER_32 while in vm86 mode because entry_SYSENTER_32,
like all kernel addresses, is greater than 0xffff and would thus
violate the CS segment limit.
Signed-off-by: Andy Lutomirski <luto@kernel.org>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/b2cdbc037031c07ecf2c40a96069318aec0e7971.1457578375.git.luto@kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2016-03-10 03:00:32 +00:00
|
|
|
|
|
|
|
/* Are we currently on the SYSENTER stack? */
|
2017-12-04 14:07:20 +00:00
|
|
|
movl PER_CPU_VAR(cpu_entry_area), %ecx
|
2017-12-05 01:25:07 +00:00
|
|
|
addl $CPU_ENTRY_AREA_entry_stack + SIZEOF_entry_stack, %ecx
|
|
|
|
subl %eax, %ecx /* ecx = (end of entry_stack) - esp */
|
|
|
|
cmpl $SIZEOF_entry_stack, %ecx
|
x86/entry/32: Simplify and fix up the SYSENTER stack #DB/NMI fixup
Right after SYSENTER, we can get a #DB or NMI. On x86_32, there's no IST,
so the exception handler is invoked on the temporary SYSENTER stack.
Because the SYSENTER stack is very small, we have a fixup to switch
off the stack quickly when this happens. The old fixup had several issues:
1. It checked the interrupt frame's CS and EIP. This wasn't
obviously correct on Xen or if vm86 mode was in use [1].
2. In the NMI handler, it did some frightening digging into the
stack frame. I'm not convinced this digging was correct.
3. The fixup didn't switch stacks and then switch back. Instead, it
synthesized a brand new stack frame that would redirect the IRET
back to the SYSENTER code. That frame was highly questionable.
For one thing, if NMI nested inside #DB, we would effectively
abort the #DB prologue, which was probably safe but was
frightening. For another, the code used PUSHFL to write the
FLAGS portion of the frame, which was simply bogus -- by the time
PUSHFL was called, at least TF, NT, VM, and all of the arithmetic
flags were clobbered.
Simplify this considerably. Instead of looking at the saved frame
to see where we came from, check the hardware ESP register against
the SYSENTER stack directly. Malicious user code cannot spoof the
kernel ESP register, and by moving the check after SAVE_ALL, we can
use normal PER_CPU accesses to find all the relevant addresses.
With this patch applied, the improved syscall_nt_32 test finally
passes on 32-bit kernels.
[1] It isn't obviously correct, but it is nonetheless safe from vm86
shenanigans as far as I can tell. A user can't point EIP at
entry_SYSENTER_32 while in vm86 mode because entry_SYSENTER_32,
like all kernel addresses, is greater than 0xffff and would thus
violate the CS segment limit.
Signed-off-by: Andy Lutomirski <luto@kernel.org>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/b2cdbc037031c07ecf2c40a96069318aec0e7971.1457578375.git.luto@kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2016-03-10 03:00:32 +00:00
|
|
|
jb .Ldebug_from_sysenter_stack
|
|
|
|
|
|
|
|
TRACE_IRQS_OFF
|
|
|
|
call do_debug
|
|
|
|
jmp ret_from_exception
|
|
|
|
|
|
|
|
.Ldebug_from_sysenter_stack:
|
|
|
|
/* We're on the SYSENTER stack. Switch off. */
|
2016-10-20 16:34:40 +00:00
|
|
|
movl %esp, %ebx
|
x86/entry/32: Simplify and fix up the SYSENTER stack #DB/NMI fixup
Right after SYSENTER, we can get a #DB or NMI. On x86_32, there's no IST,
so the exception handler is invoked on the temporary SYSENTER stack.
Because the SYSENTER stack is very small, we have a fixup to switch
off the stack quickly when this happens. The old fixup had several issues:
1. It checked the interrupt frame's CS and EIP. This wasn't
obviously correct on Xen or if vm86 mode was in use [1].
2. In the NMI handler, it did some frightening digging into the
stack frame. I'm not convinced this digging was correct.
3. The fixup didn't switch stacks and then switch back. Instead, it
synthesized a brand new stack frame that would redirect the IRET
back to the SYSENTER code. That frame was highly questionable.
For one thing, if NMI nested inside #DB, we would effectively
abort the #DB prologue, which was probably safe but was
frightening. For another, the code used PUSHFL to write the
FLAGS portion of the frame, which was simply bogus -- by the time
PUSHFL was called, at least TF, NT, VM, and all of the arithmetic
flags were clobbered.
Simplify this considerably. Instead of looking at the saved frame
to see where we came from, check the hardware ESP register against
the SYSENTER stack directly. Malicious user code cannot spoof the
kernel ESP register, and by moving the check after SAVE_ALL, we can
use normal PER_CPU accesses to find all the relevant addresses.
With this patch applied, the improved syscall_nt_32 test finally
passes on 32-bit kernels.
[1] It isn't obviously correct, but it is nonetheless safe from vm86
shenanigans as far as I can tell. A user can't point EIP at
entry_SYSENTER_32 while in vm86 mode because entry_SYSENTER_32,
like all kernel addresses, is greater than 0xffff and would thus
violate the CS segment limit.
Signed-off-by: Andy Lutomirski <luto@kernel.org>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/b2cdbc037031c07ecf2c40a96069318aec0e7971.1457578375.git.luto@kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2016-03-10 03:00:32 +00:00
|
|
|
movl PER_CPU_VAR(cpu_current_top_of_stack), %esp
|
|
|
|
TRACE_IRQS_OFF
|
2015-06-08 07:49:11 +00:00
|
|
|
call do_debug
|
2016-10-20 16:34:40 +00:00
|
|
|
movl %ebx, %esp
|
2015-06-08 07:49:11 +00:00
|
|
|
jmp ret_from_exception
|
2008-11-24 14:38:45 +00:00
|
|
|
END(debug)
|
|
|
|
|
|
|
|
/*
|
x86/entry/32: Simplify and fix up the SYSENTER stack #DB/NMI fixup
Right after SYSENTER, we can get a #DB or NMI. On x86_32, there's no IST,
so the exception handler is invoked on the temporary SYSENTER stack.
Because the SYSENTER stack is very small, we have a fixup to switch
off the stack quickly when this happens. The old fixup had several issues:
1. It checked the interrupt frame's CS and EIP. This wasn't
obviously correct on Xen or if vm86 mode was in use [1].
2. In the NMI handler, it did some frightening digging into the
stack frame. I'm not convinced this digging was correct.
3. The fixup didn't switch stacks and then switch back. Instead, it
synthesized a brand new stack frame that would redirect the IRET
back to the SYSENTER code. That frame was highly questionable.
For one thing, if NMI nested inside #DB, we would effectively
abort the #DB prologue, which was probably safe but was
frightening. For another, the code used PUSHFL to write the
FLAGS portion of the frame, which was simply bogus -- by the time
PUSHFL was called, at least TF, NT, VM, and all of the arithmetic
flags were clobbered.
Simplify this considerably. Instead of looking at the saved frame
to see where we came from, check the hardware ESP register against
the SYSENTER stack directly. Malicious user code cannot spoof the
kernel ESP register, and by moving the check after SAVE_ALL, we can
use normal PER_CPU accesses to find all the relevant addresses.
With this patch applied, the improved syscall_nt_32 test finally
passes on 32-bit kernels.
[1] It isn't obviously correct, but it is nonetheless safe from vm86
shenanigans as far as I can tell. A user can't point EIP at
entry_SYSENTER_32 while in vm86 mode because entry_SYSENTER_32,
like all kernel addresses, is greater than 0xffff and would thus
violate the CS segment limit.
Signed-off-by: Andy Lutomirski <luto@kernel.org>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/b2cdbc037031c07ecf2c40a96069318aec0e7971.1457578375.git.luto@kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2016-03-10 03:00:32 +00:00
|
|
|
* NMI is doubly nasty. It can happen on the first instruction of
|
|
|
|
* entry_SYSENTER_32 (just like #DB), but it can also interrupt the beginning
|
|
|
|
* of the #DB handler even if that #DB in turn hit before entry_SYSENTER_32
|
|
|
|
* switched stacks. We handle both conditions by simply checking whether we
|
|
|
|
* interrupted kernel code running on the SYSENTER stack.
|
2008-11-24 14:38:45 +00:00
|
|
|
*/
|
|
|
|
ENTRY(nmi)
|
2012-09-21 20:58:10 +00:00
|
|
|
ASM_CLAC
|
2014-05-04 17:36:22 +00:00
|
|
|
#ifdef CONFIG_X86_ESPFIX32
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl %eax
|
|
|
|
movl %ss, %eax
|
|
|
|
cmpw $__ESPFIX_SS, %ax
|
|
|
|
popl %eax
|
2016-09-21 21:03:59 +00:00
|
|
|
je .Lnmi_espfix_stack
|
2014-05-04 17:36:22 +00:00
|
|
|
#endif
|
x86/entry/32: Simplify and fix up the SYSENTER stack #DB/NMI fixup
Right after SYSENTER, we can get a #DB or NMI. On x86_32, there's no IST,
so the exception handler is invoked on the temporary SYSENTER stack.
Because the SYSENTER stack is very small, we have a fixup to switch
off the stack quickly when this happens. The old fixup had several issues:
1. It checked the interrupt frame's CS and EIP. This wasn't
obviously correct on Xen or if vm86 mode was in use [1].
2. In the NMI handler, it did some frightening digging into the
stack frame. I'm not convinced this digging was correct.
3. The fixup didn't switch stacks and then switch back. Instead, it
synthesized a brand new stack frame that would redirect the IRET
back to the SYSENTER code. That frame was highly questionable.
For one thing, if NMI nested inside #DB, we would effectively
abort the #DB prologue, which was probably safe but was
frightening. For another, the code used PUSHFL to write the
FLAGS portion of the frame, which was simply bogus -- by the time
PUSHFL was called, at least TF, NT, VM, and all of the arithmetic
flags were clobbered.
Simplify this considerably. Instead of looking at the saved frame
to see where we came from, check the hardware ESP register against
the SYSENTER stack directly. Malicious user code cannot spoof the
kernel ESP register, and by moving the check after SAVE_ALL, we can
use normal PER_CPU accesses to find all the relevant addresses.
With this patch applied, the improved syscall_nt_32 test finally
passes on 32-bit kernels.
[1] It isn't obviously correct, but it is nonetheless safe from vm86
shenanigans as far as I can tell. A user can't point EIP at
entry_SYSENTER_32 while in vm86 mode because entry_SYSENTER_32,
like all kernel addresses, is greater than 0xffff and would thus
violate the CS segment limit.
Signed-off-by: Andy Lutomirski <luto@kernel.org>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/b2cdbc037031c07ecf2c40a96069318aec0e7971.1457578375.git.luto@kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2016-03-10 03:00:32 +00:00
|
|
|
|
|
|
|
pushl %eax # pt_regs->orig_ax
|
2008-11-24 14:38:45 +00:00
|
|
|
SAVE_ALL
|
2016-10-20 16:34:40 +00:00
|
|
|
ENCODE_FRAME_POINTER
|
2015-06-08 07:49:11 +00:00
|
|
|
xorl %edx, %edx # zero error code
|
|
|
|
movl %esp, %eax # pt_regs pointer
|
x86/entry/32: Simplify and fix up the SYSENTER stack #DB/NMI fixup
Right after SYSENTER, we can get a #DB or NMI. On x86_32, there's no IST,
so the exception handler is invoked on the temporary SYSENTER stack.
Because the SYSENTER stack is very small, we have a fixup to switch
off the stack quickly when this happens. The old fixup had several issues:
1. It checked the interrupt frame's CS and EIP. This wasn't
obviously correct on Xen or if vm86 mode was in use [1].
2. In the NMI handler, it did some frightening digging into the
stack frame. I'm not convinced this digging was correct.
3. The fixup didn't switch stacks and then switch back. Instead, it
synthesized a brand new stack frame that would redirect the IRET
back to the SYSENTER code. That frame was highly questionable.
For one thing, if NMI nested inside #DB, we would effectively
abort the #DB prologue, which was probably safe but was
frightening. For another, the code used PUSHFL to write the
FLAGS portion of the frame, which was simply bogus -- by the time
PUSHFL was called, at least TF, NT, VM, and all of the arithmetic
flags were clobbered.
Simplify this considerably. Instead of looking at the saved frame
to see where we came from, check the hardware ESP register against
the SYSENTER stack directly. Malicious user code cannot spoof the
kernel ESP register, and by moving the check after SAVE_ALL, we can
use normal PER_CPU accesses to find all the relevant addresses.
With this patch applied, the improved syscall_nt_32 test finally
passes on 32-bit kernels.
[1] It isn't obviously correct, but it is nonetheless safe from vm86
shenanigans as far as I can tell. A user can't point EIP at
entry_SYSENTER_32 while in vm86 mode because entry_SYSENTER_32,
like all kernel addresses, is greater than 0xffff and would thus
violate the CS segment limit.
Signed-off-by: Andy Lutomirski <luto@kernel.org>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/b2cdbc037031c07ecf2c40a96069318aec0e7971.1457578375.git.luto@kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2016-03-10 03:00:32 +00:00
|
|
|
|
|
|
|
/* Are we currently on the SYSENTER stack? */
|
2017-12-04 14:07:20 +00:00
|
|
|
movl PER_CPU_VAR(cpu_entry_area), %ecx
|
2017-12-05 01:25:07 +00:00
|
|
|
addl $CPU_ENTRY_AREA_entry_stack + SIZEOF_entry_stack, %ecx
|
|
|
|
subl %eax, %ecx /* ecx = (end of entry_stack) - esp */
|
|
|
|
cmpl $SIZEOF_entry_stack, %ecx
|
x86/entry/32: Simplify and fix up the SYSENTER stack #DB/NMI fixup
Right after SYSENTER, we can get a #DB or NMI. On x86_32, there's no IST,
so the exception handler is invoked on the temporary SYSENTER stack.
Because the SYSENTER stack is very small, we have a fixup to switch
off the stack quickly when this happens. The old fixup had several issues:
1. It checked the interrupt frame's CS and EIP. This wasn't
obviously correct on Xen or if vm86 mode was in use [1].
2. In the NMI handler, it did some frightening digging into the
stack frame. I'm not convinced this digging was correct.
3. The fixup didn't switch stacks and then switch back. Instead, it
synthesized a brand new stack frame that would redirect the IRET
back to the SYSENTER code. That frame was highly questionable.
For one thing, if NMI nested inside #DB, we would effectively
abort the #DB prologue, which was probably safe but was
frightening. For another, the code used PUSHFL to write the
FLAGS portion of the frame, which was simply bogus -- by the time
PUSHFL was called, at least TF, NT, VM, and all of the arithmetic
flags were clobbered.
Simplify this considerably. Instead of looking at the saved frame
to see where we came from, check the hardware ESP register against
the SYSENTER stack directly. Malicious user code cannot spoof the
kernel ESP register, and by moving the check after SAVE_ALL, we can
use normal PER_CPU accesses to find all the relevant addresses.
With this patch applied, the improved syscall_nt_32 test finally
passes on 32-bit kernels.
[1] It isn't obviously correct, but it is nonetheless safe from vm86
shenanigans as far as I can tell. A user can't point EIP at
entry_SYSENTER_32 while in vm86 mode because entry_SYSENTER_32,
like all kernel addresses, is greater than 0xffff and would thus
violate the CS segment limit.
Signed-off-by: Andy Lutomirski <luto@kernel.org>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/b2cdbc037031c07ecf2c40a96069318aec0e7971.1457578375.git.luto@kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2016-03-10 03:00:32 +00:00
|
|
|
jb .Lnmi_from_sysenter_stack
|
|
|
|
|
|
|
|
/* Not on SYSENTER stack. */
|
2015-06-08 07:49:11 +00:00
|
|
|
call do_nmi
|
2016-09-21 21:03:59 +00:00
|
|
|
jmp .Lrestore_all_notrace
|
2008-11-24 14:38:45 +00:00
|
|
|
|
x86/entry/32: Simplify and fix up the SYSENTER stack #DB/NMI fixup
Right after SYSENTER, we can get a #DB or NMI. On x86_32, there's no IST,
so the exception handler is invoked on the temporary SYSENTER stack.
Because the SYSENTER stack is very small, we have a fixup to switch
off the stack quickly when this happens. The old fixup had several issues:
1. It checked the interrupt frame's CS and EIP. This wasn't
obviously correct on Xen or if vm86 mode was in use [1].
2. In the NMI handler, it did some frightening digging into the
stack frame. I'm not convinced this digging was correct.
3. The fixup didn't switch stacks and then switch back. Instead, it
synthesized a brand new stack frame that would redirect the IRET
back to the SYSENTER code. That frame was highly questionable.
For one thing, if NMI nested inside #DB, we would effectively
abort the #DB prologue, which was probably safe but was
frightening. For another, the code used PUSHFL to write the
FLAGS portion of the frame, which was simply bogus -- by the time
PUSHFL was called, at least TF, NT, VM, and all of the arithmetic
flags were clobbered.
Simplify this considerably. Instead of looking at the saved frame
to see where we came from, check the hardware ESP register against
the SYSENTER stack directly. Malicious user code cannot spoof the
kernel ESP register, and by moving the check after SAVE_ALL, we can
use normal PER_CPU accesses to find all the relevant addresses.
With this patch applied, the improved syscall_nt_32 test finally
passes on 32-bit kernels.
[1] It isn't obviously correct, but it is nonetheless safe from vm86
shenanigans as far as I can tell. A user can't point EIP at
entry_SYSENTER_32 while in vm86 mode because entry_SYSENTER_32,
like all kernel addresses, is greater than 0xffff and would thus
violate the CS segment limit.
Signed-off-by: Andy Lutomirski <luto@kernel.org>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/b2cdbc037031c07ecf2c40a96069318aec0e7971.1457578375.git.luto@kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2016-03-10 03:00:32 +00:00
|
|
|
.Lnmi_from_sysenter_stack:
|
|
|
|
/*
|
|
|
|
* We're on the SYSENTER stack. Switch off. No one (not even debug)
|
|
|
|
* is using the thread stack right now, so it's safe for us to use it.
|
|
|
|
*/
|
2016-10-20 16:34:40 +00:00
|
|
|
movl %esp, %ebx
|
x86/entry/32: Simplify and fix up the SYSENTER stack #DB/NMI fixup
Right after SYSENTER, we can get a #DB or NMI. On x86_32, there's no IST,
so the exception handler is invoked on the temporary SYSENTER stack.
Because the SYSENTER stack is very small, we have a fixup to switch
off the stack quickly when this happens. The old fixup had several issues:
1. It checked the interrupt frame's CS and EIP. This wasn't
obviously correct on Xen or if vm86 mode was in use [1].
2. In the NMI handler, it did some frightening digging into the
stack frame. I'm not convinced this digging was correct.
3. The fixup didn't switch stacks and then switch back. Instead, it
synthesized a brand new stack frame that would redirect the IRET
back to the SYSENTER code. That frame was highly questionable.
For one thing, if NMI nested inside #DB, we would effectively
abort the #DB prologue, which was probably safe but was
frightening. For another, the code used PUSHFL to write the
FLAGS portion of the frame, which was simply bogus -- by the time
PUSHFL was called, at least TF, NT, VM, and all of the arithmetic
flags were clobbered.
Simplify this considerably. Instead of looking at the saved frame
to see where we came from, check the hardware ESP register against
the SYSENTER stack directly. Malicious user code cannot spoof the
kernel ESP register, and by moving the check after SAVE_ALL, we can
use normal PER_CPU accesses to find all the relevant addresses.
With this patch applied, the improved syscall_nt_32 test finally
passes on 32-bit kernels.
[1] It isn't obviously correct, but it is nonetheless safe from vm86
shenanigans as far as I can tell. A user can't point EIP at
entry_SYSENTER_32 while in vm86 mode because entry_SYSENTER_32,
like all kernel addresses, is greater than 0xffff and would thus
violate the CS segment limit.
Signed-off-by: Andy Lutomirski <luto@kernel.org>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/b2cdbc037031c07ecf2c40a96069318aec0e7971.1457578375.git.luto@kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2016-03-10 03:00:32 +00:00
|
|
|
movl PER_CPU_VAR(cpu_current_top_of_stack), %esp
|
|
|
|
call do_nmi
|
2016-10-20 16:34:40 +00:00
|
|
|
movl %ebx, %esp
|
2016-09-21 21:03:59 +00:00
|
|
|
jmp .Lrestore_all_notrace
|
2008-11-24 14:38:45 +00:00
|
|
|
|
2014-05-04 17:36:22 +00:00
|
|
|
#ifdef CONFIG_X86_ESPFIX32
|
2016-09-21 21:03:59 +00:00
|
|
|
.Lnmi_espfix_stack:
|
x86/debug: Remove perpetually broken, unmaintainable dwarf annotations
So the dwarf2 annotations in low level assembly code have
become an increasing hindrance: unreadable, messy macros
mixed into some of the most security sensitive code paths
of the Linux kernel.
These debug info annotations don't even buy the upstream
kernel anything: dwarf driven stack unwinding has caused
problems in the past so it's out of tree, and the upstream
kernel only uses the much more robust framepointers based
stack unwinding method.
In addition to that there's a steady, slow bitrot going
on with these annotations, requiring frequent fixups.
There's no tooling and no functionality upstream that
keeps it correct.
So burn down the sick forest, allowing new, healthier growth:
27 files changed, 350 insertions(+), 1101 deletions(-)
Someone who has the willingness and time to do this
properly can attempt to reintroduce dwarf debuginfo in x86
assembly code plus dwarf unwinding from first principles,
with the following conditions:
- it should be maximally readable, and maximally low-key to
'ordinary' code reading and maintenance.
- find a build time method to insert dwarf annotations
automatically in the most common cases, for pop/push
instructions that manipulate the stack pointer. This could
be done for example via a preprocessing step that just
looks for common patterns - plus special annotations for
the few cases where we want to depart from the default.
We have hundreds of CFI annotations, so automating most of
that makes sense.
- it should come with build tooling checks that ensure that
CFI annotations are sensible. We've seen such efforts from
the framepointer side, and there's no reason it couldn't be
done on the dwarf side.
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: Frédéric Weisbecker <fweisbec@gmail.com
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Jan Beulich <JBeulich@suse.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2015-05-28 10:21:47 +00:00
|
|
|
/*
|
2008-11-24 14:38:45 +00:00
|
|
|
* create the pointer to lss back
|
|
|
|
*/
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl %ss
|
|
|
|
pushl %esp
|
|
|
|
addl $4, (%esp)
|
2008-11-24 14:38:45 +00:00
|
|
|
/* copy the iret frame of 12 bytes */
|
|
|
|
.rept 3
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl 16(%esp)
|
2008-11-24 14:38:45 +00:00
|
|
|
.endr
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl %eax
|
2008-11-24 14:38:45 +00:00
|
|
|
SAVE_ALL
|
2016-10-20 16:34:40 +00:00
|
|
|
ENCODE_FRAME_POINTER
|
2015-06-08 07:49:11 +00:00
|
|
|
FIXUP_ESPFIX_STACK # %eax == %esp
|
|
|
|
xorl %edx, %edx # zero error code
|
|
|
|
call do_nmi
|
2008-11-24 14:38:45 +00:00
|
|
|
RESTORE_REGS
|
2015-06-08 07:49:11 +00:00
|
|
|
lss 12+4(%esp), %esp # back to espfix stack
|
2016-09-21 21:03:59 +00:00
|
|
|
jmp .Lirq_return
|
2014-05-04 17:36:22 +00:00
|
|
|
#endif
|
2008-11-24 14:38:45 +00:00
|
|
|
END(nmi)
|
|
|
|
|
|
|
|
ENTRY(int3)
|
2012-09-21 20:58:10 +00:00
|
|
|
ASM_CLAC
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl $-1 # mark this as an int
|
2008-11-24 14:38:45 +00:00
|
|
|
SAVE_ALL
|
2016-10-20 16:34:40 +00:00
|
|
|
ENCODE_FRAME_POINTER
|
2008-11-24 14:38:45 +00:00
|
|
|
TRACE_IRQS_OFF
|
2015-06-08 07:49:11 +00:00
|
|
|
xorl %edx, %edx # zero error code
|
|
|
|
movl %esp, %eax # pt_regs pointer
|
|
|
|
call do_int3
|
|
|
|
jmp ret_from_exception
|
2008-11-24 14:38:45 +00:00
|
|
|
END(int3)
|
|
|
|
|
|
|
|
ENTRY(general_protection)
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl $do_general_protection
|
2016-09-21 21:04:00 +00:00
|
|
|
jmp common_exception
|
2008-11-24 14:38:45 +00:00
|
|
|
END(general_protection)
|
|
|
|
|
2010-10-14 09:22:52 +00:00
|
|
|
#ifdef CONFIG_KVM_GUEST
|
|
|
|
ENTRY(async_page_fault)
|
2012-09-21 20:58:10 +00:00
|
|
|
ASM_CLAC
|
2015-06-08 07:49:11 +00:00
|
|
|
pushl $do_async_page_fault
|
2016-09-21 21:04:00 +00:00
|
|
|
jmp common_exception
|
2011-03-08 21:39:24 +00:00
|
|
|
END(async_page_fault)
|
2010-10-14 09:22:52 +00:00
|
|
|
#endif
|
2016-07-14 20:22:55 +00:00
|
|
|
|
|
|
|
ENTRY(rewind_stack_do_exit)
|
|
|
|
/* Prevent any naive code from trying to unwind to our caller. */
|
|
|
|
xorl %ebp, %ebp
|
|
|
|
|
|
|
|
movl PER_CPU_VAR(cpu_current_top_of_stack), %esi
|
|
|
|
leal -TOP_OF_KERNEL_STACK_PADDING-PTREGS_SIZE(%esi), %esp
|
|
|
|
|
|
|
|
call do_exit
|
|
|
|
1: jmp 1b
|
|
|
|
END(rewind_stack_do_exit)
|