leandrof/linux
1
0
Fork 0
forked from Minki/linux
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2f123bce18
linux/arch/powerpc/include/asm/exception-64e.h

213 lines
7.1 KiB
C
Raw Normal View History

powerpc: Add definitions used by exception handling on 64-bit Book3E This adds various definitions and macros used by the exception and TLB miss handling on 64-bit BookE It also adds the definitions of the SPRGs used for various exception types Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-07-23 23:15:39 +00:00
/*
* Definitions for use by exception code on Book3-E
*
* Copyright (C) 2008 Ben. Herrenschmidt (benh@kernel.crashing.org), IBM Corp.
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version
* 2 of the License, or (at your option) any later version.
*/
#ifndef _ASM_POWERPC_EXCEPTION_64E_H
#define _ASM_POWERPC_EXCEPTION_64E_H
/*
* SPRGs usage an other considerations...
*
* Since TLB miss and other standard exceptions can be interrupted by
* critical exceptions which can themselves be interrupted by machine
* checks, and since the two later can themselves cause a TLB miss when
* hitting the linear mapping for the kernel stacks, we need to be a bit
* creative on how we use SPRGs.
*
* The base idea is that we have one SRPG reserved for critical and one
* for machine check interrupts. Those are used to save a GPR that can
* then be used to get the PACA, and store as much context as we need
* to save in there. That includes saving the SPRGs used by the TLB miss
* handler for linear mapping misses and the associated SRR0/1 due to
* the above re-entrancy issue.
*
* So here's the current usage pattern. It's done regardless of which
* SPRGs are user-readable though, thus we might have to change some of
* this later. In order to do that more easily, we use special constants
* for naming them
*
* WARNING: Some of these SPRGs are user readable. We need to do something
* about it as some point by making sure they can't be used to leak kernel
* critical data
*/
powerpc/booke64: Use GSRR registers in Guest Doorbell interrupts Guest Doorbell interrupts use guest save and restore registers. Add a new Guest Doorbell exception type to accommodate GSRR0/1 SPRs usage in exception prolog and fix the exception handler. Signed-off-by: Mihai Caraman <mihai.caraman@freescale.com> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2012-08-06 03:27:04 +00:00
#define PACA_EXGDBELL PACA_EXGEN
powerpc: Add definitions used by exception handling on 64-bit Book3E This adds various definitions and macros used by the exception and TLB miss handling on 64-bit BookE It also adds the definitions of the SPRGs used for various exception types Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-07-23 23:15:39 +00:00
/* We are out of SPRGs so we save some things in the PACA. The normal
* exception frame is smaller than the CRIT or MC one though
*/
#define EX_R1 (0 * 8)
#define EX_CR (1 * 8)
#define EX_R10 (2 * 8)
#define EX_R11 (3 * 8)
powerpc/booke64: Use SPRG7 for VDSO Previously SPRG3 was marked for use by both VDSO and critical interrupts (though critical interrupts were not fully implemented). In commit 8b64a9dfb091f1eca8b7e58da82f1e7d1d5fe0ad ("powerpc/booke64: Use SPRG0/3 scratch for bolted TLB miss & crit int"), Mihai Caraman made an attempt to resolve this conflict by restoring the VDSO value early in the critical interrupt, but this has some issues: - It's incompatible with EXCEPTION_COMMON which restores r13 from the by-then-overwritten scratch (this cost me some debugging time). - It forces critical exceptions to be a special case handled differently from even machine check and debug level exceptions. - It didn't occur to me that it was possible to make this work at all (by doing a final "ld r13, PACA_EXCRIT+EX_R13(r13)") until after I made (most of) this patch. :-) It might be worth investigating using a load rather than SPRG on return from all exceptions (except TLB misses where the scratch never leaves the SPRG) -- it could save a few cycles. Until then, let's stick with SPRG for all exceptions. Since we cannot use SPRG4-7 for scratch without corrupting the state of a KVM guest, move VDSO to SPRG7 on book3e. Since neither SPRG4-7 nor critical interrupts exist on book3s, SPRG3 is still used for VDSO there. Signed-off-by: Scott Wood <scottwood@freescale.com> Cc: Mihai Caraman <mihai.caraman@freescale.com> Cc: Anton Blanchard <anton@samba.org> Cc: Paul Mackerras <paulus@samba.org> Cc: kvm-ppc@vger.kernel.org
2014-03-10 22:29:38 +00:00
#define EX_R14 (4 * 8)
#define EX_R15 (5 * 8)
powerpc: Add definitions used by exception handling on 64-bit Book3E This adds various definitions and macros used by the exception and TLB miss handling on 64-bit BookE It also adds the definitions of the SPRGs used for various exception types Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-07-23 23:15:39 +00:00
powerpc/book3e-64: use a separate TLB handler when linear map is bolted On MMUs such as FSL where we can guarantee the entire linear mapping is bolted, we don't need to worry about linear TLB misses. If on top of that we do a full table walk, we get rid of all recursive TLB faults, and can dispense with some state saving. This gains a few percent on TLB-miss-heavy workloads, and around 50% on a benchmark that had a high rate of virtual page table faults under the normal handler. While touching the EX_TLB layout, remove EX_TLB_MMUCR0, EX_TLB_SRR0, and EX_TLB_SRR1 as they're not used. [BenH: Fixed build with 64K pages (wsp config)] Signed-off-by: Scott Wood <scottwood@freescale.com> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2011-06-22 11:25:42 +00:00
/*
* The TLB miss exception uses different slots.
*
* The bolted variant uses only the first six fields,
* which in combination with pgd and kernel_pgd fits in
* one 64-byte cache line.
*/
powerpc: Add definitions used by exception handling on 64-bit Book3E This adds various definitions and macros used by the exception and TLB miss handling on 64-bit BookE It also adds the definitions of the SPRGs used for various exception types Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-07-23 23:15:39 +00:00
#define EX_TLB_R10 ( 0 * 8)
#define EX_TLB_R11 ( 1 * 8)
powerpc/book3e-64: use a separate TLB handler when linear map is bolted On MMUs such as FSL where we can guarantee the entire linear mapping is bolted, we don't need to worry about linear TLB misses. If on top of that we do a full table walk, we get rid of all recursive TLB faults, and can dispense with some state saving. This gains a few percent on TLB-miss-heavy workloads, and around 50% on a benchmark that had a high rate of virtual page table faults under the normal handler. While touching the EX_TLB layout, remove EX_TLB_MMUCR0, EX_TLB_SRR0, and EX_TLB_SRR1 as they're not used. [BenH: Fixed build with 64K pages (wsp config)] Signed-off-by: Scott Wood <scottwood@freescale.com> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2011-06-22 11:25:42 +00:00
#define EX_TLB_R14 ( 2 * 8)
#define EX_TLB_R15 ( 3 * 8)
#define EX_TLB_R16 ( 4 * 8)
#define EX_TLB_CR ( 5 * 8)
#define EX_TLB_R12 ( 6 * 8)
#define EX_TLB_R13 ( 7 * 8)
powerpc: Add definitions used by exception handling on 64-bit Book3E This adds various definitions and macros used by the exception and TLB miss handling on 64-bit BookE It also adds the definitions of the SPRGs used for various exception types Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-07-23 23:15:39 +00:00
#define EX_TLB_DEAR ( 8 * 8) /* Level 0 and 2 only */
#define EX_TLB_ESR ( 9 * 8) /* Level 0 and 2 only */
#define EX_TLB_SRR0 (10 * 8)
#define EX_TLB_SRR1 (11 * 8)
#ifdef CONFIG_BOOK3E_MMU_TLB_STATS
powerpc/book3e-64: use a separate TLB handler when linear map is bolted On MMUs such as FSL where we can guarantee the entire linear mapping is bolted, we don't need to worry about linear TLB misses. If on top of that we do a full table walk, we get rid of all recursive TLB faults, and can dispense with some state saving. This gains a few percent on TLB-miss-heavy workloads, and around 50% on a benchmark that had a high rate of virtual page table faults under the normal handler. While touching the EX_TLB layout, remove EX_TLB_MMUCR0, EX_TLB_SRR0, and EX_TLB_SRR1 as they're not used. [BenH: Fixed build with 64K pages (wsp config)] Signed-off-by: Scott Wood <scottwood@freescale.com> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2011-06-22 11:25:42 +00:00
#define EX_TLB_R8 (12 * 8)
#define EX_TLB_R9 (13 * 8)
#define EX_TLB_LR (14 * 8)
#define EX_TLB_SIZE (15 * 8)
powerpc: Add definitions used by exception handling on 64-bit Book3E This adds various definitions and macros used by the exception and TLB miss handling on 64-bit BookE It also adds the definitions of the SPRGs used for various exception types Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-07-23 23:15:39 +00:00
#else
powerpc/book3e-64: use a separate TLB handler when linear map is bolted On MMUs such as FSL where we can guarantee the entire linear mapping is bolted, we don't need to worry about linear TLB misses. If on top of that we do a full table walk, we get rid of all recursive TLB faults, and can dispense with some state saving. This gains a few percent on TLB-miss-heavy workloads, and around 50% on a benchmark that had a high rate of virtual page table faults under the normal handler. While touching the EX_TLB layout, remove EX_TLB_MMUCR0, EX_TLB_SRR0, and EX_TLB_SRR1 as they're not used. [BenH: Fixed build with 64K pages (wsp config)] Signed-off-by: Scott Wood <scottwood@freescale.com> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2011-06-22 11:25:42 +00:00
#define EX_TLB_SIZE (12 * 8)
powerpc: Add definitions used by exception handling on 64-bit Book3E This adds various definitions and macros used by the exception and TLB miss handling on 64-bit BookE It also adds the definitions of the SPRGs used for various exception types Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-07-23 23:15:39 +00:00
#endif
#define START_EXCEPTION(label) \
.globl exc_##label##_book3e; \
exc_##label##_book3e:
/* TLB miss exception prolog
*
* This prolog handles re-entrancy (up to 3 levels supported in the PACA
* though we currently don't test for overflow). It provides you with a
* re-entrancy safe working space of r10...r16 and CR with r12 being used
* as the exception area pointer in the PACA for that level of re-entrancy
* and r13 containing the PACA pointer.
*
* SRR0 and SRR1 are saved, but DEAR and ESR are not, since they don't apply
* as-is for instruction exceptions. It's up to the actual exception code
* to save them as well if required.
*/
#define TLB_MISS_PROLOG \
mtspr SPRN_SPRG_TLB_SCRATCH,r12; \
mfspr r12,SPRN_SPRG_TLB_EXFRAME; \
std r10,EX_TLB_R10(r12); \
mfcr r10; \
std r11,EX_TLB_R11(r12); \
mfspr r11,SPRN_SPRG_TLB_SCRATCH; \
std r13,EX_TLB_R13(r12); \
mfspr r13,SPRN_SPRG_PACA; \
std r14,EX_TLB_R14(r12); \
addi r14,r12,EX_TLB_SIZE; \
std r15,EX_TLB_R15(r12); \
mfspr r15,SPRN_SRR1; \
std r16,EX_TLB_R16(r12); \
mfspr r16,SPRN_SRR0; \
std r10,EX_TLB_CR(r12); \
std r11,EX_TLB_R12(r12); \
mtspr SPRN_SPRG_TLB_EXFRAME,r14; \
std r15,EX_TLB_SRR1(r12); \
std r16,EX_TLB_SRR0(r12); \
TLB_MISS_PROLOG_STATS
/* And these are the matching epilogs that restores things
*
* There are 3 epilogs:
*
* - SUCCESS : Unwinds one level
* - ERROR : restore from level 0 and reset
* - ERROR_SPECIAL : restore from current level and reset
*
* Normal errors use ERROR, that is, they restore the initial fault context
* and trigger a fault. However, there is a special case for linear mapping
* errors. Those should basically never happen, but if they do happen, we
* want the error to point out the context that did that linear mapping
* fault, not the initial level 0 (basically, we got a bogus PGF or something
* like that). For userland errors on the linear mapping, there is no
* difference since those are always level 0 anyway
*/
#define TLB_MISS_RESTORE(freg) \
ld r14,EX_TLB_CR(r12); \
ld r10,EX_TLB_R10(r12); \
ld r15,EX_TLB_SRR0(r12); \
ld r16,EX_TLB_SRR1(r12); \
mtspr SPRN_SPRG_TLB_EXFRAME,freg; \
ld r11,EX_TLB_R11(r12); \
mtcr r14; \
ld r13,EX_TLB_R13(r12); \
ld r14,EX_TLB_R14(r12); \
mtspr SPRN_SRR0,r15; \
ld r15,EX_TLB_R15(r12); \
mtspr SPRN_SRR1,r16; \
TLB_MISS_RESTORE_STATS \
ld r16,EX_TLB_R16(r12); \
ld r12,EX_TLB_R12(r12); \
#define TLB_MISS_EPILOG_SUCCESS \
TLB_MISS_RESTORE(r12)
#define TLB_MISS_EPILOG_ERROR \
addi r12,r13,PACA_EXTLB; \
TLB_MISS_RESTORE(r12)
#define TLB_MISS_EPILOG_ERROR_SPECIAL \
addi r11,r13,PACA_EXTLB; \
TLB_MISS_RESTORE(r11)
#ifdef CONFIG_BOOK3E_MMU_TLB_STATS
#define TLB_MISS_PROLOG_STATS \
mflr r10; \
std r8,EX_TLB_R8(r12); \
std r9,EX_TLB_R9(r12); \
std r10,EX_TLB_LR(r12);
#define TLB_MISS_RESTORE_STATS \
ld r16,EX_TLB_LR(r12); \
ld r9,EX_TLB_R9(r12); \
ld r8,EX_TLB_R8(r12); \
mtlr r16;
#define TLB_MISS_STATS_D(name) \
addi r9,r13,MMSTAT_DSTATS+name; \
powerpc: No need to use dot symbols when branching to a function binutils is smart enough to know that a branch to a function descriptor is actually a branch to the functions text address. Alan tells me that binutils has been doing this for 9 years. Signed-off-by: Anton Blanchard <anton@samba.org>
2014-02-04 05:04:35 +00:00
bl tlb_stat_inc;
powerpc: Add definitions used by exception handling on 64-bit Book3E This adds various definitions and macros used by the exception and TLB miss handling on 64-bit BookE It also adds the definitions of the SPRGs used for various exception types Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-07-23 23:15:39 +00:00
#define TLB_MISS_STATS_I(name) \
addi r9,r13,MMSTAT_ISTATS+name; \
powerpc: No need to use dot symbols when branching to a function binutils is smart enough to know that a branch to a function descriptor is actually a branch to the functions text address. Alan tells me that binutils has been doing this for 9 years. Signed-off-by: Anton Blanchard <anton@samba.org>
2014-02-04 05:04:35 +00:00
bl tlb_stat_inc;
powerpc: Add definitions used by exception handling on 64-bit Book3E This adds various definitions and macros used by the exception and TLB miss handling on 64-bit BookE It also adds the definitions of the SPRGs used for various exception types Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-07-23 23:15:39 +00:00
#define TLB_MISS_STATS_X(name) \
ld r8,PACA_EXTLB+EX_TLB_ESR(r13); \
cmpdi cr2,r8,-1; \
beq cr2,61f; \
addi r9,r13,MMSTAT_DSTATS+name; \
b 62f; \
61: addi r9,r13,MMSTAT_ISTATS+name; \
powerpc: No need to use dot symbols when branching to a function binutils is smart enough to know that a branch to a function descriptor is actually a branch to the functions text address. Alan tells me that binutils has been doing this for 9 years. Signed-off-by: Anton Blanchard <anton@samba.org>
2014-02-04 05:04:35 +00:00
62: bl tlb_stat_inc;
powerpc: Add definitions used by exception handling on 64-bit Book3E This adds various definitions and macros used by the exception and TLB miss handling on 64-bit BookE It also adds the definitions of the SPRGs used for various exception types Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-07-23 23:15:39 +00:00
#define TLB_MISS_STATS_SAVE_INFO \
powerpc/book3e-64: use a separate TLB handler when linear map is bolted On MMUs such as FSL where we can guarantee the entire linear mapping is bolted, we don't need to worry about linear TLB misses. If on top of that we do a full table walk, we get rid of all recursive TLB faults, and can dispense with some state saving. This gains a few percent on TLB-miss-heavy workloads, and around 50% on a benchmark that had a high rate of virtual page table faults under the normal handler. While touching the EX_TLB layout, remove EX_TLB_MMUCR0, EX_TLB_SRR0, and EX_TLB_SRR1 as they're not used. [BenH: Fixed build with 64K pages (wsp config)] Signed-off-by: Scott Wood <scottwood@freescale.com> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2011-06-22 11:25:42 +00:00
std r14,EX_TLB_ESR(r12); /* save ESR */
#define TLB_MISS_STATS_SAVE_INFO_BOLTED \
std r14,PACA_EXTLB+EX_TLB_ESR(r13); /* save ESR */
powerpc: Add definitions used by exception handling on 64-bit Book3E This adds various definitions and macros used by the exception and TLB miss handling on 64-bit BookE It also adds the definitions of the SPRGs used for various exception types Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-07-23 23:15:39 +00:00
#else
#define TLB_MISS_PROLOG_STATS
#define TLB_MISS_RESTORE_STATS
powerpc/book3e-64: use a separate TLB handler when linear map is bolted On MMUs such as FSL where we can guarantee the entire linear mapping is bolted, we don't need to worry about linear TLB misses. If on top of that we do a full table walk, we get rid of all recursive TLB faults, and can dispense with some state saving. This gains a few percent on TLB-miss-heavy workloads, and around 50% on a benchmark that had a high rate of virtual page table faults under the normal handler. While touching the EX_TLB layout, remove EX_TLB_MMUCR0, EX_TLB_SRR0, and EX_TLB_SRR1 as they're not used. [BenH: Fixed build with 64K pages (wsp config)] Signed-off-by: Scott Wood <scottwood@freescale.com> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2011-06-22 11:25:42 +00:00
#define TLB_MISS_PROLOG_STATS_BOLTED
#define TLB_MISS_RESTORE_STATS_BOLTED
powerpc: Add definitions used by exception handling on 64-bit Book3E This adds various definitions and macros used by the exception and TLB miss handling on 64-bit BookE It also adds the definitions of the SPRGs used for various exception types Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-07-23 23:15:39 +00:00
#define TLB_MISS_STATS_D(name)
#define TLB_MISS_STATS_I(name)
#define TLB_MISS_STATS_X(name)
#define TLB_MISS_STATS_Y(name)
#define TLB_MISS_STATS_SAVE_INFO
powerpc/book3e-64: use a separate TLB handler when linear map is bolted On MMUs such as FSL where we can guarantee the entire linear mapping is bolted, we don't need to worry about linear TLB misses. If on top of that we do a full table walk, we get rid of all recursive TLB faults, and can dispense with some state saving. This gains a few percent on TLB-miss-heavy workloads, and around 50% on a benchmark that had a high rate of virtual page table faults under the normal handler. While touching the EX_TLB layout, remove EX_TLB_MMUCR0, EX_TLB_SRR0, and EX_TLB_SRR1 as they're not used. [BenH: Fixed build with 64K pages (wsp config)] Signed-off-by: Scott Wood <scottwood@freescale.com> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2011-06-22 11:25:42 +00:00
#define TLB_MISS_STATS_SAVE_INFO_BOLTED
powerpc: Add definitions used by exception handling on 64-bit Book3E This adds various definitions and macros used by the exception and TLB miss handling on 64-bit BookE It also adds the definitions of the SPRGs used for various exception types Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-07-23 23:15:39 +00:00
#endif
powerpc/book3e-64: Add helper function to setup IVORs Not all 64-bit Book-3E parts will have fixed IVORs so add a function that cpusetup code can call to setup the base IVORs (0..15) to match the fixed offsets. We need to 'or' part of interrupt_base_book3e into the IVORs since on parts that have them the IVPR doesn't extend as far down. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-08-18 19:08:32 +00:00
#define SET_IVOR(vector_number, vector_offset) \
li r3,vector_offset@l; \
ori r3,r3,interrupt_base_book3e@l; \
mtspr SPRN_IVOR##vector_number,r3;
powerpc: Add definitions used by exception handling on 64-bit Book3E This adds various definitions and macros used by the exception and TLB miss handling on 64-bit BookE It also adds the definitions of the SPRGs used for various exception types Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-07-23 23:15:39 +00:00
#endif /* _ASM_POWERPC_EXCEPTION_64E_H */
Reference in New Issue Copy Permalink