2011-01-24 07:42:41 +00:00
|
|
|
/*
|
powerpc: Rework lazy-interrupt handling
The current implementation of lazy interrupts handling has some
issues that this tries to address.
We don't do the various workarounds we need to do when re-enabling
interrupts in some cases such as when returning from an interrupt
and thus we may still lose or get delayed decrementer or doorbell
interrupts.
The current scheme also makes it much harder to handle the external
"edge" interrupts provided by some BookE processors when using the
EPR facility (External Proxy) and the Freescale Hypervisor.
Additionally, we tend to keep interrupts hard disabled in a number
of cases, such as decrementer interrupts, external interrupts, or
when a masked decrementer interrupt is pending. This is sub-optimal.
This is an attempt at fixing it all in one go by reworking the way
we do the lazy interrupt disabling from the ground up.
The base idea is to replace the "hard_enabled" field with a
"irq_happened" field in which we store a bit mask of what interrupt
occurred while soft-disabled.
When re-enabling, either via arch_local_irq_restore() or when returning
from an interrupt, we can now decide what to do by testing bits in that
field.
We then implement replaying of the missed interrupts either by
re-using the existing exception frame (in exception exit case) or via
the creation of a new one from an assembly trampoline (in the
arch_local_irq_enable case).
This removes the need to play with the decrementer to try to create
fake interrupts, among others.
In addition, this adds a few refinements:
- We no longer hard disable decrementer interrupts that occur
while soft-disabled. We now simply bump the decrementer back to max
(on BookS) or leave it stopped (on BookE) and continue with hard interrupts
enabled, which means that we'll potentially get better sample quality from
performance monitor interrupts.
- Timer, decrementer and doorbell interrupts now hard-enable
shortly after removing the source of the interrupt, which means
they no longer run entirely hard disabled. Again, this will improve
perf sample quality.
- On Book3E 64-bit, we now make the performance monitor interrupt
act as an NMI like Book3S (the necessary C code for that to work
appear to already be present in the FSL perf code, notably calling
nmi_enter instead of irq_enter). (This also fixes a bug where BookE
perfmon interrupts could clobber r14 ... oops)
- We could make "masked" decrementer interrupts act as NMIs when doing
timer-based perf sampling to improve the sample quality.
Signed-off-by-yet: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
v2:
- Add hard-enable to decrementer, timer and doorbells
- Fix CR clobber in masked irq handling on BookE
- Make embedded perf interrupt act as an NMI
- Add a PACA_HAPPENED_EE_EDGE for use by FSL if they want
to retrigger an interrupt without preventing hard-enable
v3:
- Fix or vs. ori bug on Book3E
- Fix enabling of interrupts for some exceptions on Book3E
v4:
- Fix resend of doorbells on return from interrupt on Book3E
v5:
- Rebased on top of my latest series, which involves some significant
rework of some aspects of the patch.
v6:
- 32-bit compile fix
- more compile fixes with various .config combos
- factor out the asm code to soft-disable interrupts
- remove the C wrapper around preempt_schedule_irq
v7:
- Fix a bug with hard irq state tracking on native power7
2012-03-06 07:27:59 +00:00
|
|
|
* This file contains the power_save function for Power7 CPUs.
|
2011-01-24 07:42:41 +00:00
|
|
|
*
|
|
|
|
|
* This program is free software; you can redistribute it and/or
|
|
|
|
|
* modify it under the terms of the GNU General Public License
|
|
|
|
|
* as published by the Free Software Foundation; either version
|
|
|
|
|
* 2 of the License, or (at your option) any later version.
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#include <linux/threads.h>
|
|
|
|
|
#include <asm/processor.h>
|
|
|
|
|
#include <asm/page.h>
|
|
|
|
|
#include <asm/cputable.h>
|
|
|
|
|
#include <asm/thread_info.h>
|
|
|
|
|
#include <asm/ppc_asm.h>
|
|
|
|
|
#include <asm/asm-offsets.h>
|
|
|
|
|
#include <asm/ppc-opcode.h>
|
powerpc: Rework lazy-interrupt handling
The current implementation of lazy interrupts handling has some
issues that this tries to address.
We don't do the various workarounds we need to do when re-enabling
interrupts in some cases such as when returning from an interrupt
and thus we may still lose or get delayed decrementer or doorbell
interrupts.
The current scheme also makes it much harder to handle the external
"edge" interrupts provided by some BookE processors when using the
EPR facility (External Proxy) and the Freescale Hypervisor.
Additionally, we tend to keep interrupts hard disabled in a number
of cases, such as decrementer interrupts, external interrupts, or
when a masked decrementer interrupt is pending. This is sub-optimal.
This is an attempt at fixing it all in one go by reworking the way
we do the lazy interrupt disabling from the ground up.
The base idea is to replace the "hard_enabled" field with a
"irq_happened" field in which we store a bit mask of what interrupt
occurred while soft-disabled.
When re-enabling, either via arch_local_irq_restore() or when returning
from an interrupt, we can now decide what to do by testing bits in that
field.
We then implement replaying of the missed interrupts either by
re-using the existing exception frame (in exception exit case) or via
the creation of a new one from an assembly trampoline (in the
arch_local_irq_enable case).
This removes the need to play with the decrementer to try to create
fake interrupts, among others.
In addition, this adds a few refinements:
- We no longer hard disable decrementer interrupts that occur
while soft-disabled. We now simply bump the decrementer back to max
(on BookS) or leave it stopped (on BookE) and continue with hard interrupts
enabled, which means that we'll potentially get better sample quality from
performance monitor interrupts.
- Timer, decrementer and doorbell interrupts now hard-enable
shortly after removing the source of the interrupt, which means
they no longer run entirely hard disabled. Again, this will improve
perf sample quality.
- On Book3E 64-bit, we now make the performance monitor interrupt
act as an NMI like Book3S (the necessary C code for that to work
appear to already be present in the FSL perf code, notably calling
nmi_enter instead of irq_enter). (This also fixes a bug where BookE
perfmon interrupts could clobber r14 ... oops)
- We could make "masked" decrementer interrupts act as NMIs when doing
timer-based perf sampling to improve the sample quality.
Signed-off-by-yet: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
v2:
- Add hard-enable to decrementer, timer and doorbells
- Fix CR clobber in masked irq handling on BookE
- Make embedded perf interrupt act as an NMI
- Add a PACA_HAPPENED_EE_EDGE for use by FSL if they want
to retrigger an interrupt without preventing hard-enable
v3:
- Fix or vs. ori bug on Book3E
- Fix enabling of interrupts for some exceptions on Book3E
v4:
- Fix resend of doorbells on return from interrupt on Book3E
v5:
- Rebased on top of my latest series, which involves some significant
rework of some aspects of the patch.
v6:
- 32-bit compile fix
- more compile fixes with various .config combos
- factor out the asm code to soft-disable interrupts
- remove the C wrapper around preempt_schedule_irq
v7:
- Fix a bug with hard irq state tracking on native power7
2012-03-06 07:27:59 +00:00
|
|
|
#include <asm/hw_irq.h>
|
2012-02-03 00:54:17 +00:00
|
|
|
#include <asm/kvm_book3s_asm.h>
|
2014-02-26 00:08:43 +00:00
|
|
|
#include <asm/opal.h>
|
2014-12-09 18:56:52 +00:00
|
|
|
#include <asm/cpuidle.h>
|
2014-12-09 18:56:53 +00:00
|
|
|
#include <asm/mmu-hash64.h>
|
2011-01-24 07:42:41 +00:00
|
|
|
|
|
|
|
|
#undef DEBUG
|
|
|
|
|
|
2014-12-09 18:56:53 +00:00
|
|
|
/*
|
|
|
|
|
* Use unused space in the interrupt stack to save and restore
|
|
|
|
|
* registers for winkle support.
|
|
|
|
|
*/
|
|
|
|
|
#define _SDR1 GPR3
|
|
|
|
|
#define _RPR GPR4
|
|
|
|
|
#define _SPURR GPR5
|
|
|
|
|
#define _PURR GPR6
|
|
|
|
|
#define _TSCR GPR7
|
|
|
|
|
#define _DSCR GPR8
|
|
|
|
|
#define _AMOR GPR9
|
|
|
|
|
#define _WORT GPR10
|
|
|
|
|
#define _WORC GPR11
|
|
|
|
|
|
2014-02-26 00:08:25 +00:00
|
|
|
/* Idle state entry routines */
|
2011-01-24 07:42:41 +00:00
|
|
|
|
2014-02-26 00:08:25 +00:00
|
|
|
#define IDLE_STATE_ENTER_SEQ(IDLE_INST) \
|
|
|
|
|
/* Magic NAP/SLEEP/WINKLE mode enter sequence */ \
|
|
|
|
|
std r0,0(r1); \
|
|
|
|
|
ptesync; \
|
|
|
|
|
ld r0,0(r1); \
|
|
|
|
|
1: cmp cr0,r0,r0; \
|
|
|
|
|
bne 1b; \
|
|
|
|
|
IDLE_INST; \
|
|
|
|
|
b .
|
2011-01-24 07:42:41 +00:00
|
|
|
|
2014-02-26 00:08:25 +00:00
|
|
|
.text
|
|
|
|
|
|
powerpc/powernv: Fix race in updating core_idle_state
core_idle_state is maintained for each core. It uses 0-7 bits to track
whether a thread in the core has entered fastsleep or winkle. 8th bit is
used as a lock bit.
The lock bit is set in these 2 scenarios-
- The thread is first in subcore to wakeup from sleep/winkle.
- If its the last thread in the core about to enter sleep/winkle
While the lock bit is set, if any other thread in the core wakes up, it
loops until the lock bit is cleared before proceeding in the wakeup
path. This helps prevent race conditions w.r.t fastsleep workaround and
prevents threads from switching to process context before core/subcore
resources are restored.
But, in the path to sleep/winkle entry, we currently don't check for
lock-bit. This exposes us to following race when running with subcore
on-
First thread in the subcorea Another thread in the same
waking up core entering sleep/winkle
lwarx r15,0,r14
ori r15,r15,PNV_CORE_IDLE_LOCK_BIT
stwcx. r15,0,r14
[Code to restore subcore state]
lwarx r15,0,r14
[clear thread bit]
stwcx. r15,0,r14
andi. r15,r15,PNV_CORE_IDLE_THREAD_BITS
stw r15,0(r14)
Here, after the thread entering sleep clears its thread bit in
core_idle_state, the value is overwritten by the thread waking up.
In such cases when the core enters fastsleep, code mistakes an idle
thread as running. Because of this, the first thread waking up from
fastsleep which is supposed to resync timebase skips it. So we can
end up having a core with stale timebase value.
This patch fixes the above race by looping on the lock bit even while
entering the idle states.
Signed-off-by: Shreyas B. Prabhu <shreyas@linux.vnet.ibm.com>
Fixes: 7b54e9f213f76 'powernv/powerpc: Add winkle support for offline cpus'
Cc: stable@vger.kernel.org # 3.19+
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2015-07-06 20:09:23 +00:00
|
|
|
/*
|
|
|
|
|
* Used by threads when the lock bit of core_idle_state is set.
|
|
|
|
|
* Threads will spin in HMT_LOW until the lock bit is cleared.
|
|
|
|
|
* r14 - pointer to core_idle_state
|
|
|
|
|
* r15 - used to load contents of core_idle_state
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
core_idle_lock_held:
|
|
|
|
|
HMT_LOW
|
|
|
|
|
3: lwz r15,0(r14)
|
|
|
|
|
andi. r15,r15,PNV_CORE_IDLE_LOCK_BIT
|
|
|
|
|
bne 3b
|
|
|
|
|
HMT_MEDIUM
|
|
|
|
|
lwarx r15,0,r14
|
|
|
|
|
blr
|
|
|
|
|
|
2014-02-26 00:08:25 +00:00
|
|
|
/*
|
|
|
|
|
* Pass requested state in r3:
|
2014-12-09 18:56:52 +00:00
|
|
|
* r3 - PNV_THREAD_NAP/SLEEP/WINKLE
|
2014-05-23 08:15:26 +00:00
|
|
|
*
|
|
|
|
|
* To check IRQ_HAPPENED in r4
|
|
|
|
|
* 0 - don't check
|
|
|
|
|
* 1 - check
|
2014-02-26 00:08:25 +00:00
|
|
|
*/
|
|
|
|
|
_GLOBAL(power7_powersave_common)
|
|
|
|
|
/* Use r3 to pass state nap/sleep/winkle */
|
2011-01-24 07:42:41 +00:00
|
|
|
/* NAP is a state loss, we create a regs frame on the
|
|
|
|
|
* stack, fill it up with the state we care about and
|
|
|
|
|
* stick a pointer to it in PACAR1. We really only
|
|
|
|
|
* need to save PC, some CR bits and the NV GPRs,
|
|
|
|
|
* but for now an interrupt frame will do.
|
|
|
|
|
*/
|
|
|
|
|
mflr r0
|
|
|
|
|
std r0,16(r1)
|
|
|
|
|
stdu r1,-INT_FRAME_SIZE(r1)
|
|
|
|
|
std r0,_LINK(r1)
|
|
|
|
|
std r0,_NIP(r1)
|
|
|
|
|
|
|
|
|
|
/* Hard disable interrupts */
|
|
|
|
|
mfmsr r9
|
|
|
|
|
rldicl r9,r9,48,1
|
|
|
|
|
rotldi r9,r9,16
|
|
|
|
|
mtmsrd r9,1 /* hard-disable interrupts */
|
powerpc: Rework lazy-interrupt handling
The current implementation of lazy interrupts handling has some
issues that this tries to address.
We don't do the various workarounds we need to do when re-enabling
interrupts in some cases such as when returning from an interrupt
and thus we may still lose or get delayed decrementer or doorbell
interrupts.
The current scheme also makes it much harder to handle the external
"edge" interrupts provided by some BookE processors when using the
EPR facility (External Proxy) and the Freescale Hypervisor.
Additionally, we tend to keep interrupts hard disabled in a number
of cases, such as decrementer interrupts, external interrupts, or
when a masked decrementer interrupt is pending. This is sub-optimal.
This is an attempt at fixing it all in one go by reworking the way
we do the lazy interrupt disabling from the ground up.
The base idea is to replace the "hard_enabled" field with a
"irq_happened" field in which we store a bit mask of what interrupt
occurred while soft-disabled.
When re-enabling, either via arch_local_irq_restore() or when returning
from an interrupt, we can now decide what to do by testing bits in that
field.
We then implement replaying of the missed interrupts either by
re-using the existing exception frame (in exception exit case) or via
the creation of a new one from an assembly trampoline (in the
arch_local_irq_enable case).
This removes the need to play with the decrementer to try to create
fake interrupts, among others.
In addition, this adds a few refinements:
- We no longer hard disable decrementer interrupts that occur
while soft-disabled. We now simply bump the decrementer back to max
(on BookS) or leave it stopped (on BookE) and continue with hard interrupts
enabled, which means that we'll potentially get better sample quality from
performance monitor interrupts.
- Timer, decrementer and doorbell interrupts now hard-enable
shortly after removing the source of the interrupt, which means
they no longer run entirely hard disabled. Again, this will improve
perf sample quality.
- On Book3E 64-bit, we now make the performance monitor interrupt
act as an NMI like Book3S (the necessary C code for that to work
appear to already be present in the FSL perf code, notably calling
nmi_enter instead of irq_enter). (This also fixes a bug where BookE
perfmon interrupts could clobber r14 ... oops)
- We could make "masked" decrementer interrupts act as NMIs when doing
timer-based perf sampling to improve the sample quality.
Signed-off-by-yet: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
v2:
- Add hard-enable to decrementer, timer and doorbells
- Fix CR clobber in masked irq handling on BookE
- Make embedded perf interrupt act as an NMI
- Add a PACA_HAPPENED_EE_EDGE for use by FSL if they want
to retrigger an interrupt without preventing hard-enable
v3:
- Fix or vs. ori bug on Book3E
- Fix enabling of interrupts for some exceptions on Book3E
v4:
- Fix resend of doorbells on return from interrupt on Book3E
v5:
- Rebased on top of my latest series, which involves some significant
rework of some aspects of the patch.
v6:
- 32-bit compile fix
- more compile fixes with various .config combos
- factor out the asm code to soft-disable interrupts
- remove the C wrapper around preempt_schedule_irq
v7:
- Fix a bug with hard irq state tracking on native power7
2012-03-06 07:27:59 +00:00
|
|
|
|
|
|
|
|
/* Check if something happened while soft-disabled */
|
|
|
|
|
lbz r0,PACAIRQHAPPENED(r13)
|
powerpc/powernv: Don't call generic code on offline cpus
On PowerNV platforms, when a CPU is offline, we put it into nap mode.
It's possible that the CPU wakes up from nap mode while it is still
offline due to a stray IPI. A misdirected device interrupt could also
potentially cause it to wake up. In that circumstance, we need to clear
the interrupt so that the CPU can go back to nap mode.
In the past the clearing of the interrupt was accomplished by briefly
enabling interrupts and allowing the normal interrupt handling code
(do_IRQ() etc.) to handle the interrupt. This has the problem that
this code calls irq_enter() and irq_exit(), which call functions such
as account_system_vtime() which use RCU internally. Use of RCU is not
permitted on offline CPUs and will trigger errors if RCU checking is
enabled.
To avoid calling into any generic code which might use RCU, we adopt
a different method of clearing interrupts on offline CPUs. Since we
are on the PowerNV platform, we know that the system interrupt
controller is a XICS being driven directly (i.e. not via hcalls) by
the kernel. Hence this adds a new icp_native_flush_interrupt()
function to the native-mode XICS driver and arranges to call that
when an offline CPU is woken from nap. This new function reads the
interrupt from the XICS. If it is an IPI, it clears the IPI; if it
is a device interrupt, it prints a warning and disables the source.
Then it does the end-of-interrupt processing for the interrupt.
The other thing that briefly enabling interrupts did was to check and
clear the irq_happened flag in this CPU's PACA. Therefore, after
flushing the interrupt from the XICS, we also clear all bits except
the PACA_IRQ_HARD_DIS (interrupts are hard disabled) bit from the
irq_happened flag. The PACA_IRQ_HARD_DIS flag is set by power7_nap()
and is left set to indicate that interrupts are hard disabled. This
means we then have to ignore that flag in power7_nap(), which is
reasonable since it doesn't indicate that any interrupt event needs
servicing.
Signed-off-by: Paul Mackerras <paulus@samba.org>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2014-09-02 04:23:16 +00:00
|
|
|
andi. r0,r0,~PACA_IRQ_HARD_DIS@l
|
powerpc: Rework lazy-interrupt handling
The current implementation of lazy interrupts handling has some
issues that this tries to address.
We don't do the various workarounds we need to do when re-enabling
interrupts in some cases such as when returning from an interrupt
and thus we may still lose or get delayed decrementer or doorbell
interrupts.
The current scheme also makes it much harder to handle the external
"edge" interrupts provided by some BookE processors when using the
EPR facility (External Proxy) and the Freescale Hypervisor.
Additionally, we tend to keep interrupts hard disabled in a number
of cases, such as decrementer interrupts, external interrupts, or
when a masked decrementer interrupt is pending. This is sub-optimal.
This is an attempt at fixing it all in one go by reworking the way
we do the lazy interrupt disabling from the ground up.
The base idea is to replace the "hard_enabled" field with a
"irq_happened" field in which we store a bit mask of what interrupt
occurred while soft-disabled.
When re-enabling, either via arch_local_irq_restore() or when returning
from an interrupt, we can now decide what to do by testing bits in that
field.
We then implement replaying of the missed interrupts either by
re-using the existing exception frame (in exception exit case) or via
the creation of a new one from an assembly trampoline (in the
arch_local_irq_enable case).
This removes the need to play with the decrementer to try to create
fake interrupts, among others.
In addition, this adds a few refinements:
- We no longer hard disable decrementer interrupts that occur
while soft-disabled. We now simply bump the decrementer back to max
(on BookS) or leave it stopped (on BookE) and continue with hard interrupts
enabled, which means that we'll potentially get better sample quality from
performance monitor interrupts.
- Timer, decrementer and doorbell interrupts now hard-enable
shortly after removing the source of the interrupt, which means
they no longer run entirely hard disabled. Again, this will improve
perf sample quality.
- On Book3E 64-bit, we now make the performance monitor interrupt
act as an NMI like Book3S (the necessary C code for that to work
appear to already be present in the FSL perf code, notably calling
nmi_enter instead of irq_enter). (This also fixes a bug where BookE
perfmon interrupts could clobber r14 ... oops)
- We could make "masked" decrementer interrupts act as NMIs when doing
timer-based perf sampling to improve the sample quality.
Signed-off-by-yet: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
v2:
- Add hard-enable to decrementer, timer and doorbells
- Fix CR clobber in masked irq handling on BookE
- Make embedded perf interrupt act as an NMI
- Add a PACA_HAPPENED_EE_EDGE for use by FSL if they want
to retrigger an interrupt without preventing hard-enable
v3:
- Fix or vs. ori bug on Book3E
- Fix enabling of interrupts for some exceptions on Book3E
v4:
- Fix resend of doorbells on return from interrupt on Book3E
v5:
- Rebased on top of my latest series, which involves some significant
rework of some aspects of the patch.
v6:
- 32-bit compile fix
- more compile fixes with various .config combos
- factor out the asm code to soft-disable interrupts
- remove the C wrapper around preempt_schedule_irq
v7:
- Fix a bug with hard irq state tracking on native power7
2012-03-06 07:27:59 +00:00
|
|
|
beq 1f
|
2014-05-23 08:15:26 +00:00
|
|
|
cmpwi cr0,r4,0
|
|
|
|
|
beq 1f
|
powerpc: Rework lazy-interrupt handling
The current implementation of lazy interrupts handling has some
issues that this tries to address.
We don't do the various workarounds we need to do when re-enabling
interrupts in some cases such as when returning from an interrupt
and thus we may still lose or get delayed decrementer or doorbell
interrupts.
The current scheme also makes it much harder to handle the external
"edge" interrupts provided by some BookE processors when using the
EPR facility (External Proxy) and the Freescale Hypervisor.
Additionally, we tend to keep interrupts hard disabled in a number
of cases, such as decrementer interrupts, external interrupts, or
when a masked decrementer interrupt is pending. This is sub-optimal.
This is an attempt at fixing it all in one go by reworking the way
we do the lazy interrupt disabling from the ground up.
The base idea is to replace the "hard_enabled" field with a
"irq_happened" field in which we store a bit mask of what interrupt
occurred while soft-disabled.
When re-enabling, either via arch_local_irq_restore() or when returning
from an interrupt, we can now decide what to do by testing bits in that
field.
We then implement replaying of the missed interrupts either by
re-using the existing exception frame (in exception exit case) or via
the creation of a new one from an assembly trampoline (in the
arch_local_irq_enable case).
This removes the need to play with the decrementer to try to create
fake interrupts, among others.
In addition, this adds a few refinements:
- We no longer hard disable decrementer interrupts that occur
while soft-disabled. We now simply bump the decrementer back to max
(on BookS) or leave it stopped (on BookE) and continue with hard interrupts
enabled, which means that we'll potentially get better sample quality from
performance monitor interrupts.
- Timer, decrementer and doorbell interrupts now hard-enable
shortly after removing the source of the interrupt, which means
they no longer run entirely hard disabled. Again, this will improve
perf sample quality.
- On Book3E 64-bit, we now make the performance monitor interrupt
act as an NMI like Book3S (the necessary C code for that to work
appear to already be present in the FSL perf code, notably calling
nmi_enter instead of irq_enter). (This also fixes a bug where BookE
perfmon interrupts could clobber r14 ... oops)
- We could make "masked" decrementer interrupts act as NMIs when doing
timer-based perf sampling to improve the sample quality.
Signed-off-by-yet: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
v2:
- Add hard-enable to decrementer, timer and doorbells
- Fix CR clobber in masked irq handling on BookE
- Make embedded perf interrupt act as an NMI
- Add a PACA_HAPPENED_EE_EDGE for use by FSL if they want
to retrigger an interrupt without preventing hard-enable
v3:
- Fix or vs. ori bug on Book3E
- Fix enabling of interrupts for some exceptions on Book3E
v4:
- Fix resend of doorbells on return from interrupt on Book3E
v5:
- Rebased on top of my latest series, which involves some significant
rework of some aspects of the patch.
v6:
- 32-bit compile fix
- more compile fixes with various .config combos
- factor out the asm code to soft-disable interrupts
- remove the C wrapper around preempt_schedule_irq
v7:
- Fix a bug with hard irq state tracking on native power7
2012-03-06 07:27:59 +00:00
|
|
|
addi r1,r1,INT_FRAME_SIZE
|
|
|
|
|
ld r0,16(r1)
|
2015-03-19 23:10:18 +00:00
|
|
|
li r3,0 /* Return 0 (no nap) */
|
powerpc: Rework lazy-interrupt handling
The current implementation of lazy interrupts handling has some
issues that this tries to address.
We don't do the various workarounds we need to do when re-enabling
interrupts in some cases such as when returning from an interrupt
and thus we may still lose or get delayed decrementer or doorbell
interrupts.
The current scheme also makes it much harder to handle the external
"edge" interrupts provided by some BookE processors when using the
EPR facility (External Proxy) and the Freescale Hypervisor.
Additionally, we tend to keep interrupts hard disabled in a number
of cases, such as decrementer interrupts, external interrupts, or
when a masked decrementer interrupt is pending. This is sub-optimal.
This is an attempt at fixing it all in one go by reworking the way
we do the lazy interrupt disabling from the ground up.
The base idea is to replace the "hard_enabled" field with a
"irq_happened" field in which we store a bit mask of what interrupt
occurred while soft-disabled.
When re-enabling, either via arch_local_irq_restore() or when returning
from an interrupt, we can now decide what to do by testing bits in that
field.
We then implement replaying of the missed interrupts either by
re-using the existing exception frame (in exception exit case) or via
the creation of a new one from an assembly trampoline (in the
arch_local_irq_enable case).
This removes the need to play with the decrementer to try to create
fake interrupts, among others.
In addition, this adds a few refinements:
- We no longer hard disable decrementer interrupts that occur
while soft-disabled. We now simply bump the decrementer back to max
(on BookS) or leave it stopped (on BookE) and continue with hard interrupts
enabled, which means that we'll potentially get better sample quality from
performance monitor interrupts.
- Timer, decrementer and doorbell interrupts now hard-enable
shortly after removing the source of the interrupt, which means
they no longer run entirely hard disabled. Again, this will improve
perf sample quality.
- On Book3E 64-bit, we now make the performance monitor interrupt
act as an NMI like Book3S (the necessary C code for that to work
appear to already be present in the FSL perf code, notably calling
nmi_enter instead of irq_enter). (This also fixes a bug where BookE
perfmon interrupts could clobber r14 ... oops)
- We could make "masked" decrementer interrupts act as NMIs when doing
timer-based perf sampling to improve the sample quality.
Signed-off-by-yet: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
v2:
- Add hard-enable to decrementer, timer and doorbells
- Fix CR clobber in masked irq handling on BookE
- Make embedded perf interrupt act as an NMI
- Add a PACA_HAPPENED_EE_EDGE for use by FSL if they want
to retrigger an interrupt without preventing hard-enable
v3:
- Fix or vs. ori bug on Book3E
- Fix enabling of interrupts for some exceptions on Book3E
v4:
- Fix resend of doorbells on return from interrupt on Book3E
v5:
- Rebased on top of my latest series, which involves some significant
rework of some aspects of the patch.
v6:
- 32-bit compile fix
- more compile fixes with various .config combos
- factor out the asm code to soft-disable interrupts
- remove the C wrapper around preempt_schedule_irq
v7:
- Fix a bug with hard irq state tracking on native power7
2012-03-06 07:27:59 +00:00
|
|
|
mtlr r0
|
|
|
|
|
blr
|
|
|
|
|
|
|
|
|
|
1: /* We mark irqs hard disabled as this is the state we'll
|
|
|
|
|
* be in when returning and we need to tell arch_local_irq_restore()
|
|
|
|
|
* about it
|
|
|
|
|
*/
|
|
|
|
|
li r0,PACA_IRQ_HARD_DIS
|
|
|
|
|
stb r0,PACAIRQHAPPENED(r13)
|
|
|
|
|
|
|
|
|
|
/* We haven't lost state ... yet */
|
2011-01-24 07:42:41 +00:00
|
|
|
li r0,0
|
2011-12-05 19:47:26 +00:00
|
|
|
stb r0,PACA_NAPSTATELOST(r13)
|
2011-01-24 07:42:41 +00:00
|
|
|
|
|
|
|
|
/* Continue saving state */
|
|
|
|
|
SAVE_GPR(2, r1)
|
|
|
|
|
SAVE_NVGPRS(r1)
|
2014-02-26 00:08:25 +00:00
|
|
|
mfcr r4
|
|
|
|
|
std r4,_CCR(r1)
|
2011-01-24 07:42:41 +00:00
|
|
|
std r9,_MSR(r1)
|
|
|
|
|
std r1,PACAR1(r13)
|
|
|
|
|
|
powerpc/powernv: Switch off MMU before entering nap/sleep/rvwinkle mode
Currently, when going idle, we set the flag indicating that we are in
nap mode (paca->kvm_hstate.hwthread_state) and then execute the nap
(or sleep or rvwinkle) instruction, all with the MMU on. This is bad
for two reasons: (a) the architecture specifies that those instructions
must be executed with the MMU off, and in fact with only the SF, HV, ME
and possibly RI bits set, and (b) this introduces a race, because as
soon as we set the flag, another thread can switch the MMU to a guest
context. If the race is lost, this thread will typically start looping
on relocation-on ISIs at 0xc...4400.
This fixes it by setting the MSR as required by the architecture before
setting the flag or executing the nap/sleep/rvwinkle instruction.
Cc: stable@vger.kernel.org
[ shreyas@linux.vnet.ibm.com: Edited to handle LE ]
Signed-off-by: Paul Mackerras <paulus@samba.org>
Signed-off-by: Shreyas B. Prabhu <shreyas@linux.vnet.ibm.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: linuxppc-dev@lists.ozlabs.org
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2014-12-09 18:56:50 +00:00
|
|
|
/*
|
|
|
|
|
* Go to real mode to do the nap, as required by the architecture.
|
|
|
|
|
* Also, we need to be in real mode before setting hwthread_state,
|
|
|
|
|
* because as soon as we do that, another thread can switch
|
|
|
|
|
* the MMU context to the guest.
|
|
|
|
|
*/
|
|
|
|
|
LOAD_REG_IMMEDIATE(r5, MSR_IDLE)
|
|
|
|
|
li r6, MSR_RI
|
|
|
|
|
andc r6, r9, r6
|
|
|
|
|
LOAD_REG_ADDR(r7, power7_enter_nap_mode)
|
|
|
|
|
mtmsrd r6, 1 /* clear RI before setting SRR0/1 */
|
|
|
|
|
mtspr SPRN_SRR0, r7
|
|
|
|
|
mtspr SPRN_SRR1, r5
|
|
|
|
|
rfid
|
|
|
|
|
|
|
|
|
|
.globl power7_enter_nap_mode
|
|
|
|
|
power7_enter_nap_mode:
|
2013-10-07 16:47:52 +00:00
|
|
|
#ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE
|
2012-02-03 00:54:17 +00:00
|
|
|
/* Tell KVM we're napping */
|
|
|
|
|
li r4,KVM_HWTHREAD_IN_NAP
|
|
|
|
|
stb r4,HSTATE_HWTHREAD_STATE(r13)
|
|
|
|
|
#endif
|
2014-12-09 18:56:52 +00:00
|
|
|
stb r3,PACA_THREAD_IDLE_STATE(r13)
|
2014-12-09 18:56:53 +00:00
|
|
|
cmpwi cr3,r3,PNV_THREAD_SLEEP
|
|
|
|
|
bge cr3,2f
|
2014-02-26 00:08:25 +00:00
|
|
|
IDLE_STATE_ENTER_SEQ(PPC_NAP)
|
|
|
|
|
/* No return */
|
2014-12-09 18:56:52 +00:00
|
|
|
2:
|
|
|
|
|
/* Sleep or winkle */
|
|
|
|
|
lbz r7,PACA_THREAD_MASK(r13)
|
|
|
|
|
ld r14,PACA_CORE_IDLE_STATE_PTR(r13)
|
|
|
|
|
lwarx_loop1:
|
|
|
|
|
lwarx r15,0,r14
|
powerpc/powernv: Fix race in updating core_idle_state
core_idle_state is maintained for each core. It uses 0-7 bits to track
whether a thread in the core has entered fastsleep or winkle. 8th bit is
used as a lock bit.
The lock bit is set in these 2 scenarios-
- The thread is first in subcore to wakeup from sleep/winkle.
- If its the last thread in the core about to enter sleep/winkle
While the lock bit is set, if any other thread in the core wakes up, it
loops until the lock bit is cleared before proceeding in the wakeup
path. This helps prevent race conditions w.r.t fastsleep workaround and
prevents threads from switching to process context before core/subcore
resources are restored.
But, in the path to sleep/winkle entry, we currently don't check for
lock-bit. This exposes us to following race when running with subcore
on-
First thread in the subcorea Another thread in the same
waking up core entering sleep/winkle
lwarx r15,0,r14
ori r15,r15,PNV_CORE_IDLE_LOCK_BIT
stwcx. r15,0,r14
[Code to restore subcore state]
lwarx r15,0,r14
[clear thread bit]
stwcx. r15,0,r14
andi. r15,r15,PNV_CORE_IDLE_THREAD_BITS
stw r15,0(r14)
Here, after the thread entering sleep clears its thread bit in
core_idle_state, the value is overwritten by the thread waking up.
In such cases when the core enters fastsleep, code mistakes an idle
thread as running. Because of this, the first thread waking up from
fastsleep which is supposed to resync timebase skips it. So we can
end up having a core with stale timebase value.
This patch fixes the above race by looping on the lock bit even while
entering the idle states.
Signed-off-by: Shreyas B. Prabhu <shreyas@linux.vnet.ibm.com>
Fixes: 7b54e9f213f76 'powernv/powerpc: Add winkle support for offline cpus'
Cc: stable@vger.kernel.org # 3.19+
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2015-07-06 20:09:23 +00:00
|
|
|
|
|
|
|
|
andi. r9,r15,PNV_CORE_IDLE_LOCK_BIT
|
|
|
|
|
bnel core_idle_lock_held
|
|
|
|
|
|
2014-12-09 18:56:52 +00:00
|
|
|
andc r15,r15,r7 /* Clear thread bit */
|
|
|
|
|
|
|
|
|
|
andi. r15,r15,PNV_CORE_IDLE_THREAD_BITS
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If cr0 = 0, then current thread is the last thread of the core entering
|
|
|
|
|
* sleep. Last thread needs to execute the hardware bug workaround code if
|
|
|
|
|
* required by the platform.
|
|
|
|
|
* Make the workaround call unconditionally here. The below branch call is
|
|
|
|
|
* patched out when the idle states are discovered if the platform does not
|
|
|
|
|
* require it.
|
|
|
|
|
*/
|
|
|
|
|
.global pnv_fastsleep_workaround_at_entry
|
|
|
|
|
pnv_fastsleep_workaround_at_entry:
|
|
|
|
|
beq fastsleep_workaround_at_entry
|
|
|
|
|
|
|
|
|
|
stwcx. r15,0,r14
|
|
|
|
|
bne- lwarx_loop1
|
|
|
|
|
isync
|
|
|
|
|
|
2014-12-09 18:56:53 +00:00
|
|
|
common_enter: /* common code for all the threads entering sleep or winkle */
|
|
|
|
|
bgt cr3,enter_winkle
|
2014-12-09 18:56:52 +00:00
|
|
|
IDLE_STATE_ENTER_SEQ(PPC_SLEEP)
|
|
|
|
|
|
|
|
|
|
fastsleep_workaround_at_entry:
|
|
|
|
|
ori r15,r15,PNV_CORE_IDLE_LOCK_BIT
|
|
|
|
|
stwcx. r15,0,r14
|
|
|
|
|
bne- lwarx_loop1
|
|
|
|
|
isync
|
|
|
|
|
|
|
|
|
|
/* Fast sleep workaround */
|
|
|
|
|
li r3,1
|
|
|
|
|
li r4,1
|
|
|
|
|
li r0,OPAL_CONFIG_CPU_IDLE_STATE
|
|
|
|
|
bl opal_call_realmode
|
|
|
|
|
|
|
|
|
|
/* Clear Lock bit */
|
|
|
|
|
li r0,0
|
|
|
|
|
lwsync
|
|
|
|
|
stw r0,0(r14)
|
|
|
|
|
b common_enter
|
|
|
|
|
|
2014-12-09 18:56:53 +00:00
|
|
|
enter_winkle:
|
|
|
|
|
/*
|
|
|
|
|
* Note all register i.e per-core, per-subcore or per-thread is saved
|
|
|
|
|
* here since any thread in the core might wake up first
|
|
|
|
|
*/
|
|
|
|
|
mfspr r3,SPRN_SDR1
|
|
|
|
|
std r3,_SDR1(r1)
|
|
|
|
|
mfspr r3,SPRN_RPR
|
|
|
|
|
std r3,_RPR(r1)
|
|
|
|
|
mfspr r3,SPRN_SPURR
|
|
|
|
|
std r3,_SPURR(r1)
|
|
|
|
|
mfspr r3,SPRN_PURR
|
|
|
|
|
std r3,_PURR(r1)
|
|
|
|
|
mfspr r3,SPRN_TSCR
|
|
|
|
|
std r3,_TSCR(r1)
|
|
|
|
|
mfspr r3,SPRN_DSCR
|
|
|
|
|
std r3,_DSCR(r1)
|
|
|
|
|
mfspr r3,SPRN_AMOR
|
|
|
|
|
std r3,_AMOR(r1)
|
|
|
|
|
mfspr r3,SPRN_WORT
|
|
|
|
|
std r3,_WORT(r1)
|
|
|
|
|
mfspr r3,SPRN_WORC
|
|
|
|
|
std r3,_WORC(r1)
|
|
|
|
|
IDLE_STATE_ENTER_SEQ(PPC_WINKLE)
|
2012-02-03 00:54:17 +00:00
|
|
|
|
2014-02-26 00:08:25 +00:00
|
|
|
_GLOBAL(power7_idle)
|
|
|
|
|
/* Now check if user or arch enabled NAP mode */
|
|
|
|
|
LOAD_REG_ADDRBASE(r3,powersave_nap)
|
|
|
|
|
lwz r4,ADDROFF(powersave_nap)(r3)
|
|
|
|
|
cmpwi 0,r4,0
|
|
|
|
|
beqlr
|
2014-05-23 08:15:26 +00:00
|
|
|
li r3, 1
|
2014-02-26 00:08:25 +00:00
|
|
|
/* fall through */
|
|
|
|
|
|
|
|
|
|
_GLOBAL(power7_nap)
|
2014-05-23 08:15:26 +00:00
|
|
|
mr r4,r3
|
2014-12-09 18:56:52 +00:00
|
|
|
li r3,PNV_THREAD_NAP
|
2014-02-26 00:08:25 +00:00
|
|
|
b power7_powersave_common
|
|
|
|
|
/* No return */
|
|
|
|
|
|
|
|
|
|
_GLOBAL(power7_sleep)
|
2014-12-09 18:56:52 +00:00
|
|
|
li r3,PNV_THREAD_SLEEP
|
2014-07-02 03:49:35 +00:00
|
|
|
li r4,1
|
2014-02-26 00:08:25 +00:00
|
|
|
b power7_powersave_common
|
|
|
|
|
/* No return */
|
2011-01-24 07:42:41 +00:00
|
|
|
|
2014-12-09 18:56:53 +00:00
|
|
|
_GLOBAL(power7_winkle)
|
|
|
|
|
li r3,3
|
|
|
|
|
li r4,1
|
|
|
|
|
b power7_powersave_common
|
|
|
|
|
/* No return */
|
|
|
|
|
|
2014-07-29 13:10:13 +00:00
|
|
|
#define CHECK_HMI_INTERRUPT \
|
|
|
|
|
mfspr r0,SPRN_SRR1; \
|
|
|
|
|
BEGIN_FTR_SECTION_NESTED(66); \
|
|
|
|
|
rlwinm r0,r0,45-31,0xf; /* extract wake reason field (P8) */ \
|
|
|
|
|
FTR_SECTION_ELSE_NESTED(66); \
|
|
|
|
|
rlwinm r0,r0,45-31,0xe; /* P7 wake reason field is 3 bits */ \
|
|
|
|
|
ALT_FTR_SECTION_END_NESTED_IFSET(CPU_FTR_ARCH_207S, 66); \
|
|
|
|
|
cmpwi r0,0xa; /* Hypervisor maintenance ? */ \
|
|
|
|
|
bne 20f; \
|
|
|
|
|
/* Invoke opal call to handle hmi */ \
|
|
|
|
|
ld r2,PACATOC(r13); \
|
|
|
|
|
ld r1,PACAR1(r13); \
|
|
|
|
|
std r3,ORIG_GPR3(r1); /* Save original r3 */ \
|
2014-12-09 18:56:52 +00:00
|
|
|
li r0,OPAL_HANDLE_HMI; /* Pass opal token argument*/ \
|
2014-07-31 12:47:52 +00:00
|
|
|
bl opal_call_realmode; \
|
2014-07-29 13:10:13 +00:00
|
|
|
ld r3,ORIG_GPR3(r1); /* Restore original r3 */ \
|
|
|
|
|
20: nop;
|
|
|
|
|
|
|
|
|
|
|
2014-02-26 00:08:43 +00:00
|
|
|
_GLOBAL(power7_wakeup_tb_loss)
|
|
|
|
|
ld r2,PACATOC(r13);
|
|
|
|
|
ld r1,PACAR1(r13)
|
2014-12-09 18:56:52 +00:00
|
|
|
/*
|
|
|
|
|
* Before entering any idle state, the NVGPRs are saved in the stack
|
|
|
|
|
* and they are restored before switching to the process context. Hence
|
|
|
|
|
* until they are restored, they are free to be used.
|
|
|
|
|
*
|
|
|
|
|
* Save SRR1 in a NVGPR as it might be clobbered in opal_call_realmode
|
|
|
|
|
* (called in CHECK_HMI_INTERRUPT). SRR1 is required to determine the
|
|
|
|
|
* wakeup reason if we branch to kvm_start_guest.
|
|
|
|
|
*/
|
2014-02-26 00:08:43 +00:00
|
|
|
|
2014-12-09 18:56:52 +00:00
|
|
|
mfspr r16,SPRN_SRR1
|
2014-07-29 13:10:13 +00:00
|
|
|
BEGIN_FTR_SECTION
|
|
|
|
|
CHECK_HMI_INTERRUPT
|
|
|
|
|
END_FTR_SECTION_IFSET(CPU_FTR_HVMODE)
|
2014-12-09 18:56:52 +00:00
|
|
|
|
|
|
|
|
lbz r7,PACA_THREAD_MASK(r13)
|
|
|
|
|
ld r14,PACA_CORE_IDLE_STATE_PTR(r13)
|
|
|
|
|
lwarx_loop2:
|
|
|
|
|
lwarx r15,0,r14
|
|
|
|
|
andi. r9,r15,PNV_CORE_IDLE_LOCK_BIT
|
|
|
|
|
/*
|
|
|
|
|
* Lock bit is set in one of the 2 cases-
|
|
|
|
|
* a. In the sleep/winkle enter path, the last thread is executing
|
|
|
|
|
* fastsleep workaround code.
|
|
|
|
|
* b. In the wake up path, another thread is executing fastsleep
|
|
|
|
|
* workaround undo code or resyncing timebase or restoring context
|
|
|
|
|
* In either case loop until the lock bit is cleared.
|
|
|
|
|
*/
|
powerpc/powernv: Fix race in updating core_idle_state
core_idle_state is maintained for each core. It uses 0-7 bits to track
whether a thread in the core has entered fastsleep or winkle. 8th bit is
used as a lock bit.
The lock bit is set in these 2 scenarios-
- The thread is first in subcore to wakeup from sleep/winkle.
- If its the last thread in the core about to enter sleep/winkle
While the lock bit is set, if any other thread in the core wakes up, it
loops until the lock bit is cleared before proceeding in the wakeup
path. This helps prevent race conditions w.r.t fastsleep workaround and
prevents threads from switching to process context before core/subcore
resources are restored.
But, in the path to sleep/winkle entry, we currently don't check for
lock-bit. This exposes us to following race when running with subcore
on-
First thread in the subcorea Another thread in the same
waking up core entering sleep/winkle
lwarx r15,0,r14
ori r15,r15,PNV_CORE_IDLE_LOCK_BIT
stwcx. r15,0,r14
[Code to restore subcore state]
lwarx r15,0,r14
[clear thread bit]
stwcx. r15,0,r14
andi. r15,r15,PNV_CORE_IDLE_THREAD_BITS
stw r15,0(r14)
Here, after the thread entering sleep clears its thread bit in
core_idle_state, the value is overwritten by the thread waking up.
In such cases when the core enters fastsleep, code mistakes an idle
thread as running. Because of this, the first thread waking up from
fastsleep which is supposed to resync timebase skips it. So we can
end up having a core with stale timebase value.
This patch fixes the above race by looping on the lock bit even while
entering the idle states.
Signed-off-by: Shreyas B. Prabhu <shreyas@linux.vnet.ibm.com>
Fixes: 7b54e9f213f76 'powernv/powerpc: Add winkle support for offline cpus'
Cc: stable@vger.kernel.org # 3.19+
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2015-07-06 20:09:23 +00:00
|
|
|
bnel core_idle_lock_held
|
2014-12-09 18:56:52 +00:00
|
|
|
|
|
|
|
|
cmpwi cr2,r15,0
|
2014-12-09 18:56:53 +00:00
|
|
|
lbz r4,PACA_SUBCORE_SIBLING_MASK(r13)
|
|
|
|
|
and r4,r4,r15
|
|
|
|
|
cmpwi cr1,r4,0 /* Check if first in subcore */
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* At this stage
|
|
|
|
|
* cr1 - 0b0100 if first thread to wakeup in subcore
|
|
|
|
|
* cr2 - 0b0100 if first thread to wakeup in core
|
|
|
|
|
* cr3- 0b0010 if waking up from sleep or winkle
|
|
|
|
|
* cr4 - 0b0100 if waking up from winkle
|
|
|
|
|
*/
|
|
|
|
|
|
2014-12-09 18:56:52 +00:00
|
|
|
or r15,r15,r7 /* Set thread bit */
|
|
|
|
|
|
2014-12-09 18:56:53 +00:00
|
|
|
beq cr1,first_thread_in_subcore
|
2014-12-09 18:56:52 +00:00
|
|
|
|
2014-12-09 18:56:53 +00:00
|
|
|
/* Not first thread in subcore to wake up */
|
2014-12-09 18:56:52 +00:00
|
|
|
stwcx. r15,0,r14
|
|
|
|
|
bne- lwarx_loop2
|
|
|
|
|
isync
|
|
|
|
|
b common_exit
|
|
|
|
|
|
2014-12-09 18:56:53 +00:00
|
|
|
first_thread_in_subcore:
|
|
|
|
|
/* First thread in subcore to wakeup */
|
2014-12-09 18:56:52 +00:00
|
|
|
ori r15,r15,PNV_CORE_IDLE_LOCK_BIT
|
|
|
|
|
stwcx. r15,0,r14
|
|
|
|
|
bne- lwarx_loop2
|
|
|
|
|
isync
|
|
|
|
|
|
2014-12-09 18:56:53 +00:00
|
|
|
/*
|
|
|
|
|
* If waking up from sleep, subcore state is not lost. Hence
|
|
|
|
|
* skip subcore state restore
|
|
|
|
|
*/
|
|
|
|
|
bne cr4,subcore_state_restored
|
|
|
|
|
|
|
|
|
|
/* Restore per-subcore state */
|
|
|
|
|
ld r4,_SDR1(r1)
|
|
|
|
|
mtspr SPRN_SDR1,r4
|
|
|
|
|
ld r4,_RPR(r1)
|
|
|
|
|
mtspr SPRN_RPR,r4
|
|
|
|
|
ld r4,_AMOR(r1)
|
|
|
|
|
mtspr SPRN_AMOR,r4
|
|
|
|
|
|
|
|
|
|
subcore_state_restored:
|
|
|
|
|
/*
|
|
|
|
|
* Check if the thread is also the first thread in the core. If not,
|
|
|
|
|
* skip to clear_lock.
|
|
|
|
|
*/
|
|
|
|
|
bne cr2,clear_lock
|
|
|
|
|
|
|
|
|
|
first_thread_in_core:
|
|
|
|
|
|
2014-12-09 18:56:52 +00:00
|
|
|
/*
|
|
|
|
|
* First thread in the core waking up from fastsleep. It needs to
|
|
|
|
|
* call the fastsleep workaround code if the platform requires it.
|
|
|
|
|
* Call it unconditionally here. The below branch instruction will
|
|
|
|
|
* be patched out when the idle states are discovered if platform
|
|
|
|
|
* does not require workaround.
|
|
|
|
|
*/
|
|
|
|
|
.global pnv_fastsleep_workaround_at_exit
|
|
|
|
|
pnv_fastsleep_workaround_at_exit:
|
|
|
|
|
b fastsleep_workaround_at_exit
|
|
|
|
|
|
|
|
|
|
timebase_resync:
|
|
|
|
|
/* Do timebase resync if we are waking up from sleep. Use cr3 value
|
|
|
|
|
* set in exceptions-64s.S */
|
|
|
|
|
ble cr3,clear_lock
|
2014-02-26 00:08:43 +00:00
|
|
|
/* Time base re-sync */
|
2014-12-09 18:56:52 +00:00
|
|
|
li r0,OPAL_RESYNC_TIMEBASE
|
2014-07-31 12:47:52 +00:00
|
|
|
bl opal_call_realmode;
|
2014-02-26 00:08:43 +00:00
|
|
|
/* TODO: Check r3 for failure */
|
|
|
|
|
|
2014-12-09 18:56:53 +00:00
|
|
|
/*
|
|
|
|
|
* If waking up from sleep, per core state is not lost, skip to
|
|
|
|
|
* clear_lock.
|
|
|
|
|
*/
|
|
|
|
|
bne cr4,clear_lock
|
|
|
|
|
|
|
|
|
|
/* Restore per core state */
|
|
|
|
|
ld r4,_TSCR(r1)
|
|
|
|
|
mtspr SPRN_TSCR,r4
|
|
|
|
|
ld r4,_WORC(r1)
|
|
|
|
|
mtspr SPRN_WORC,r4
|
|
|
|
|
|
2014-12-09 18:56:52 +00:00
|
|
|
clear_lock:
|
|
|
|
|
andi. r15,r15,PNV_CORE_IDLE_THREAD_BITS
|
|
|
|
|
lwsync
|
|
|
|
|
stw r15,0(r14)
|
|
|
|
|
|
|
|
|
|
common_exit:
|
2014-12-09 18:56:53 +00:00
|
|
|
/*
|
|
|
|
|
* Common to all threads.
|
|
|
|
|
*
|
|
|
|
|
* If waking up from sleep, hypervisor state is not lost. Hence
|
|
|
|
|
* skip hypervisor state restore.
|
|
|
|
|
*/
|
|
|
|
|
bne cr4,hypervisor_state_restored
|
|
|
|
|
|
|
|
|
|
/* Waking up from winkle */
|
|
|
|
|
|
|
|
|
|
/* Restore per thread state */
|
|
|
|
|
bl __restore_cpu_power8
|
|
|
|
|
|
|
|
|
|
/* Restore SLB from PACA */
|
|
|
|
|
ld r8,PACA_SLBSHADOWPTR(r13)
|
|
|
|
|
|
|
|
|
|
.rept SLB_NUM_BOLTED
|
|
|
|
|
li r3, SLBSHADOW_SAVEAREA
|
|
|
|
|
LDX_BE r5, r8, r3
|
|
|
|
|
addi r3, r3, 8
|
|
|
|
|
LDX_BE r6, r8, r3
|
|
|
|
|
andis. r7,r5,SLB_ESID_V@h
|
|
|
|
|
beq 1f
|
|
|
|
|
slbmte r6,r5
|
|
|
|
|
1: addi r8,r8,16
|
|
|
|
|
.endr
|
|
|
|
|
|
|
|
|
|
ld r4,_SPURR(r1)
|
|
|
|
|
mtspr SPRN_SPURR,r4
|
|
|
|
|
ld r4,_PURR(r1)
|
|
|
|
|
mtspr SPRN_PURR,r4
|
|
|
|
|
ld r4,_DSCR(r1)
|
|
|
|
|
mtspr SPRN_DSCR,r4
|
|
|
|
|
ld r4,_WORT(r1)
|
|
|
|
|
mtspr SPRN_WORT,r4
|
|
|
|
|
|
|
|
|
|
hypervisor_state_restored:
|
|
|
|
|
|
2014-12-09 18:56:52 +00:00
|
|
|
li r5,PNV_THREAD_RUNNING
|
|
|
|
|
stb r5,PACA_THREAD_IDLE_STATE(r13)
|
|
|
|
|
|
|
|
|
|
mtspr SPRN_SRR1,r16
|
|
|
|
|
#ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE
|
|
|
|
|
li r0,KVM_HWTHREAD_IN_KERNEL
|
|
|
|
|
stb r0,HSTATE_HWTHREAD_STATE(r13)
|
|
|
|
|
/* Order setting hwthread_state vs. testing hwthread_req */
|
|
|
|
|
sync
|
|
|
|
|
lbz r0,HSTATE_HWTHREAD_REQ(r13)
|
|
|
|
|
cmpwi r0,0
|
|
|
|
|
beq 6f
|
|
|
|
|
b kvm_start_guest
|
|
|
|
|
6:
|
|
|
|
|
#endif
|
|
|
|
|
|
2014-02-26 00:08:43 +00:00
|
|
|
REST_NVGPRS(r1)
|
|
|
|
|
REST_GPR(2, r1)
|
|
|
|
|
ld r3,_CCR(r1)
|
|
|
|
|
ld r4,_MSR(r1)
|
|
|
|
|
ld r5,_NIP(r1)
|
|
|
|
|
addi r1,r1,INT_FRAME_SIZE
|
|
|
|
|
mtcr r3
|
|
|
|
|
mfspr r3,SPRN_SRR1 /* Return SRR1 */
|
|
|
|
|
mtspr SPRN_SRR1,r4
|
|
|
|
|
mtspr SPRN_SRR0,r5
|
|
|
|
|
rfid
|
|
|
|
|
|
2014-12-09 18:56:52 +00:00
|
|
|
fastsleep_workaround_at_exit:
|
|
|
|
|
li r3,1
|
|
|
|
|
li r4,0
|
|
|
|
|
li r0,OPAL_CONFIG_CPU_IDLE_STATE
|
|
|
|
|
bl opal_call_realmode
|
|
|
|
|
b timebase_resync
|
|
|
|
|
|
powerpc/powernv: Return to cpu offline loop when finished in KVM guest
When a secondary hardware thread has finished running a KVM guest, we
currently put that thread into nap mode using a nap instruction in
the KVM code. This changes the code so that instead of doing a nap
instruction directly, we instead cause the call to power7_nap() that
put the thread into nap mode to return. The reason for doing this is
to avoid having the KVM code having to know what low-power mode to
put the thread into.
In the case of a secondary thread used to run a KVM guest, the thread
will be offline from the point of view of the host kernel, and the
relevant power7_nap() call is the one in pnv_smp_cpu_disable().
In this case we don't want to clear pending IPIs in the offline loop
in that function, since that might cause us to miss the wakeup for
the next time the thread needs to run a guest. To tell whether or
not to clear the interrupt, we use the SRR1 value returned from
power7_nap(), and check if it indicates an external interrupt. We
arrange that the return from power7_nap() when we have finished running
a guest returns 0, so pending interrupts don't get flushed in that
case.
Note that it is important a secondary thread that has finished
executing in the guest, or that didn't have a guest to run, should
not return to power7_nap's caller while the kvm_hstate.hwthread_req
flag in the PACA is non-zero, because the return from power7_nap
will reenable the MMU, and the MMU might still be in guest context.
In this situation we spin at low priority in real mode waiting for
hwthread_req to become zero.
Signed-off-by: Paul Mackerras <paulus@samba.org>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2014-12-03 03:48:40 +00:00
|
|
|
/*
|
|
|
|
|
* R3 here contains the value that will be returned to the caller
|
|
|
|
|
* of power7_nap.
|
|
|
|
|
*/
|
2011-01-24 07:42:41 +00:00
|
|
|
_GLOBAL(power7_wakeup_loss)
|
|
|
|
|
ld r1,PACAR1(r13)
|
2014-07-29 13:10:13 +00:00
|
|
|
BEGIN_FTR_SECTION
|
|
|
|
|
CHECK_HMI_INTERRUPT
|
|
|
|
|
END_FTR_SECTION_IFSET(CPU_FTR_HVMODE)
|
2011-01-24 07:42:41 +00:00
|
|
|
REST_NVGPRS(r1)
|
|
|
|
|
REST_GPR(2, r1)
|
powerpc/powernv: Return to cpu offline loop when finished in KVM guest
When a secondary hardware thread has finished running a KVM guest, we
currently put that thread into nap mode using a nap instruction in
the KVM code. This changes the code so that instead of doing a nap
instruction directly, we instead cause the call to power7_nap() that
put the thread into nap mode to return. The reason for doing this is
to avoid having the KVM code having to know what low-power mode to
put the thread into.
In the case of a secondary thread used to run a KVM guest, the thread
will be offline from the point of view of the host kernel, and the
relevant power7_nap() call is the one in pnv_smp_cpu_disable().
In this case we don't want to clear pending IPIs in the offline loop
in that function, since that might cause us to miss the wakeup for
the next time the thread needs to run a guest. To tell whether or
not to clear the interrupt, we use the SRR1 value returned from
power7_nap(), and check if it indicates an external interrupt. We
arrange that the return from power7_nap() when we have finished running
a guest returns 0, so pending interrupts don't get flushed in that
case.
Note that it is important a secondary thread that has finished
executing in the guest, or that didn't have a guest to run, should
not return to power7_nap's caller while the kvm_hstate.hwthread_req
flag in the PACA is non-zero, because the return from power7_nap
will reenable the MMU, and the MMU might still be in guest context.
In this situation we spin at low priority in real mode waiting for
hwthread_req to become zero.
Signed-off-by: Paul Mackerras <paulus@samba.org>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2014-12-03 03:48:40 +00:00
|
|
|
ld r6,_CCR(r1)
|
2011-01-24 07:42:41 +00:00
|
|
|
ld r4,_MSR(r1)
|
|
|
|
|
ld r5,_NIP(r1)
|
|
|
|
|
addi r1,r1,INT_FRAME_SIZE
|
powerpc/powernv: Return to cpu offline loop when finished in KVM guest
When a secondary hardware thread has finished running a KVM guest, we
currently put that thread into nap mode using a nap instruction in
the KVM code. This changes the code so that instead of doing a nap
instruction directly, we instead cause the call to power7_nap() that
put the thread into nap mode to return. The reason for doing this is
to avoid having the KVM code having to know what low-power mode to
put the thread into.
In the case of a secondary thread used to run a KVM guest, the thread
will be offline from the point of view of the host kernel, and the
relevant power7_nap() call is the one in pnv_smp_cpu_disable().
In this case we don't want to clear pending IPIs in the offline loop
in that function, since that might cause us to miss the wakeup for
the next time the thread needs to run a guest. To tell whether or
not to clear the interrupt, we use the SRR1 value returned from
power7_nap(), and check if it indicates an external interrupt. We
arrange that the return from power7_nap() when we have finished running
a guest returns 0, so pending interrupts don't get flushed in that
case.
Note that it is important a secondary thread that has finished
executing in the guest, or that didn't have a guest to run, should
not return to power7_nap's caller while the kvm_hstate.hwthread_req
flag in the PACA is non-zero, because the return from power7_nap
will reenable the MMU, and the MMU might still be in guest context.
In this situation we spin at low priority in real mode waiting for
hwthread_req to become zero.
Signed-off-by: Paul Mackerras <paulus@samba.org>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2014-12-03 03:48:40 +00:00
|
|
|
mtcr r6
|
2011-01-24 07:42:41 +00:00
|
|
|
mtspr SPRN_SRR1,r4
|
|
|
|
|
mtspr SPRN_SRR0,r5
|
|
|
|
|
rfid
|
|
|
|
|
|
powerpc/powernv: Return to cpu offline loop when finished in KVM guest
When a secondary hardware thread has finished running a KVM guest, we
currently put that thread into nap mode using a nap instruction in
the KVM code. This changes the code so that instead of doing a nap
instruction directly, we instead cause the call to power7_nap() that
put the thread into nap mode to return. The reason for doing this is
to avoid having the KVM code having to know what low-power mode to
put the thread into.
In the case of a secondary thread used to run a KVM guest, the thread
will be offline from the point of view of the host kernel, and the
relevant power7_nap() call is the one in pnv_smp_cpu_disable().
In this case we don't want to clear pending IPIs in the offline loop
in that function, since that might cause us to miss the wakeup for
the next time the thread needs to run a guest. To tell whether or
not to clear the interrupt, we use the SRR1 value returned from
power7_nap(), and check if it indicates an external interrupt. We
arrange that the return from power7_nap() when we have finished running
a guest returns 0, so pending interrupts don't get flushed in that
case.
Note that it is important a secondary thread that has finished
executing in the guest, or that didn't have a guest to run, should
not return to power7_nap's caller while the kvm_hstate.hwthread_req
flag in the PACA is non-zero, because the return from power7_nap
will reenable the MMU, and the MMU might still be in guest context.
In this situation we spin at low priority in real mode waiting for
hwthread_req to become zero.
Signed-off-by: Paul Mackerras <paulus@samba.org>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2014-12-03 03:48:40 +00:00
|
|
|
/*
|
|
|
|
|
* R3 here contains the value that will be returned to the caller
|
|
|
|
|
* of power7_nap.
|
|
|
|
|
*/
|
2011-01-24 07:42:41 +00:00
|
|
|
_GLOBAL(power7_wakeup_noloss)
|
2011-12-05 19:47:26 +00:00
|
|
|
lbz r0,PACA_NAPSTATELOST(r13)
|
|
|
|
|
cmpwi r0,0
|
2014-02-04 05:04:35 +00:00
|
|
|
bne power7_wakeup_loss
|
2014-07-29 13:10:13 +00:00
|
|
|
BEGIN_FTR_SECTION
|
|
|
|
|
CHECK_HMI_INTERRUPT
|
|
|
|
|
END_FTR_SECTION_IFSET(CPU_FTR_HVMODE)
|
2011-01-24 07:42:41 +00:00
|
|
|
ld r1,PACAR1(r13)
|
powerpc/powernv: Restore non-volatile CRs after nap
Patches 7cba160ad "powernv/cpuidle: Redesign idle states management"
and 77b54e9f2 "powernv/powerpc: Add winkle support for offline cpus"
use non-volatile condition registers (cr2, cr3 and cr4) early in the system
reset interrupt handler (system_reset_pSeries()) before it has been determined
if state loss has occurred. If state loss has not occurred, control returns via
the power7_wakeup_noloss() path which does not restore those condition
registers, leaving them corrupted.
Fix this by restoring the condition registers in the power7_wakeup_noloss()
case.
This is apparent when running a KVM guest on hardware that does not
support winkle or sleep and the guest makes use of secondary threads. In
practice this means Power7 machines, though some early unreleased Power8
machines may also be susceptible.
The secondary CPUs are taken off line before the guest is started and
they call pnv_smp_cpu_kill_self(). This checks support for sleep
states (in this case there is no support) and power7_nap() is called.
When the CPU is woken, power7_nap() returns and because the CPU is
still off line, the main while loop executes again. The sleep states
support test is executed again, but because the tested values cannot
have changed, the compiler has optimized the test away and instead we
rely on the result of the first test, which has been left in cr3
and/or cr4. With the result overwritten, the wrong branch is taken and
power7_winkle() is called on a CPU that does not support it, leading
to it stalling.
Fixes: 7cba160ad789 ("powernv/cpuidle: Redesign idle states management")
Fixes: 77b54e9f213f ("powernv/powerpc: Add winkle support for offline cpus")
[mpe: Massage change log a bit more]
Signed-off-by: Sam Bobroff <sam.bobroff@au1.ibm.com>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2015-05-01 06:50:34 +00:00
|
|
|
ld r6,_CCR(r1)
|
2011-01-24 07:42:41 +00:00
|
|
|
ld r4,_MSR(r1)
|
|
|
|
|
ld r5,_NIP(r1)
|
|
|
|
|
addi r1,r1,INT_FRAME_SIZE
|
powerpc/powernv: Restore non-volatile CRs after nap
Patches 7cba160ad "powernv/cpuidle: Redesign idle states management"
and 77b54e9f2 "powernv/powerpc: Add winkle support for offline cpus"
use non-volatile condition registers (cr2, cr3 and cr4) early in the system
reset interrupt handler (system_reset_pSeries()) before it has been determined
if state loss has occurred. If state loss has not occurred, control returns via
the power7_wakeup_noloss() path which does not restore those condition
registers, leaving them corrupted.
Fix this by restoring the condition registers in the power7_wakeup_noloss()
case.
This is apparent when running a KVM guest on hardware that does not
support winkle or sleep and the guest makes use of secondary threads. In
practice this means Power7 machines, though some early unreleased Power8
machines may also be susceptible.
The secondary CPUs are taken off line before the guest is started and
they call pnv_smp_cpu_kill_self(). This checks support for sleep
states (in this case there is no support) and power7_nap() is called.
When the CPU is woken, power7_nap() returns and because the CPU is
still off line, the main while loop executes again. The sleep states
support test is executed again, but because the tested values cannot
have changed, the compiler has optimized the test away and instead we
rely on the result of the first test, which has been left in cr3
and/or cr4. With the result overwritten, the wrong branch is taken and
power7_winkle() is called on a CPU that does not support it, leading
to it stalling.
Fixes: 7cba160ad789 ("powernv/cpuidle: Redesign idle states management")
Fixes: 77b54e9f213f ("powernv/powerpc: Add winkle support for offline cpus")
[mpe: Massage change log a bit more]
Signed-off-by: Sam Bobroff <sam.bobroff@au1.ibm.com>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2015-05-01 06:50:34 +00:00
|
|
|
mtcr r6
|
2011-01-24 07:42:41 +00:00
|
|
|
mtspr SPRN_SRR1,r4
|
|
|
|
|
mtspr SPRN_SRR0,r5
|
|
|
|
|
rfid
|
