This reverts commit d532f3d267.
The original commit has several problems:
1) Doesn't work with 64-bit kernels.
2) Calls TLBMISS_HANDLER_SETUP() before the code is generated.
3) Calls TLBMISS_HANDLER_SETUP() twice in per_cpu_trap_init() when
only one call is needed.
[ralf@linux-mips.org: Also revert the bits of the ASID patch which were
hidden in the KVM merge.]
Signed-off-by: David Daney <david.daney@cavium.com>
Cc: linux-mips@linux-mips.org
Cc: linux-kernel@vger.kernel.org
Cc: "Steven J. Hill" <Steven.Hill@imgtec.com>
Cc: David Daney <david.daney@cavium.com>
Patchwork: https://patchwork.linux-mips.org/patch/5242/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
All exceptions must be taken in microMIPS mode, never in classic
MIPS mode or the kernel falls apart. A few NOP instructions are
used to maintain the correct alignment of microMIPS versions of
the exception vectors.
Signed-off-by: Steven J. Hill <Steven.Hill@imgtec.com>
Changes for pure microMIPS cores to dynamically determine the ASID
size at boot time.
Includes bug fix https://patchwork.linux-mips.org/patch/5230/
Signed-off-by: Steven J. Hill <Steven.Hill@imgtec.com>
Signed-off-by: Jonas Gorski <jogo@openwrt.org>
Original patch by Ralf Baechle and removed by Harold Koerfgen
with commit f67e4ffc79905482c3b9b8c8dd65197bac7eb508. This
allows for more generic kernels since the size of the ASID
and corresponding masks can be determined at run-time. This
patch is also required for the new Aptiv cores and has been
tested on Malta and Malta Aptiv platforms.
[ralf@linux-mips.org: Added relevant part of fix
https://patchwork.linux-mips.org/patch/5213/]
Signed-off-by: Steven J. Hill <Steven.Hill@imgtec.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
This and the next patch resolve memory corruption problems while CPU
hotplug. Without these patches, memory corruption can triggered easily
as below:
On a quad-core MIPS platform, use "spawn" of UnixBench-5.1.3 (http://
code.google.com/p/byte-unixbench/) and a CPU hotplug script like this
(hotplug.sh):
while true; do
echo 0 >/sys/devices/system/cpu/cpu1/online
echo 0 >/sys/devices/system/cpu/cpu2/online
echo 0 >/sys/devices/system/cpu/cpu3/online
sleep 1
echo 1 >/sys/devices/system/cpu/cpu1/online
echo 1 >/sys/devices/system/cpu/cpu2/online
echo 1 >/sys/devices/system/cpu/cpu3/online
sleep 1
done
Run "hotplug.sh" and then run "spawn 10000", spawn will get segfault
after a few minutes.
This patch:
Currently, clear_page()/copy_page() are generated by Micro-assembler
dynamically. But they are unavailable until uasm_resolve_relocs() has
finished because jump labels are illegal before that. Since these
functions are shared by every CPU, we only call build_clear_page()/
build_copy_page() only once at boot time. Without this patch, programs
will get random memory corruption (segmentation fault, bus error, etc.)
while CPU Hotplug (e.g. one CPU is using clear_page() while another is
generating it in cpu_cache_init()).
For similar reasons we modify build_tlb_refill_handler()'s invocation.
V2:
1, Rework the code to make CPU#0 can be online/offline.
2, Introduce cpu_has_local_ebase feature since some types of MIPS CPU
need a per-CPU tlb_refill_handler().
Signed-off-by: Huacai Chen <chenhc@lemote.com>
Signed-off-by: Hongbing Hu <huhb@lemote.com>
Acked-by: David Daney <david.daney@cavium.com>
Patchwork: http://patchwork.linux-mips.org/patch/4994/
Acked-by: John Crispin <blogic@openwrt.org>
tlbmiss_handler_setup_pgd is run-time generated code and it was convenient
to pretend the symbol was an array in the generator but a function for
the users. LTO gcc won't tolerate this kind of lie anymore so solve the
problem through a cast and function pointer instead.
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Having received another series of whitespace patches I decided to do this
once and for all rather than dealing with this kind of patches trickling
in forever.
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
We need Huge TLBs for HUGETLB_PAGE, or the soon to follow
TRANSPARENT_HUGEPAGE. collect this information under a single Kconfig
symbol.
Signed-off-by: David Daney <david.daney@cavium.com>
Pgtable bits are assigned dynamically depending on processor feature and
statically based on kernel configuration. To make sense out of the
disassembled TLB exception handlers a list of the actual assignments
used for a particular configuration and hardware setup can be very useful.
Output the actual TLB exception handlers in a format that simplifies their
post processsing from dmesg output.
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
From a software perspective R5000 and R5000A are the same thing which is
why the symbol CPU_R5000A never got used, so finally delete it.
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
The new functions introduced in commit 02a5417751 (MIPS: tlbex: Deal with
re-definition of label) should be marked __cpuinit, to eliminate a
warning that can pop up when CONFIG_EXPORT_UASM is disabled:
LD arch/mips/mm/built-in.o
WARNING: arch/mips/mm/built-in.o(.text+0x2a4c): Section mismatch in reference from the function uasm_bgezl_hazard() to the function .cpuinit.text:uasm_il_bgezl()
The function uasm_bgezl_hazard() references
the function __cpuinit uasm_il_bgezl().
This is often because uasm_bgezl_hazard lacks a __cpuinit
annotation or the annotation of uasm_il_bgezl is wrong.
WARNING: arch/mips/mm/built-in.o(.text+0x2a68): Section mismatch in reference from the function uasm_bgezl_label() to the function .cpuinit.text:uasm_build_label()
The function uasm_bgezl_label() references
the function __cpuinit uasm_build_label().
This is often because uasm_bgezl_label lacks a __cpuinit
annotation or the annotation of uasm_build_label is wrong.
(This warning might not occur if the function was inlined.)
Signed-off-by: Kevin Cernekee <cernekee@gmail.com>
Patchwork: http://patchwork.linux-mips.org/patch/4517
Signed-off-by: John Crispin <blogic@openwrt.org>
R5000 and the Nevada CPUs (RM5230, RM5231, RM5260, RM5261, RM5270 and
RM5271) are basically the same CPU core and all are documented to require
two instructions separating a write to c0_pagemask, c0_entryhi, c0_entrylo0,
c0_entrylo1 or c0_index.
So far we were only providing on cycle before / after a TLBR/TLBWI
for R5000 but 3 cycles before and 1 cycles after for the Nevadas.
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
The microassembler used in tlbex.c does not notice if a label is redefined
resulting in relocations against such labels silently missrelocated.
The issues exists since commit add6eb04776db4189ea89f596cbcde31b899be9d
[Synthesize TLB exception handlers at runtime.] in 2.6.10 and went unnoticed
for so long because the relocations for the affected branches got computed
to do something *almost* sensible.
The issue affects R4000, R4400, QED/IDT RM5230, RM5231, RM5260, RM5261,
RM5270 and RM5271 processors.
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
We don't have to do a separate shift to eliminate the software bits,
just rotate them into the fill and they will be ignored.
Signed-off-by: David Daney <ddaney@caviumnetworks.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/4294/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Remove usage of the 'kernel_uses_smartmips_rixi' macro from all files
and use new 'cpu_has_rixi' instead.
Signed-off-by: Steven J. Hill <sjhill@mips.com>
Acked-by: David Daney <david.daney@cavium.com>
The EXT and INS instructions can be used to decrease code size and
thus speed up TLB handlers on MIPS32R2 and MIPS64R2 cores.
Signed-off-by: Steven J. Hill <sjhill@mips.com>
The architecture specification says that an EHB instruction is
needed to avoid a hazard when writing TLB entries. However, some
cores do not have this hazard, and thus the EHB instruction causes
a costly pipeline stall. Detect these cores and do not use the EHB
instruction.
Signed-off-by: Steven J. Hill <sjhill@mips.com>
For the case PM_DEFAULT_MASK == 0, we were placing a branch in the
delay slot of another branch. This leads to undefined behavior.
Signed-off-by: David Daney <david.daney@cavium.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/2775/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Only some GCC versions such as gcc 4.2 notice that the variable wr in
build_r3000_tlb_modify_handler is used uninitialized. When using one
of those GCCs the build will fail due to -Werror. GCC 4.6 does not
warn about the uninitialized use of wr.
This issue was introduced by 7211f4d7a3dcbe57c5d396c334dca525315dceb2
[MIPS: Close races in TLB modify handlers.]
Reported-by: Ganesan Ramalingam <ganesan18@gmail.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Page table entries are made invalid by writing a zero into the the PTE
slot in a page table. This creates a race condition with the TLB
modify handlers when they are updating the PTE.
CPU0 CPU1
Test for _PAGE_PRESENT
. set to not _PAGE_PRESENT (zero)
Set to _PAGE_VALID
So now the page not present value (zero) is suddenly valid and user
space programs have access to physical page zero.
We close the race by putting the test for _PAGE_PRESENT and setting of
_PAGE_VALID into an atomic LL/SC section. This requires more registers
than just K0 and K1 in the handlers, so we need to save some registers
to a save area and then restore them when we are done.
The save area is an array of cacheline aligned structures that should
not suffer cache line bouncing as they are CPU private.
[ralf@linux-mips.org: Fix !defined(CONFIG_MIPS_PGD_C0_CONTEXT) build error.]
Signed-off-by: David Daney <david.daney@cavium.com>
To: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/2577/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
CPU_XLR case added to mm/tlbex.c
CPU_XLR case added to mm/c-r4k.c for PINDEX attribute
Feature overrides for XLR cpu.
Signed-off-by: Jayachandran C <jayachandranc@netlogicmicro.com>
To: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/2333/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
CC arch/mips/mm/tlbex.o
arch/mips/mm/tlbex.c: In function 'build_r4000_tlb_refill_handler':
arch/mips/mm/tlbex.c:1155:22: error: variable 'vmalloc_mode' set but not used [-Werror=unused-but-set-variable]
arch/mips/mm/tlbex.c:1154:28: error: variable 'htlb_info' set but not used [-Werror=unused-but-set-variable]
cc1: all warnings being treated as errors
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
It was reported that GCC-4.3.3 (with CodeSourcery extensions) fails
without this.
Reported-by: Jonas Gorski <jonas.gorski@gmail.com>
Signed-off-by: David Daney <ddaney@caviumnetworks.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/2010/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Octeon can use scratch registers in the TLB handlers. Octeon II can
use LDX instructions.
Signed-off-by: David Daney <ddaney@caviumnetworks.com>
To: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/1904/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
If the CPU supports BBIT0 and BBIT1, use them in TLB handlers as they
are more efficient than an AND followed by an branch and then
restoring the clobbered register.
Signed-off-by: David Daney <ddaney@caviumnetworks.com>
To: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/1873/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
BMIPS processor cores are used in 50+ different chipsets spread across
5+ product lines. In many cases the chipsets do not share the same
peripheral register layouts, the same register blocks, the same
interrupt controllers, the same memory maps, or much of anything else.
But, across radically different SoCs that share nothing more than the
same BMIPS CPU, a few things are still mostly constant:
SMP operations
Access to performance counters
DMA cache coherency quirks
Cache and memory bus configuration
So, it makes sense to treat each BMIPS processor type as a generic
"building block," rather than tying it to a specific SoC. This makes it
easier to support a large number of BMIPS-based chipsets without
unnecessary duplication of code, and provides the infrastructure needed
to support BMIPS-proprietary features.
Signed-off-by: Kevin Cernekee <cernekee@gmail.com>
Cc: mbizon@freebox.fr
Cc: linux-mips@linux-mips.org
Cc: linux-kernel@vger.kernel.org
Tested-by: Florian Fainelli <ffainelli@freebox.fr>
Patchwork: https://patchwork.linux-mips.org/patch/1706/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org
For some combinations of PAGE_SIZE and vmbits, it is possible to have
userspace access that are beyond what is covered by the PGD, but within
vmbits. Such an access would cause the TLB refill handler to load garbage
values for PMD and PTE potentially giving userspace access to parts of the
physical address space to which it is not entitled.
In the TLB refill hot path, we add a single dsrl instruction so we can
check if any bits outside of the range covered by the PGD are set. In
the vmalloc side we then separate the bad case from the normal vmalloc
case and call tlb_do_page_fault_0 if warranted. This slows us down a
bit, but has the benefit of yielding deterministic behavior.
[Ralf: Fixed build error for 32-bit kernels.]
[Ralf: Folded lmo commit c8c0e22b2aa3982852b44279638ef37f9aa31b7d into this
commit.]
Signed-off-by: David Daney <ddaney@caviumnetworks.com>
To: linux-mips@linux-mips.org
Patchwork: http://patchwork.linux-mips.org/patch/1152/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
---
This makes the code somewhat cleaner while reducing the risk of shift
amount overflows when various page table related options are changed.
Signed-off-by: David Daney <ddaney@caviumnetworks.com>
To: linux-mips@linux-mips.org
Patchwork: http://patchwork.linux-mips.org/patch/1154/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
The SmartMIPS ASE specifies how Read Inhibit (RI) and eXecute Inhibit
(XI) bits in the page tables work. The upper two bits of EntryLo{0,1}
are RI and XI when the feature is enabled in the PageGrain register.
SmartMIPS only covers 32-bit systems. Cavium Octeon+ extends this to
64-bit systems by continuing to place the RI and XI bits in the top of
EntryLo even when EntryLo is 64-bits wide.
Because we need to carry the RI and XI bits in the PTE, the layout of
the PTE is changed. There is a two instruction overhead in the TLB
refill hot path to get the EntryLo bits into the proper position.
Also the TLB load exception has to probe the TLB to check if RI or XI
caused the exception.
Also of note is that the layout of the PTE bits is done at compile and
runtime rather than statically. In the 32-bit case this allows for
the same number of PFN bits as before the patch as the _PAGE_HUGE is
not supported in 32-bit kernels (we have _PAGE_NO_EXEC and
_PAGE_NO_READ instead of _PAGE_READ and _PAGE_HUGE).
The patch is tested on Cavium Octeon+, but should also work on 32-bit
systems with the Smart-MIPS ASE.
Signed-off-by: David Daney <ddaney@caviumnetworks.com>
To: linux-mips@linux-mips.org
Patchwork: http://patchwork.linux-mips.org/patch/952/
Patchwork: http://patchwork.linux-mips.org/patch/956/
Patchwork: http://patchwork.linux-mips.org/patch/962/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
64-bit CPUs have 64-bit c0_entrylo{0,1} registers. We should use the
64-bit dmtc0 instruction to set them. This becomes important if we
want to set the RI and XI bits present in some processors.
Signed-off-by: David Daney <ddaney@caviumnetworks.com>
To: linux-mips@linux-mips.org
Patchwork: http://patchwork.linux-mips.org/patch/954/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
For 64-bit kernels with 64KB pages and two level page tables, there are
42 bits worth of virtual address space This is larger than the 40 bits of
virtual address space obtained with the default 4KB Page size and three
levels, so there are no draw backs for using two level tables with this
configuration.
Signed-off-by: David Daney <ddaney@caviumnetworks.com>
Cc: linux-mips@linux-mips.org
Patchwork: http://patchwork.linux-mips.org/patch/761/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
