Minki/linux
1
0
Fork 1
mirror of https://github.com/torvalds/linux.git synced 2024-12-29 14:21:47 +00:00
Code Issues Packages Projects Releases Wiki Activity
19fc577579
linux/arch/arm64/include/asm/pgtable.h

511 lines
15 KiB
C
Raw Normal View History

arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
/*
* Copyright (C) 2012 ARM Ltd.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#ifndef __ASM_PGTABLE_H
#define __ASM_PGTABLE_H
#include <asm/proc-fns.h>
#include <asm/memory.h>
#include <asm/pgtable-hwdef.h>
/*
* Software defined PTE bits definition.
*/
arm64: mm: introduce present, faulting entries for PAGE_NONE This is mostly a port of dbf62d50067e ("ARM: mm: introduce L_PTE_VALID for page table entries") and 26ffd0d43b18 ("ARM: mm: introduce present, faulting entries for PAGE_NONE") from ARM, which makes use of present, faulting page table entries for page table entries mapped as PROT_NONE. The main difference with this implementation is that we can make use of the two pte type bits in order to avoid allocating a software bit for identifying PROT_NONE pages, instead reserving the 10b suffix for these types of mappings. This is required to prevent users from accessing such pages via syscalls such as read/write over a pipe. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2012-12-18 14:15:15 +00:00
#define PTE_VALID (_AT(pteval_t, 1) << 0)
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
#define PTE_DIRTY (_AT(pteval_t, 1) << 55)
#define PTE_SPECIAL (_AT(pteval_t, 1) << 56)
arm64: mm: Introduce PTE_WRITE We have the following means for encoding writable or dirty ptes: PTE_DIRTY PTE_RDONLY !pte_dirty && !pte_write 0 1 !pte_dirty && pte_write 0 1 pte_dirty && !pte_write 1 1 pte_dirty && pte_write 1 0 So we can't distinguish between writable clean ptes and read only ptes. This can cause problems with ptes being incorrectly flagged as read only when they are writable but not dirty. This patch introduces a new software bit PTE_WRITE which allows us to correctly identify writable ptes. PTE_RDONLY is now only clear for valid ptes where a page is both writable and dirty. Signed-off-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-01-15 14:07:13 +00:00
#define PTE_WRITE (_AT(pteval_t, 1) << 57)
arm64: Move PTE_PROT_NONE higher up PTE_PROT_NONE means that a pte is present but does not have any read/write attributes. However, setting the memory type like pgprot_writecombine() is allowed and such bits overlap with PTE_PROT_NONE. This causes mmap/munmap issues in drivers that change the vma->vm_pg_prot on PROT_NONE mappings. This patch reverts the PTE_FILE/PTE_PROT_NONE shift in commit 59911ca4325d (ARM64: mm: Move PTE_PROT_NONE bit) and moves PTE_PROT_NONE together with the other software bits. Signed-off-by: Steve Capper <steve.capper@linaro.org> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Tested-by: Steve Capper <steve.capper@linaro.org> Cc: <stable@vger.kernel.org> # 3.11+
2013-11-27 16:59:27 +00:00
#define PTE_PROT_NONE (_AT(pteval_t, 1) << 58) /* only when !PTE_VALID */
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
/*
* VMALLOC and SPARSEMEM_VMEMMAP ranges.
arm64: Determine the vmalloc/vmemmap space at build time based on VA_BITS Rather than guessing what the maximum vmmemap space should be, this patch allows the calculation based on the VA_BITS and sizeof(struct page). The vmalloc space extends to the beginning of the vmemmap space. Since the virtual kernel memory layout now depends on the build configuration, this patch removes the detailed description in Documentation/arm64/memory.txt in favour of information printed during kernel booting. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Tested-by: Jungseok Lee <jungseoklee85@gmail.com>
2014-07-16 16:42:43 +00:00
*
* VMEMAP_SIZE: allows the whole VA space to be covered by a struct page array
* (rounded up to PUD_SIZE).
* VMALLOC_START: beginning of the kernel VA space
* VMALLOC_END: extends to the available space below vmmemmap, PCI I/O space,
* fixed mappings and modules
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
*/
arm64: Determine the vmalloc/vmemmap space at build time based on VA_BITS Rather than guessing what the maximum vmmemap space should be, this patch allows the calculation based on the VA_BITS and sizeof(struct page). The vmalloc space extends to the beginning of the vmemmap space. Since the virtual kernel memory layout now depends on the build configuration, this patch removes the detailed description in Documentation/arm64/memory.txt in favour of information printed during kernel booting. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Tested-by: Jungseok Lee <jungseoklee85@gmail.com>
2014-07-16 16:42:43 +00:00
#define VMEMMAP_SIZE ALIGN((1UL << (VA_BITS - PAGE_SHIFT)) * sizeof(struct page), PUD_SIZE)
arm64: Use 42-bit address space with 64K pages This patch expands the VA_BITS to 42 when the 64K page configuration is enabled allowing 2TB kernel linear mapping. Linux still uses 2 levels of page tables in this configuration with pgd now being a full page. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Will Deacon <will.deacon@arm.com> Acked-by: Marc Zyngier <marc.zyngier@arm.com>
2013-10-23 15:50:07 +00:00
#define VMALLOC_START (UL(0xffffffffffffffff) << VA_BITS)
arm64: Determine the vmalloc/vmemmap space at build time based on VA_BITS Rather than guessing what the maximum vmmemap space should be, this patch allows the calculation based on the VA_BITS and sizeof(struct page). The vmalloc space extends to the beginning of the vmemmap space. Since the virtual kernel memory layout now depends on the build configuration, this patch removes the detailed description in Documentation/arm64/memory.txt in favour of information printed during kernel booting. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Tested-by: Jungseok Lee <jungseoklee85@gmail.com>
2014-07-16 16:42:43 +00:00
#define VMALLOC_END (PAGE_OFFSET - PUD_SIZE - VMEMMAP_SIZE - SZ_64K)
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
#define vmemmap ((struct page *)(VMALLOC_END + SZ_64K))
mm: make FIRST_USER_ADDRESS unsigned long on all archs LKP has triggered a compiler warning after my recent patch "mm: account pmd page tables to the process": mm/mmap.c: In function 'exit_mmap': >> mm/mmap.c:2857:2: warning: right shift count >= width of type [enabled by default] The code: > 2857 WARN_ON(mm_nr_pmds(mm) > 2858 round_up(FIRST_USER_ADDRESS, PUD_SIZE) >> PUD_SHIFT); In this, on tile, we have FIRST_USER_ADDRESS defined as 0. round_up() has the same type -- int. PUD_SHIFT. I think the best way to fix it is to define FIRST_USER_ADDRESS as unsigned long. On every arch for consistency. Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Reported-by: Wu Fengguang <fengguang.wu@intel.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2015-02-11 23:26:41 +00:00
#define FIRST_USER_ADDRESS 0UL
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
#ifndef __ASSEMBLY__
extern void __pte_error(const char *file, int line, unsigned long val);
extern void __pmd_error(const char *file, int line, unsigned long val);
arm64: mm: Implement 4 levels of translation tables This patch implements 4 levels of translation tables since 3 levels of page tables with 4KB pages cannot support 40-bit physical address space described in [1] due to the following issue. It is a restriction that kernel logical memory map with 4KB + 3 levels (0xffffffc000000000-0xffffffffffffffff) cannot cover RAM region from 544GB to 1024GB in [1]. Specifically, ARM64 kernel fails to create mapping for this region in map_mem function since __phys_to_virt for this region reaches to address overflow. If SoC design follows the document, [1], over 32GB RAM would be placed from 544GB. Even 64GB system is supposed to use the region from 544GB to 576GB for only 32GB RAM. Naturally, it would reach to enable 4 levels of page tables to avoid hacking __virt_to_phys and __phys_to_virt. However, it is recommended 4 levels of page table should be only enabled if memory map is too sparse or there is about 512GB RAM. References ---------- [1]: Principles of ARM Memory Maps, White Paper, Issue C Signed-off-by: Jungseok Lee <jays.lee@samsung.com> Reviewed-by: Sungjinn Chung <sungjinn.chung@samsung.com> Acked-by: Kukjin Kim <kgene.kim@samsung.com> Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org> Reviewed-by: Steve Capper <steve.capper@linaro.org> [catalin.marinas@arm.com: MEMBLOCK_INITIAL_LIMIT removed, same as PUD_SIZE] [catalin.marinas@arm.com: early_ioremap_init() updated for 4 levels] [catalin.marinas@arm.com: 48-bit VA depends on BROKEN until KVM is fixed] Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Tested-by: Jungseok Lee <jungseoklee85@gmail.com>
2014-05-12 09:40:51 +00:00
extern void __pud_error(const char *file, int line, unsigned long val);
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
extern void __pgd_error(const char *file, int line, unsigned long val);
arm64: Clean up the default pgprot setting The primary aim of this patchset is to remove the pgprot_default and prot_sect_default global variables and rely strictly on predefined values. The original goal was to be able to run SMP kernels on UP hardware by not setting the Shareability bit. However, it is unlikely to see UP ARMv8 hardware and even if we do, the Shareability bit is no longer assumed to disable cacheable accesses. A side effect is that the device mappings now have the Shareability attribute set. The hardware, however, should ignore it since Device accesses are always Outer Shareable. Following the removal of the two global variables, there is some PROT_* macro reshuffling and cleanup, including the __PAGE_* macros (replaced by PAGE_*). Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Will Deacon <will.deacon@arm.com>
2014-04-03 14:57:15 +00:00
#ifdef CONFIG_SMP
#define PROT_DEFAULT (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED)
#define PROT_SECT_DEFAULT (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S)
#else
#define PROT_DEFAULT (PTE_TYPE_PAGE | PTE_AF)
#define PROT_SECT_DEFAULT (PMD_TYPE_SECT | PMD_SECT_AF)
#endif
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
arm64: Clean up the default pgprot setting The primary aim of this patchset is to remove the pgprot_default and prot_sect_default global variables and rely strictly on predefined values. The original goal was to be able to run SMP kernels on UP hardware by not setting the Shareability bit. However, it is unlikely to see UP ARMv8 hardware and even if we do, the Shareability bit is no longer assumed to disable cacheable accesses. A side effect is that the device mappings now have the Shareability attribute set. The hardware, however, should ignore it since Device accesses are always Outer Shareable. Following the removal of the two global variables, there is some PROT_* macro reshuffling and cleanup, including the __PAGE_* macros (replaced by PAGE_*). Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Will Deacon <will.deacon@arm.com>
2014-04-03 14:57:15 +00:00
#define PROT_DEVICE_nGnRE (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_ATTRINDX(MT_DEVICE_nGnRE))
#define PROT_NORMAL_NC (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_ATTRINDX(MT_NORMAL_NC))
#define PROT_NORMAL (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_ATTRINDX(MT_NORMAL))
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
arm64: Clean up the default pgprot setting The primary aim of this patchset is to remove the pgprot_default and prot_sect_default global variables and rely strictly on predefined values. The original goal was to be able to run SMP kernels on UP hardware by not setting the Shareability bit. However, it is unlikely to see UP ARMv8 hardware and even if we do, the Shareability bit is no longer assumed to disable cacheable accesses. A side effect is that the device mappings now have the Shareability attribute set. The hardware, however, should ignore it since Device accesses are always Outer Shareable. Following the removal of the two global variables, there is some PROT_* macro reshuffling and cleanup, including the __PAGE_* macros (replaced by PAGE_*). Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Will Deacon <will.deacon@arm.com>
2014-04-03 14:57:15 +00:00
#define PROT_SECT_DEVICE_nGnRE (PROT_SECT_DEFAULT | PMD_SECT_PXN | PMD_SECT_UXN | PMD_ATTRINDX(MT_DEVICE_nGnRE))
#define PROT_SECT_NORMAL (PROT_SECT_DEFAULT | PMD_SECT_PXN | PMD_SECT_UXN | PMD_ATTRINDX(MT_NORMAL))
#define PROT_SECT_NORMAL_EXEC (PROT_SECT_DEFAULT | PMD_SECT_UXN | PMD_ATTRINDX(MT_NORMAL))
arm64: mm: introduce present, faulting entries for PAGE_NONE This is mostly a port of dbf62d50067e ("ARM: mm: introduce L_PTE_VALID for page table entries") and 26ffd0d43b18 ("ARM: mm: introduce present, faulting entries for PAGE_NONE") from ARM, which makes use of present, faulting page table entries for page table entries mapped as PROT_NONE. The main difference with this implementation is that we can make use of the two pte type bits in order to avoid allocating a software bit for identifying PROT_NONE pages, instead reserving the 10b suffix for these types of mappings. This is required to prevent users from accessing such pages via syscalls such as read/write over a pipe. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2012-12-18 14:15:15 +00:00
arm64: Clean up the default pgprot setting The primary aim of this patchset is to remove the pgprot_default and prot_sect_default global variables and rely strictly on predefined values. The original goal was to be able to run SMP kernels on UP hardware by not setting the Shareability bit. However, it is unlikely to see UP ARMv8 hardware and even if we do, the Shareability bit is no longer assumed to disable cacheable accesses. A side effect is that the device mappings now have the Shareability attribute set. The hardware, however, should ignore it since Device accesses are always Outer Shareable. Following the removal of the two global variables, there is some PROT_* macro reshuffling and cleanup, including the __PAGE_* macros (replaced by PAGE_*). Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Will Deacon <will.deacon@arm.com>
2014-04-03 14:57:15 +00:00
#define _PAGE_DEFAULT (PROT_DEFAULT | PTE_ATTRINDX(MT_NORMAL))
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
arm64: Clean up the default pgprot setting The primary aim of this patchset is to remove the pgprot_default and prot_sect_default global variables and rely strictly on predefined values. The original goal was to be able to run SMP kernels on UP hardware by not setting the Shareability bit. However, it is unlikely to see UP ARMv8 hardware and even if we do, the Shareability bit is no longer assumed to disable cacheable accesses. A side effect is that the device mappings now have the Shareability attribute set. The hardware, however, should ignore it since Device accesses are always Outer Shareable. Following the removal of the two global variables, there is some PROT_* macro reshuffling and cleanup, including the __PAGE_* macros (replaced by PAGE_*). Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Will Deacon <will.deacon@arm.com>
2014-04-03 14:57:15 +00:00
#define PAGE_KERNEL __pgprot(_PAGE_DEFAULT | PTE_PXN | PTE_UXN | PTE_DIRTY | PTE_WRITE)
#define PAGE_KERNEL_EXEC __pgprot(_PAGE_DEFAULT | PTE_UXN | PTE_DIRTY | PTE_WRITE)
arm64: Distinguish between user and kernel XN bits On AArch64, the meaning of the XN bit has changed to UXN (user). The PXN (privileged) bit must be set to prevent kernel execution. Without the PXN bit set, the CPU may speculatively access device memory. This patch ensures that all the mappings that the kernel must not execute from (including user mappings) have the PXN bit set. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2012-11-15 17:21:16 +00:00
arm64: Clean up the default pgprot setting The primary aim of this patchset is to remove the pgprot_default and prot_sect_default global variables and rely strictly on predefined values. The original goal was to be able to run SMP kernels on UP hardware by not setting the Shareability bit. However, it is unlikely to see UP ARMv8 hardware and even if we do, the Shareability bit is no longer assumed to disable cacheable accesses. A side effect is that the device mappings now have the Shareability attribute set. The hardware, however, should ignore it since Device accesses are always Outer Shareable. Following the removal of the two global variables, there is some PROT_* macro reshuffling and cleanup, including the __PAGE_* macros (replaced by PAGE_*). Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Will Deacon <will.deacon@arm.com>
2014-04-03 14:57:15 +00:00
#define PAGE_HYP __pgprot(_PAGE_DEFAULT | PTE_HYP)
arm64: KVM: define HYP and Stage-2 translation page flags Add HYP and S2 page flags, for both normal and device memory. Reviewed-by: Christopher Covington <cov@codeaurora.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2012-12-07 18:35:41 +00:00
#define PAGE_HYP_DEVICE __pgprot(PROT_DEVICE_nGnRE | PTE_HYP)
arm64: Clean up the default pgprot setting The primary aim of this patchset is to remove the pgprot_default and prot_sect_default global variables and rely strictly on predefined values. The original goal was to be able to run SMP kernels on UP hardware by not setting the Shareability bit. However, it is unlikely to see UP ARMv8 hardware and even if we do, the Shareability bit is no longer assumed to disable cacheable accesses. A side effect is that the device mappings now have the Shareability attribute set. The hardware, however, should ignore it since Device accesses are always Outer Shareable. Following the removal of the two global variables, there is some PROT_* macro reshuffling and cleanup, including the __PAGE_* macros (replaced by PAGE_*). Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Will Deacon <will.deacon@arm.com>
2014-04-03 14:57:15 +00:00
#define PAGE_S2 __pgprot(PROT_DEFAULT | PTE_S2_MEMATTR(MT_S2_NORMAL) | PTE_S2_RDONLY)
arm64: kvm: define PAGE_S2_DEVICE as read-only by default Now that we support read-only memslots, we need to make sure that pass-through device mappings are not mapped writable if the guest has requested them to be read-only. The existing implementation already honours this by calling kvm_set_s2pte_writable() on the new pte in case of writable mappings, so all we need to do is define the default pgprot_t value used for devices to be PTE_S2_RDONLY. Acked-by: Marc Zyngier <marc.zyngier@arm.com> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
2014-09-17 21:56:20 +00:00
#define PAGE_S2_DEVICE __pgprot(PROT_DEFAULT | PTE_S2_MEMATTR(MT_S2_DEVICE_nGnRE) | PTE_S2_RDONLY | PTE_UXN)
arm64: KVM: define HYP and Stage-2 translation page flags Add HYP and S2 page flags, for both normal and device memory. Reviewed-by: Christopher Covington <cov@codeaurora.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2012-12-07 18:35:41 +00:00
arm64: Clean up the default pgprot setting The primary aim of this patchset is to remove the pgprot_default and prot_sect_default global variables and rely strictly on predefined values. The original goal was to be able to run SMP kernels on UP hardware by not setting the Shareability bit. However, it is unlikely to see UP ARMv8 hardware and even if we do, the Shareability bit is no longer assumed to disable cacheable accesses. A side effect is that the device mappings now have the Shareability attribute set. The hardware, however, should ignore it since Device accesses are always Outer Shareable. Following the removal of the two global variables, there is some PROT_* macro reshuffling and cleanup, including the __PAGE_* macros (replaced by PAGE_*). Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Will Deacon <will.deacon@arm.com>
2014-04-03 14:57:15 +00:00
#define PAGE_NONE __pgprot(((_PAGE_DEFAULT) & ~PTE_TYPE_MASK) | PTE_PROT_NONE | PTE_PXN | PTE_UXN)
#define PAGE_SHARED __pgprot(_PAGE_DEFAULT | PTE_USER | PTE_NG | PTE_PXN | PTE_UXN | PTE_WRITE)
#define PAGE_SHARED_EXEC __pgprot(_PAGE_DEFAULT | PTE_USER | PTE_NG | PTE_PXN | PTE_WRITE)
#define PAGE_COPY __pgprot(_PAGE_DEFAULT | PTE_USER | PTE_NG | PTE_PXN | PTE_UXN)
#define PAGE_COPY_EXEC __pgprot(_PAGE_DEFAULT | PTE_USER | PTE_NG | PTE_PXN)
#define PAGE_READONLY __pgprot(_PAGE_DEFAULT | PTE_USER | PTE_NG | PTE_PXN | PTE_UXN)
#define PAGE_READONLY_EXEC __pgprot(_PAGE_DEFAULT | PTE_USER | PTE_NG | PTE_PXN)
#define __P000 PAGE_NONE
#define __P001 PAGE_READONLY
#define __P010 PAGE_COPY
#define __P011 PAGE_COPY
Revert "arm64: Introduce execute-only page access permissions" This reverts commit bc07c2c6e9ed125d362af0214b6313dca180cb08. While the aim is increased security for --x memory maps, it does not protect against kernel level reads. Until SECCOMP is implemented for arm64, revert this patch to avoid giving a false idea of execute-only mappings. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-05-16 15:44:32 +00:00
#define __P100 PAGE_READONLY_EXEC
arm64: Clean up the default pgprot setting The primary aim of this patchset is to remove the pgprot_default and prot_sect_default global variables and rely strictly on predefined values. The original goal was to be able to run SMP kernels on UP hardware by not setting the Shareability bit. However, it is unlikely to see UP ARMv8 hardware and even if we do, the Shareability bit is no longer assumed to disable cacheable accesses. A side effect is that the device mappings now have the Shareability attribute set. The hardware, however, should ignore it since Device accesses are always Outer Shareable. Following the removal of the two global variables, there is some PROT_* macro reshuffling and cleanup, including the __PAGE_* macros (replaced by PAGE_*). Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Will Deacon <will.deacon@arm.com>
2014-04-03 14:57:15 +00:00
#define __P101 PAGE_READONLY_EXEC
#define __P110 PAGE_COPY_EXEC
#define __P111 PAGE_COPY_EXEC
#define __S000 PAGE_NONE
#define __S001 PAGE_READONLY
#define __S010 PAGE_SHARED
#define __S011 PAGE_SHARED
Revert "arm64: Introduce execute-only page access permissions" This reverts commit bc07c2c6e9ed125d362af0214b6313dca180cb08. While the aim is increased security for --x memory maps, it does not protect against kernel level reads. Until SECCOMP is implemented for arm64, revert this patch to avoid giving a false idea of execute-only mappings. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-05-16 15:44:32 +00:00
#define __S100 PAGE_READONLY_EXEC
arm64: Clean up the default pgprot setting The primary aim of this patchset is to remove the pgprot_default and prot_sect_default global variables and rely strictly on predefined values. The original goal was to be able to run SMP kernels on UP hardware by not setting the Shareability bit. However, it is unlikely to see UP ARMv8 hardware and even if we do, the Shareability bit is no longer assumed to disable cacheable accesses. A side effect is that the device mappings now have the Shareability attribute set. The hardware, however, should ignore it since Device accesses are always Outer Shareable. Following the removal of the two global variables, there is some PROT_* macro reshuffling and cleanup, including the __PAGE_* macros (replaced by PAGE_*). Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Will Deacon <will.deacon@arm.com>
2014-04-03 14:57:15 +00:00
#define __S101 PAGE_READONLY_EXEC
#define __S110 PAGE_SHARED_EXEC
#define __S111 PAGE_SHARED_EXEC
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
/*
* ZERO_PAGE is a global shared page that is always zero: used
* for zero-mapped memory areas etc..
*/
extern struct page *empty_zero_page;
#define ZERO_PAGE(vaddr) (empty_zero_page)
arm64: asm/pgtable.h pmd/pud definitions clean-up Non-functional change to group together the pmd/pud definitions and reduce the amount of #if CONFIG_ARM64_PGTABLE_LEVELS. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Tested-by: Jungseok Lee <jungseoklee85@gmail.com>
2014-07-21 13:52:49 +00:00
#define pte_ERROR(pte) __pte_error(__FILE__, __LINE__, pte_val(pte))
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
#define pte_pfn(pte) ((pte_val(pte) & PHYS_MASK) >> PAGE_SHIFT)
#define pfn_pte(pfn,prot) (__pte(((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)))
#define pte_none(pte) (!pte_val(pte))
#define pte_clear(mm,addr,ptep) set_pte(ptep, __pte(0))
#define pte_page(pte) (pfn_to_page(pte_pfn(pte)))
arm64: asm/pgtable.h pmd/pud definitions clean-up Non-functional change to group together the pmd/pud definitions and reduce the amount of #if CONFIG_ARM64_PGTABLE_LEVELS. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Tested-by: Jungseok Lee <jungseoklee85@gmail.com>
2014-07-21 13:52:49 +00:00
/* Find an entry in the third-level page table. */
#define pte_index(addr) (((addr) >> PAGE_SHIFT) & (PTRS_PER_PTE - 1))
arm64: pgtable: use pte_index instead of __pte_index pte_index is a useful helper outside of arch/arm64, for things like the ARM SMMU driver, so rename __pte_index to pte_index to be consistent with both arch/arm/ and also the definitions of pmd_index and pgd_index. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2013-06-10 18:34:41 +00:00
#define pte_offset_kernel(dir,addr) (pmd_page_vaddr(*(dir)) + pte_index(addr))
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
#define pte_offset_map(dir,addr) pte_offset_kernel((dir), (addr))
#define pte_offset_map_nested(dir,addr) pte_offset_kernel((dir), (addr))
#define pte_unmap(pte) do { } while (0)
#define pte_unmap_nested(pte) do { } while (0)
/*
* The following only work if pte_present(). Undefined behaviour otherwise.
*/
arm64: mm: Add double logical invert to pte accessors Page table entries on ARM64 are 64 bits, and some pte functions such as pte_dirty return a bitwise-and of a flag with the pte value. If the flag to be tested resides in the upper 32 bits of the pte, then we run into the danger of the result being dropped if downcast. For example: gather_stats(page, md, pte_dirty(*pte), 1); where pte_dirty(*pte) is downcast to an int. This patch adds a double logical invert to all the pte_ accessors to ensure predictable downcasting. Signed-off-by: Steve Capper <steve.capper@linaro.org> Cc: <stable@vger.kernel.org> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-02-25 11:38:53 +00:00
#define pte_present(pte) (!!(pte_val(pte) & (PTE_VALID | PTE_PROT_NONE)))
#define pte_dirty(pte) (!!(pte_val(pte) & PTE_DIRTY))
#define pte_young(pte) (!!(pte_val(pte) & PTE_AF))
#define pte_special(pte) (!!(pte_val(pte) & PTE_SPECIAL))
#define pte_write(pte) (!!(pte_val(pte) & PTE_WRITE))
arm64: Distinguish between user and kernel XN bits On AArch64, the meaning of the XN bit has changed to UXN (user). The PXN (privileged) bit must be set to prevent kernel execution. Without the PXN bit set, the CPU may speculatively access device memory. This patch ensures that all the mappings that the kernel must not execute from (including user mappings) have the PXN bit set. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2012-11-15 17:21:16 +00:00
#define pte_exec(pte) (!(pte_val(pte) & PTE_UXN))
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
arm64: mm: introduce present, faulting entries for PAGE_NONE This is mostly a port of dbf62d50067e ("ARM: mm: introduce L_PTE_VALID for page table entries") and 26ffd0d43b18 ("ARM: mm: introduce present, faulting entries for PAGE_NONE") from ARM, which makes use of present, faulting page table entries for page table entries mapped as PROT_NONE. The main difference with this implementation is that we can make use of the two pte type bits in order to avoid allocating a software bit for identifying PROT_NONE pages, instead reserving the 10b suffix for these types of mappings. This is required to prevent users from accessing such pages via syscalls such as read/write over a pipe. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2012-12-18 14:15:15 +00:00
#define pte_valid_user(pte) \
arm64: mm: only wrprotect clean ptes if they are present Marking non-present ptes as read-only can corrupt file ptes, breaking things like swap and file mappings. This patch ensures that we only manipulate user pte bits when the pte is marked present. Cc: <stable@vger.kernel.org> Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2013-01-09 11:08:10 +00:00
((pte_val(pte) & (PTE_VALID | PTE_USER)) == (PTE_VALID | PTE_USER))
arm64: Fix barriers used for page table modifications The architecture specification states that both DSB and ISB are required between page table modifications and subsequent memory accesses using the corresponding virtual address. When TLB invalidation takes place, the tlb_flush_* functions already have the necessary barriers. However, there are other functions like create_mapping() for which this is not the case. The patch adds the DSB+ISB instructions in the set_pte() function for valid kernel mappings. The invalid pte case is handled by tlb_flush_* and the user mappings in general have a corresponding update_mmu_cache() call containing a DSB. Even when update_mmu_cache() isn't called, the kernel can still cope with an unlikely spurious page fault by re-executing the instruction. In addition, the set_pmd, set_pud() functions gain an ISB for architecture compliance when block mappings are created. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Reported-by: Leif Lindholm <leif.lindholm@linaro.org> Acked-by: Steve Capper <steve.capper@linaro.org> Cc: Will Deacon <will.deacon@arm.com> Cc: <stable@vger.kernel.org>
2014-06-09 10:55:03 +00:00
#define pte_valid_not_user(pte) \
((pte_val(pte) & (PTE_VALID | PTE_USER)) == PTE_VALID)
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
arm64: Introduce {set,clear}_pte_bit It's useful to be able to change individual bits in ptes at times. Introduce functions for this and update existing pte_mk* functions to use these primatives. Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Laura Abbott <lauraa@codeaurora.org> [will: added missing inline keyword for new header functions] Signed-off-by: Will Deacon <will.deacon@arm.com>
2014-08-19 19:41:42 +00:00
static inline pte_t clear_pte_bit(pte_t pte, pgprot_t prot)
arm64: mm: Remove PTE_BIT_FUNC macro Expand out the pte manipulation functions. This makes our life easier when using things like tags and cscope. Signed-off-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-01-15 14:07:12 +00:00
{
arm64: Introduce {set,clear}_pte_bit It's useful to be able to change individual bits in ptes at times. Introduce functions for this and update existing pte_mk* functions to use these primatives. Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Laura Abbott <lauraa@codeaurora.org> [will: added missing inline keyword for new header functions] Signed-off-by: Will Deacon <will.deacon@arm.com>
2014-08-19 19:41:42 +00:00
pte_val(pte) &= ~pgprot_val(prot);
arm64: mm: Remove PTE_BIT_FUNC macro Expand out the pte manipulation functions. This makes our life easier when using things like tags and cscope. Signed-off-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-01-15 14:07:12 +00:00
return pte;
}
arm64: Introduce {set,clear}_pte_bit It's useful to be able to change individual bits in ptes at times. Introduce functions for this and update existing pte_mk* functions to use these primatives. Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Laura Abbott <lauraa@codeaurora.org> [will: added missing inline keyword for new header functions] Signed-off-by: Will Deacon <will.deacon@arm.com>
2014-08-19 19:41:42 +00:00
static inline pte_t set_pte_bit(pte_t pte, pgprot_t prot)
arm64: mm: Remove PTE_BIT_FUNC macro Expand out the pte manipulation functions. This makes our life easier when using things like tags and cscope. Signed-off-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-01-15 14:07:12 +00:00
{
arm64: Introduce {set,clear}_pte_bit It's useful to be able to change individual bits in ptes at times. Introduce functions for this and update existing pte_mk* functions to use these primatives. Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Laura Abbott <lauraa@codeaurora.org> [will: added missing inline keyword for new header functions] Signed-off-by: Will Deacon <will.deacon@arm.com>
2014-08-19 19:41:42 +00:00
pte_val(pte) |= pgprot_val(prot);
arm64: mm: Remove PTE_BIT_FUNC macro Expand out the pte manipulation functions. This makes our life easier when using things like tags and cscope. Signed-off-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-01-15 14:07:12 +00:00
return pte;
}
arm64: Introduce {set,clear}_pte_bit It's useful to be able to change individual bits in ptes at times. Introduce functions for this and update existing pte_mk* functions to use these primatives. Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Laura Abbott <lauraa@codeaurora.org> [will: added missing inline keyword for new header functions] Signed-off-by: Will Deacon <will.deacon@arm.com>
2014-08-19 19:41:42 +00:00
static inline pte_t pte_wrprotect(pte_t pte)
{
return clear_pte_bit(pte, __pgprot(PTE_WRITE));
}
static inline pte_t pte_mkwrite(pte_t pte)
{
return set_pte_bit(pte, __pgprot(PTE_WRITE));
}
arm64: mm: Remove PTE_BIT_FUNC macro Expand out the pte manipulation functions. This makes our life easier when using things like tags and cscope. Signed-off-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-01-15 14:07:12 +00:00
static inline pte_t pte_mkclean(pte_t pte)
{
arm64: Introduce {set,clear}_pte_bit It's useful to be able to change individual bits in ptes at times. Introduce functions for this and update existing pte_mk* functions to use these primatives. Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Laura Abbott <lauraa@codeaurora.org> [will: added missing inline keyword for new header functions] Signed-off-by: Will Deacon <will.deacon@arm.com>
2014-08-19 19:41:42 +00:00
return clear_pte_bit(pte, __pgprot(PTE_DIRTY));
arm64: mm: Remove PTE_BIT_FUNC macro Expand out the pte manipulation functions. This makes our life easier when using things like tags and cscope. Signed-off-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-01-15 14:07:12 +00:00
}
static inline pte_t pte_mkdirty(pte_t pte)
{
arm64: Introduce {set,clear}_pte_bit It's useful to be able to change individual bits in ptes at times. Introduce functions for this and update existing pte_mk* functions to use these primatives. Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Laura Abbott <lauraa@codeaurora.org> [will: added missing inline keyword for new header functions] Signed-off-by: Will Deacon <will.deacon@arm.com>
2014-08-19 19:41:42 +00:00
return set_pte_bit(pte, __pgprot(PTE_DIRTY));
arm64: mm: Remove PTE_BIT_FUNC macro Expand out the pte manipulation functions. This makes our life easier when using things like tags and cscope. Signed-off-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-01-15 14:07:12 +00:00
}
static inline pte_t pte_mkold(pte_t pte)
{
arm64: Introduce {set,clear}_pte_bit It's useful to be able to change individual bits in ptes at times. Introduce functions for this and update existing pte_mk* functions to use these primatives. Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Laura Abbott <lauraa@codeaurora.org> [will: added missing inline keyword for new header functions] Signed-off-by: Will Deacon <will.deacon@arm.com>
2014-08-19 19:41:42 +00:00
return clear_pte_bit(pte, __pgprot(PTE_AF));
arm64: mm: Remove PTE_BIT_FUNC macro Expand out the pte manipulation functions. This makes our life easier when using things like tags and cscope. Signed-off-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-01-15 14:07:12 +00:00
}
static inline pte_t pte_mkyoung(pte_t pte)
{
arm64: Introduce {set,clear}_pte_bit It's useful to be able to change individual bits in ptes at times. Introduce functions for this and update existing pte_mk* functions to use these primatives. Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Laura Abbott <lauraa@codeaurora.org> [will: added missing inline keyword for new header functions] Signed-off-by: Will Deacon <will.deacon@arm.com>
2014-08-19 19:41:42 +00:00
return set_pte_bit(pte, __pgprot(PTE_AF));
arm64: mm: Remove PTE_BIT_FUNC macro Expand out the pte manipulation functions. This makes our life easier when using things like tags and cscope. Signed-off-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-01-15 14:07:12 +00:00
}
static inline pte_t pte_mkspecial(pte_t pte)
{
arm64: Introduce {set,clear}_pte_bit It's useful to be able to change individual bits in ptes at times. Introduce functions for this and update existing pte_mk* functions to use these primatives. Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Laura Abbott <lauraa@codeaurora.org> [will: added missing inline keyword for new header functions] Signed-off-by: Will Deacon <will.deacon@arm.com>
2014-08-19 19:41:42 +00:00
return set_pte_bit(pte, __pgprot(PTE_SPECIAL));
arm64: mm: Remove PTE_BIT_FUNC macro Expand out the pte manipulation functions. This makes our life easier when using things like tags and cscope. Signed-off-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-01-15 14:07:12 +00:00
}
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
static inline void set_pte(pte_t *ptep, pte_t pte)
{
*ptep = pte;
arm64: Fix barriers used for page table modifications The architecture specification states that both DSB and ISB are required between page table modifications and subsequent memory accesses using the corresponding virtual address. When TLB invalidation takes place, the tlb_flush_* functions already have the necessary barriers. However, there are other functions like create_mapping() for which this is not the case. The patch adds the DSB+ISB instructions in the set_pte() function for valid kernel mappings. The invalid pte case is handled by tlb_flush_* and the user mappings in general have a corresponding update_mmu_cache() call containing a DSB. Even when update_mmu_cache() isn't called, the kernel can still cope with an unlikely spurious page fault by re-executing the instruction. In addition, the set_pmd, set_pud() functions gain an ISB for architecture compliance when block mappings are created. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Reported-by: Leif Lindholm <leif.lindholm@linaro.org> Acked-by: Steve Capper <steve.capper@linaro.org> Cc: Will Deacon <will.deacon@arm.com> Cc: <stable@vger.kernel.org>
2014-06-09 10:55:03 +00:00
/*
* Only if the new pte is valid and kernel, otherwise TLB maintenance
* or update_mmu_cache() have the necessary barriers.
*/
if (pte_valid_not_user(pte)) {
dsb(ishst);
isb();
}
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
}
extern void __sync_icache_dcache(pte_t pteval, unsigned long addr);
static inline void set_pte_at(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t pte)
{
arm64: mm: introduce present, faulting entries for PAGE_NONE This is mostly a port of dbf62d50067e ("ARM: mm: introduce L_PTE_VALID for page table entries") and 26ffd0d43b18 ("ARM: mm: introduce present, faulting entries for PAGE_NONE") from ARM, which makes use of present, faulting page table entries for page table entries mapped as PROT_NONE. The main difference with this implementation is that we can make use of the two pte type bits in order to avoid allocating a software bit for identifying PROT_NONE pages, instead reserving the 10b suffix for these types of mappings. This is required to prevent users from accessing such pages via syscalls such as read/write over a pipe. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2012-12-18 14:15:15 +00:00
if (pte_valid_user(pte)) {
arm64: Do not synchronise I and D caches for special ptes Special pte mappings are not intended to be executable and do not even have an associated struct page. This patch ensures that we do not call __sync_icache_dcache() on such ptes. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Reported-by: Steve Capper <Steve.Capper@arm.com> Tested-by: Laura Abbott <lauraa@codeaurora.org> Tested-by: Bharat Bhushan <Bharat.Bhushan@freescale.com> Cc: <stable@vger.kernel.org>
2014-03-12 16:28:09 +00:00
if (!pte_special(pte) && pte_exec(pte))
arm64: mm: only wrprotect clean ptes if they are present Marking non-present ptes as read-only can corrupt file ptes, breaking things like swap and file mappings. This patch ensures that we only manipulate user pte bits when the pte is marked present. Cc: <stable@vger.kernel.org> Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2013-01-09 11:08:10 +00:00
__sync_icache_dcache(pte, addr);
arm64: mm: Introduce PTE_WRITE We have the following means for encoding writable or dirty ptes: PTE_DIRTY PTE_RDONLY !pte_dirty && !pte_write 0 1 !pte_dirty && pte_write 0 1 pte_dirty && !pte_write 1 1 pte_dirty && pte_write 1 0 So we can't distinguish between writable clean ptes and read only ptes. This can cause problems with ptes being incorrectly flagged as read only when they are writable but not dirty. This patch introduces a new software bit PTE_WRITE which allows us to correctly identify writable ptes. PTE_RDONLY is now only clear for valid ptes where a page is both writable and dirty. Signed-off-by: Steve Capper <steve.capper@arm.com> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-01-15 14:07:13 +00:00
if (pte_dirty(pte) && pte_write(pte))
pte_val(pte) &= ~PTE_RDONLY;
else
pte_val(pte) |= PTE_RDONLY;
arm64: mm: only wrprotect clean ptes if they are present Marking non-present ptes as read-only can corrupt file ptes, breaking things like swap and file mappings. This patch ensures that we only manipulate user pte bits when the pte is marked present. Cc: <stable@vger.kernel.org> Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2013-01-09 11:08:10 +00:00
}
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
set_pte(ptep, pte);
}
/*
* Huge pte definitions.
*/
ARM64: mm: HugeTLB support. Add huge page support to ARM64, different huge page sizes are supported depending on the size of normal pages: PAGE_SIZE is 4KB: 2MB - (pmds) these can be allocated at any time. 1024MB - (puds) usually allocated on bootup with the command line with something like: hugepagesz=1G hugepages=6 PAGE_SIZE is 64KB: 512MB - (pmds) usually allocated on bootup via command line. Signed-off-by: Steve Capper <steve.capper@linaro.org> Acked-by: Catalin Marinas <catalin.marinas@arm.com>
2013-04-10 12:48:00 +00:00
#define pte_huge(pte) (!(pte_val(pte) & PTE_TABLE_BIT))
#define pte_mkhuge(pte) (__pte(pte_val(pte) & ~PTE_TABLE_BIT))
/*
* Hugetlb definitions.
*/
#define HUGE_MAX_HSTATE 2
#define HPAGE_SHIFT PMD_SHIFT
#define HPAGE_SIZE (_AC(1, UL) << HPAGE_SHIFT)
#define HPAGE_MASK (~(HPAGE_SIZE - 1))
#define HUGETLB_PAGE_ORDER (HPAGE_SHIFT - PAGE_SHIFT)
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
#define __HAVE_ARCH_PTE_SPECIAL
arm64: mm: enable RCU fast_gup Activate the RCU fast_gup for ARM64. We also need to force THP splits to broadcast an IPI s.t. we block in the fast_gup page walker. As THP splits are comparatively rare, this should not lead to a noticeable performance degradation. Some pre-requisite functions pud_write and pud_page are also added. [akpm@linux-foundation.org: coding-style fixes] Signed-off-by: Steve Capper <steve.capper@linaro.org> Tested-by: Dann Frazier <dann.frazier@canonical.com> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Cc: Hugh Dickins <hughd@google.com> Cc: Russell King <rmk@arm.linux.org.uk> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Mel Gorman <mel@csn.ul.ie> Cc: Will Deacon <will.deacon@arm.com> Cc: Christoffer Dall <christoffer.dall@linaro.org> Cc: Andrea Arcangeli <aarcange@redhat.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2014-10-09 22:29:25 +00:00
static inline pte_t pud_pte(pud_t pud)
{
return __pte(pud_val(pud));
}
static inline pmd_t pud_pmd(pud_t pud)
{
return __pmd(pud_val(pud));
}
arm64: mm: Route pmd thp functions through pte equivalents Rather than have separate hugetlb and transparent huge page pmd manipulation functions, re-wire our thp functions to simply call the pte equivalents. This allows THP to take advantage of the new PTE_WRITE logic introduced in: c2c93e5 arm64: mm: Introduce PTE_WRITE To represent splitting THPs we use the PTE_SPECIAL bit as this is not used for pmds. Signed-off-by: Steve Capper <steve.capper@linaro.org> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-02-25 10:02:13 +00:00
static inline pte_t pmd_pte(pmd_t pmd)
{
return __pte(pmd_val(pmd));
}
ARM64: mm: THP support. Bring Transparent HugePage support to ARM. The size of a transparent huge page depends on the normal page size. A transparent huge page is always represented as a pmd. If PAGE_SIZE is 4KB, THPs are 2MB. If PAGE_SIZE is 64KB, THPs are 512MB. Signed-off-by: Steve Capper <steve.capper@linaro.org> Acked-by: Catalin Marinas <catalin.marinas@arm.com>
2013-04-19 15:23:57 +00:00
arm64: mm: Route pmd thp functions through pte equivalents Rather than have separate hugetlb and transparent huge page pmd manipulation functions, re-wire our thp functions to simply call the pte equivalents. This allows THP to take advantage of the new PTE_WRITE logic introduced in: c2c93e5 arm64: mm: Introduce PTE_WRITE To represent splitting THPs we use the PTE_SPECIAL bit as this is not used for pmds. Signed-off-by: Steve Capper <steve.capper@linaro.org> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-02-25 10:02:13 +00:00
static inline pmd_t pte_pmd(pte_t pte)
{
return __pmd(pte_val(pte));
}
ARM64: mm: THP support. Bring Transparent HugePage support to ARM. The size of a transparent huge page depends on the normal page size. A transparent huge page is always represented as a pmd. If PAGE_SIZE is 4KB, THPs are 2MB. If PAGE_SIZE is 64KB, THPs are 512MB. Signed-off-by: Steve Capper <steve.capper@linaro.org> Acked-by: Catalin Marinas <catalin.marinas@arm.com>
2013-04-19 15:23:57 +00:00
arm64/mm: add create_pgd_mapping() to create private page tables For UEFI, we need to install the memory mappings used for Runtime Services in a dedicated set of page tables. Add create_pgd_mapping(), which allows us to allocate and install those page table entries early. Reviewed-by: Will Deacon <will.deacon@arm.com> Tested-by: Leif Lindholm <leif.lindholm@linaro.org> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
2014-10-20 13:42:07 +00:00
static inline pgprot_t mk_sect_prot(pgprot_t prot)
{
return __pgprot(pgprot_val(prot) & ~PTE_TABLE_BIT);
}
ARM64: mm: THP support. Bring Transparent HugePage support to ARM. The size of a transparent huge page depends on the normal page size. A transparent huge page is always represented as a pmd. If PAGE_SIZE is 4KB, THPs are 2MB. If PAGE_SIZE is 64KB, THPs are 512MB. Signed-off-by: Steve Capper <steve.capper@linaro.org> Acked-by: Catalin Marinas <catalin.marinas@arm.com>
2013-04-19 15:23:57 +00:00
/*
* THP definitions.
*/
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
#define pmd_trans_huge(pmd) (pmd_val(pmd) && !(pmd_val(pmd) & PMD_TABLE_BIT))
arm64: mm: Route pmd thp functions through pte equivalents Rather than have separate hugetlb and transparent huge page pmd manipulation functions, re-wire our thp functions to simply call the pte equivalents. This allows THP to take advantage of the new PTE_WRITE logic introduced in: c2c93e5 arm64: mm: Introduce PTE_WRITE To represent splitting THPs we use the PTE_SPECIAL bit as this is not used for pmds. Signed-off-by: Steve Capper <steve.capper@linaro.org> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-02-25 10:02:13 +00:00
#define pmd_trans_splitting(pmd) pte_special(pmd_pte(pmd))
arm64: mm: enable RCU fast_gup Activate the RCU fast_gup for ARM64. We also need to force THP splits to broadcast an IPI s.t. we block in the fast_gup page walker. As THP splits are comparatively rare, this should not lead to a noticeable performance degradation. Some pre-requisite functions pud_write and pud_page are also added. [akpm@linux-foundation.org: coding-style fixes] Signed-off-by: Steve Capper <steve.capper@linaro.org> Tested-by: Dann Frazier <dann.frazier@canonical.com> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Cc: Hugh Dickins <hughd@google.com> Cc: Russell King <rmk@arm.linux.org.uk> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Mel Gorman <mel@csn.ul.ie> Cc: Will Deacon <will.deacon@arm.com> Cc: Christoffer Dall <christoffer.dall@linaro.org> Cc: Andrea Arcangeli <aarcange@redhat.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2014-10-09 22:29:25 +00:00
#ifdef CONFIG_HAVE_RCU_TABLE_FREE
#define __HAVE_ARCH_PMDP_SPLITTING_FLUSH
struct vm_area_struct;
void pmdp_splitting_flush(struct vm_area_struct *vma, unsigned long address,
pmd_t *pmdp);
#endif /* CONFIG_HAVE_RCU_TABLE_FREE */
#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
ARM64: mm: THP support. Bring Transparent HugePage support to ARM. The size of a transparent huge page depends on the normal page size. A transparent huge page is always represented as a pmd. If PAGE_SIZE is 4KB, THPs are 2MB. If PAGE_SIZE is 64KB, THPs are 512MB. Signed-off-by: Steve Capper <steve.capper@linaro.org> Acked-by: Catalin Marinas <catalin.marinas@arm.com>
2013-04-19 15:23:57 +00:00
mm: fix huge zero page accounting in smaps report As a small zero page, huge zero page should not be accounted in smaps report as normal page. For small pages we rely on vm_normal_page() to filter out zero page, but vm_normal_page() is not designed to handle pmds. We only get here due hackish cast pmd to pte in smaps_pte_range() -- pte and pmd format is not necessary compatible on each and every architecture. Let's add separate codepath to handle pmds. follow_trans_huge_pmd() will detect huge zero page for us. We would need pmd_dirty() helper to do this properly. The patch adds it to THP-enabled architectures which don't yet have one. [akpm@linux-foundation.org: use do_div to fix 32-bit build] Signed-off-by: "Kirill A. Shutemov" <kirill@shutemov.name> Reported-by: Fengguang Wu <fengguang.wu@intel.com> Tested-by: Fengwei Yin <yfw.kernel@gmail.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2014-12-10 23:44:36 +00:00
#define pmd_dirty(pmd) pte_dirty(pmd_pte(pmd))
arm64: mm: Route pmd thp functions through pte equivalents Rather than have separate hugetlb and transparent huge page pmd manipulation functions, re-wire our thp functions to simply call the pte equivalents. This allows THP to take advantage of the new PTE_WRITE logic introduced in: c2c93e5 arm64: mm: Introduce PTE_WRITE To represent splitting THPs we use the PTE_SPECIAL bit as this is not used for pmds. Signed-off-by: Steve Capper <steve.capper@linaro.org> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-02-25 10:02:13 +00:00
#define pmd_young(pmd) pte_young(pmd_pte(pmd))
#define pmd_wrprotect(pmd) pte_pmd(pte_wrprotect(pmd_pte(pmd)))
#define pmd_mksplitting(pmd) pte_pmd(pte_mkspecial(pmd_pte(pmd)))
#define pmd_mkold(pmd) pte_pmd(pte_mkold(pmd_pte(pmd)))
#define pmd_mkwrite(pmd) pte_pmd(pte_mkwrite(pmd_pte(pmd)))
#define pmd_mkdirty(pmd) pte_pmd(pte_mkdirty(pmd_pte(pmd)))
#define pmd_mkyoung(pmd) pte_pmd(pte_mkyoung(pmd_pte(pmd)))
arm64: mm: remove broken &= operator from pmd_mknotpresent This should be a plain old '&' and could easily lead to undefined behaviour if the target of a pmd_mknotpresent invocation was the same as the parameter. Fixes: 9c7e535fcc17 (arm64: mm: Route pmd thp functions through pte equivalents) Signed-off-by: Will Deacon <will.deacon@arm.com> Cc: <stable@vger.kernel.org> # v3.15 Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-06-18 13:06:27 +00:00
#define pmd_mknotpresent(pmd) (__pmd(pmd_val(pmd) & ~PMD_TYPE_MASK))
ARM64: mm: THP support. Bring Transparent HugePage support to ARM. The size of a transparent huge page depends on the normal page size. A transparent huge page is always represented as a pmd. If PAGE_SIZE is 4KB, THPs are 2MB. If PAGE_SIZE is 64KB, THPs are 512MB. Signed-off-by: Steve Capper <steve.capper@linaro.org> Acked-by: Catalin Marinas <catalin.marinas@arm.com>
2013-04-19 15:23:57 +00:00
arm64: mm: Route pmd thp functions through pte equivalents Rather than have separate hugetlb and transparent huge page pmd manipulation functions, re-wire our thp functions to simply call the pte equivalents. This allows THP to take advantage of the new PTE_WRITE logic introduced in: c2c93e5 arm64: mm: Introduce PTE_WRITE To represent splitting THPs we use the PTE_SPECIAL bit as this is not used for pmds. Signed-off-by: Steve Capper <steve.capper@linaro.org> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-02-25 10:02:13 +00:00
#define __HAVE_ARCH_PMD_WRITE
#define pmd_write(pmd) pte_write(pmd_pte(pmd))
ARM64: mm: THP support. Bring Transparent HugePage support to ARM. The size of a transparent huge page depends on the normal page size. A transparent huge page is always represented as a pmd. If PAGE_SIZE is 4KB, THPs are 2MB. If PAGE_SIZE is 64KB, THPs are 512MB. Signed-off-by: Steve Capper <steve.capper@linaro.org> Acked-by: Catalin Marinas <catalin.marinas@arm.com>
2013-04-19 15:23:57 +00:00
#define pmd_mkhuge(pmd) (__pmd(pmd_val(pmd) & ~PMD_TABLE_BIT))
#define pmd_pfn(pmd) (((pmd_val(pmd) & PMD_MASK) & PHYS_MASK) >> PAGE_SHIFT)
#define pfn_pmd(pfn,prot) (__pmd(((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)))
#define mk_pmd(page,prot) pfn_pmd(page_to_pfn(page),prot)
arm64: mm: enable RCU fast_gup Activate the RCU fast_gup for ARM64. We also need to force THP splits to broadcast an IPI s.t. we block in the fast_gup page walker. As THP splits are comparatively rare, this should not lead to a noticeable performance degradation. Some pre-requisite functions pud_write and pud_page are also added. [akpm@linux-foundation.org: coding-style fixes] Signed-off-by: Steve Capper <steve.capper@linaro.org> Tested-by: Dann Frazier <dann.frazier@canonical.com> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Cc: Hugh Dickins <hughd@google.com> Cc: Russell King <rmk@arm.linux.org.uk> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Mel Gorman <mel@csn.ul.ie> Cc: Will Deacon <will.deacon@arm.com> Cc: Christoffer Dall <christoffer.dall@linaro.org> Cc: Andrea Arcangeli <aarcange@redhat.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2014-10-09 22:29:25 +00:00
#define pud_write(pud) pte_write(pud_pte(pud))
arm64: mm: Create gigabyte kernel logical mappings where possible We have the capability to map 1GB level 1 blocks when using a 4K granule. This patch adjusts the create_mapping logic s.t. when mapping physical memory on boot, we attempt to use a 1GB block if both the VA and PA start and end are 1GB aligned. This both reduces the levels of lookup required to resolve a kernel logical address, as well as reduces TLB pressure on cores that support 1GB TLB entries. Signed-off-by: Steve Capper <steve.capper@linaro.org> Tested-by: Jungseok Lee <jays.lee@samsung.com> [catalin.marinas@arm.com: s/prot_sect_kernel/PROT_SECT_NORMAL_EXEC/] Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-05-06 13:02:27 +00:00
#define pud_pfn(pud) (((pud_val(pud) & PUD_MASK) & PHYS_MASK) >> PAGE_SHIFT)
ARM64: mm: THP support. Bring Transparent HugePage support to ARM. The size of a transparent huge page depends on the normal page size. A transparent huge page is always represented as a pmd. If PAGE_SIZE is 4KB, THPs are 2MB. If PAGE_SIZE is 64KB, THPs are 512MB. Signed-off-by: Steve Capper <steve.capper@linaro.org> Acked-by: Catalin Marinas <catalin.marinas@arm.com>
2013-04-19 15:23:57 +00:00
arm64: mm: fix pmd_write CoW brokenness Commit 9c7e535fcc17 ("arm64: mm: Route pmd thp functions through pte equivalents") changed the pmd manipulator and accessor functions to convert the target pmd to a pte, process it with the pte functions, then convert it back. Along the way, we gained support for PTE_WRITE, however this is completely ignored by set_pmd_at, and so we fail to set the PMD_SECT_RDONLY for PMDs, resulting in all sorts of lovely failures (like CoW not working). Partially reverting the offending commit (by making use of PMD_SECT_RDONLY explicitly for pmd_{write,wrprotect,mkwrite} functions) leads to further issues because pmd_write can then return potentially incorrect values for page table entries marked as RDONLY, leading to BUG_ON(pmd_write(entry)) tripping under some THP workloads. This patch fixes the issue by routing set_pmd_at through set_pte_at, which correctly takes the PTE_WRITE flag into account. Given that THP mappings are always anonymous, the additional cache-flushing code in __sync_icache_dcache won't impose any significant overhead as the flush will be skipped. Cc: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Steve Capper <steve.capper@arm.com> Tested-by: Marc Zyngier <marc.zyngier@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com>
2014-05-27 18:11:58 +00:00
#define set_pmd_at(mm, addr, pmdp, pmd) set_pte_at(mm, addr, (pte_t *)pmdp, pmd_pte(pmd))
ARM64: mm: THP support. Bring Transparent HugePage support to ARM. The size of a transparent huge page depends on the normal page size. A transparent huge page is always represented as a pmd. If PAGE_SIZE is 4KB, THPs are 2MB. If PAGE_SIZE is 64KB, THPs are 512MB. Signed-off-by: Steve Capper <steve.capper@linaro.org> Acked-by: Catalin Marinas <catalin.marinas@arm.com>
2013-04-19 15:23:57 +00:00
static inline int has_transparent_hugepage(void)
{
return 1;
}
arm64: Clean up the default pgprot setting The primary aim of this patchset is to remove the pgprot_default and prot_sect_default global variables and rely strictly on predefined values. The original goal was to be able to run SMP kernels on UP hardware by not setting the Shareability bit. However, it is unlikely to see UP ARMv8 hardware and even if we do, the Shareability bit is no longer assumed to disable cacheable accesses. A side effect is that the device mappings now have the Shareability attribute set. The hardware, however, should ignore it since Device accesses are always Outer Shareable. Following the removal of the two global variables, there is some PROT_* macro reshuffling and cleanup, including the __PAGE_* macros (replaced by PAGE_*). Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Will Deacon <will.deacon@arm.com>
2014-04-03 14:57:15 +00:00
#define __pgprot_modify(prot,mask,bits) \
__pgprot((pgprot_val(prot) & ~(mask)) | (bits))
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
/*
* Mark the prot value as uncacheable and unbufferable.
*/
#define pgprot_noncached(prot) \
arm64: Make DMA coherent and strongly ordered mappings not executable pgprot_{dmacoherent,writecombine,noncached} don't need to generate executable mappings with side-effects like __sync_icache_dcache() being called when the mapping is in user space. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Reported-by: Bharat Bhushan <Bharat.Bhushan@freescale.com> Tested-by: Laura Abbott <lauraa@codeaurora.org> Tested-by: Bharat Bhushan <Bharat.Bhushan@freescale.com> Cc: <stable@vger.kernel.org>
2014-03-12 16:07:06 +00:00
__pgprot_modify(prot, PTE_ATTRINDX_MASK, PTE_ATTRINDX(MT_DEVICE_nGnRnE) | PTE_PXN | PTE_UXN)
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
#define pgprot_writecombine(prot) \
arm64: Make DMA coherent and strongly ordered mappings not executable pgprot_{dmacoherent,writecombine,noncached} don't need to generate executable mappings with side-effects like __sync_icache_dcache() being called when the mapping is in user space. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Reported-by: Bharat Bhushan <Bharat.Bhushan@freescale.com> Tested-by: Laura Abbott <lauraa@codeaurora.org> Tested-by: Bharat Bhushan <Bharat.Bhushan@freescale.com> Cc: <stable@vger.kernel.org>
2014-03-12 16:07:06 +00:00
__pgprot_modify(prot, PTE_ATTRINDX_MASK, PTE_ATTRINDX(MT_NORMAL_NC) | PTE_PXN | PTE_UXN)
arm64: Add architectural support for PCI Use the generic PCI domain and OF functions to provide support for PCI on arm64. [bhelgaas: Change comments to use generic PCI, not just PCIe. Nothing at this level is PCIe-specific.] Signed-off-by: Liviu Dudau <Liviu.Dudau@arm.com> Signed-off-by: Bjorn Helgaas <bhelgaas@google.com> Acked-by: Catalin Marinas <catalin.marinas@arm.com>
2014-09-29 14:29:31 +00:00
#define pgprot_device(prot) \
__pgprot_modify(prot, PTE_ATTRINDX_MASK, PTE_ATTRINDX(MT_DEVICE_nGnRE) | PTE_PXN | PTE_UXN)
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
#define __HAVE_PHYS_MEM_ACCESS_PROT
struct file;
extern pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn,
unsigned long size, pgprot_t vma_prot);
#define pmd_none(pmd) (!pmd_val(pmd))
#define pmd_present(pmd) (pmd_val(pmd))
#define pmd_bad(pmd) (!(pmd_val(pmd) & 2))
arm64: KVM: define HYP and Stage-2 translation page flags Add HYP and S2 page flags, for both normal and device memory. Reviewed-by: Christopher Covington <cov@codeaurora.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2012-12-07 18:35:41 +00:00
#define pmd_table(pmd) ((pmd_val(pmd) & PMD_TYPE_MASK) == \
PMD_TYPE_TABLE)
#define pmd_sect(pmd) ((pmd_val(pmd) & PMD_TYPE_MASK) == \
PMD_TYPE_SECT)
arm64: mm: Fix horrendous config typo The define ARM64_64K_PAGES is tested for rather than CONFIG_ARM64_64K_PAGES. Correct that typo here. Signed-off-by: Steve Capper <steve.capper@linaro.org> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-06-25 07:41:45 +00:00
#ifdef CONFIG_ARM64_64K_PAGES
arm64: mm: Create gigabyte kernel logical mappings where possible We have the capability to map 1GB level 1 blocks when using a 4K granule. This patch adjusts the create_mapping logic s.t. when mapping physical memory on boot, we attempt to use a 1GB block if both the VA and PA start and end are 1GB aligned. This both reduces the levels of lookup required to resolve a kernel logical address, as well as reduces TLB pressure on cores that support 1GB TLB entries. Signed-off-by: Steve Capper <steve.capper@linaro.org> Tested-by: Jungseok Lee <jays.lee@samsung.com> [catalin.marinas@arm.com: s/prot_sect_kernel/PROT_SECT_NORMAL_EXEC/] Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-05-06 13:02:27 +00:00
#define pud_sect(pud) (0)
arm64:mm: free the useless initial page table For 64K page system, after mapping a PMD section, the corresponding initial page table is not needed any more. That page can be freed. Signed-off-by: Zhichang Yuan <zhichang.yuan@linaro.org> [catalin.marinas@arm.com: added BUG_ON() to catch late memblock freeing] Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-12-09 07:26:47 +00:00
#define pud_table(pud) (1)
arm64: mm: Create gigabyte kernel logical mappings where possible We have the capability to map 1GB level 1 blocks when using a 4K granule. This patch adjusts the create_mapping logic s.t. when mapping physical memory on boot, we attempt to use a 1GB block if both the VA and PA start and end are 1GB aligned. This both reduces the levels of lookup required to resolve a kernel logical address, as well as reduces TLB pressure on cores that support 1GB TLB entries. Signed-off-by: Steve Capper <steve.capper@linaro.org> Tested-by: Jungseok Lee <jays.lee@samsung.com> [catalin.marinas@arm.com: s/prot_sect_kernel/PROT_SECT_NORMAL_EXEC/] Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-05-06 13:02:27 +00:00
#else
#define pud_sect(pud) ((pud_val(pud) & PUD_TYPE_MASK) == \
PUD_TYPE_SECT)
arm64:mm: free the useless initial page table For 64K page system, after mapping a PMD section, the corresponding initial page table is not needed any more. That page can be freed. Signed-off-by: Zhichang Yuan <zhichang.yuan@linaro.org> [catalin.marinas@arm.com: added BUG_ON() to catch late memblock freeing] Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-12-09 07:26:47 +00:00
#define pud_table(pud) ((pud_val(pud) & PUD_TYPE_MASK) == \
PUD_TYPE_TABLE)
arm64: mm: Create gigabyte kernel logical mappings where possible We have the capability to map 1GB level 1 blocks when using a 4K granule. This patch adjusts the create_mapping logic s.t. when mapping physical memory on boot, we attempt to use a 1GB block if both the VA and PA start and end are 1GB aligned. This both reduces the levels of lookup required to resolve a kernel logical address, as well as reduces TLB pressure on cores that support 1GB TLB entries. Signed-off-by: Steve Capper <steve.capper@linaro.org> Tested-by: Jungseok Lee <jays.lee@samsung.com> [catalin.marinas@arm.com: s/prot_sect_kernel/PROT_SECT_NORMAL_EXEC/] Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-05-06 13:02:27 +00:00
#endif
arm64: KVM: define HYP and Stage-2 translation page flags Add HYP and S2 page flags, for both normal and device memory. Reviewed-by: Christopher Covington <cov@codeaurora.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2012-12-07 18:35:41 +00:00
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
static inline void set_pmd(pmd_t *pmdp, pmd_t pmd)
{
*pmdp = pmd;
arm64: barriers: make use of barrier options with explicit barriers When calling our low-level barrier macros directly, we can often suffice with more relaxed behaviour than the default "all accesses, full system" option. This patch updates the users of dsb() to specify the option which they actually require. Acked-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-05-02 15:24:10 +00:00
dsb(ishst);
arm64: Fix barriers used for page table modifications The architecture specification states that both DSB and ISB are required between page table modifications and subsequent memory accesses using the corresponding virtual address. When TLB invalidation takes place, the tlb_flush_* functions already have the necessary barriers. However, there are other functions like create_mapping() for which this is not the case. The patch adds the DSB+ISB instructions in the set_pte() function for valid kernel mappings. The invalid pte case is handled by tlb_flush_* and the user mappings in general have a corresponding update_mmu_cache() call containing a DSB. Even when update_mmu_cache() isn't called, the kernel can still cope with an unlikely spurious page fault by re-executing the instruction. In addition, the set_pmd, set_pud() functions gain an ISB for architecture compliance when block mappings are created. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Reported-by: Leif Lindholm <leif.lindholm@linaro.org> Acked-by: Steve Capper <steve.capper@linaro.org> Cc: Will Deacon <will.deacon@arm.com> Cc: <stable@vger.kernel.org>
2014-06-09 10:55:03 +00:00
isb();
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
}
static inline void pmd_clear(pmd_t *pmdp)
{
set_pmd(pmdp, __pmd(0));
}
static inline pte_t *pmd_page_vaddr(pmd_t pmd)
{
return __va(pmd_val(pmd) & PHYS_MASK & (s32)PAGE_MASK);
}
#define pmd_page(pmd) pfn_to_page(__phys_to_pfn(pmd_val(pmd) & PHYS_MASK))
/*
* Conversion functions: convert a page and protection to a page entry,
* and a page entry and page directory to the page they refer to.
*/
#define mk_pte(page,prot) pfn_pte(page_to_pfn(page),prot)
arm64: Convert bool ARM64_x_LEVELS to int ARM64_PGTABLE_LEVELS Rather than having several Kconfig options, define int ARM64_PGTABLE_LEVELS which will be also useful in converting some of the pgtable macros. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Tested-by: Jungseok Lee <jungseoklee85@gmail.com>
2014-07-15 14:37:21 +00:00
#if CONFIG_ARM64_PGTABLE_LEVELS > 2
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
arm64: asm/pgtable.h pmd/pud definitions clean-up Non-functional change to group together the pmd/pud definitions and reduce the amount of #if CONFIG_ARM64_PGTABLE_LEVELS. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Tested-by: Jungseok Lee <jungseoklee85@gmail.com>
2014-07-21 13:52:49 +00:00
#define pmd_ERROR(pmd) __pmd_error(__FILE__, __LINE__, pmd_val(pmd))
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
#define pud_none(pud) (!pud_val(pud))
#define pud_bad(pud) (!(pud_val(pud) & 2))
#define pud_present(pud) (pud_val(pud))
static inline void set_pud(pud_t *pudp, pud_t pud)
{
*pudp = pud;
arm64: barriers: make use of barrier options with explicit barriers When calling our low-level barrier macros directly, we can often suffice with more relaxed behaviour than the default "all accesses, full system" option. This patch updates the users of dsb() to specify the option which they actually require. Acked-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-05-02 15:24:10 +00:00
dsb(ishst);
arm64: Fix barriers used for page table modifications The architecture specification states that both DSB and ISB are required between page table modifications and subsequent memory accesses using the corresponding virtual address. When TLB invalidation takes place, the tlb_flush_* functions already have the necessary barriers. However, there are other functions like create_mapping() for which this is not the case. The patch adds the DSB+ISB instructions in the set_pte() function for valid kernel mappings. The invalid pte case is handled by tlb_flush_* and the user mappings in general have a corresponding update_mmu_cache() call containing a DSB. Even when update_mmu_cache() isn't called, the kernel can still cope with an unlikely spurious page fault by re-executing the instruction. In addition, the set_pmd, set_pud() functions gain an ISB for architecture compliance when block mappings are created. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Reported-by: Leif Lindholm <leif.lindholm@linaro.org> Acked-by: Steve Capper <steve.capper@linaro.org> Cc: Will Deacon <will.deacon@arm.com> Cc: <stable@vger.kernel.org>
2014-06-09 10:55:03 +00:00
isb();
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
}
static inline void pud_clear(pud_t *pudp)
{
set_pud(pudp, __pud(0));
}
static inline pmd_t *pud_page_vaddr(pud_t pud)
{
return __va(pud_val(pud) & PHYS_MASK & (s32)PAGE_MASK);
}
arm64: asm/pgtable.h pmd/pud definitions clean-up Non-functional change to group together the pmd/pud definitions and reduce the amount of #if CONFIG_ARM64_PGTABLE_LEVELS. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Tested-by: Jungseok Lee <jungseoklee85@gmail.com>
2014-07-21 13:52:49 +00:00
/* Find an entry in the second-level page table. */
#define pmd_index(addr) (((addr) >> PMD_SHIFT) & (PTRS_PER_PMD - 1))
static inline pmd_t *pmd_offset(pud_t *pud, unsigned long addr)
{
return (pmd_t *)pud_page_vaddr(*pud) + pmd_index(addr);
}
arm64: mm: Add pgd_page to support RCU fast_gup This patch adds pgd_page definition in order to keep supporting HAVE_GENERIC_RCU_GUP configuration. In addition, it changes pud_page expression to align with pmd_page for readability. An introduction of pgd_page resolves the following build breakage under 4KB + 4Level memory management combo. mm/gup.c: In function 'gup_huge_pgd': mm/gup.c:889:2: error: implicit declaration of function 'pgd_page' [-Werror=implicit-function-declaration] head = pgd_page(orig); ^ mm/gup.c:889:7: warning: assignment makes pointer from integer without a cast head = pgd_page(orig); Cc: Will Deacon <will.deacon@arm.com> Cc: Steve Capper <steve.capper@linaro.org> Signed-off-by: Jungseok Lee <jungseoklee85@gmail.com> [catalin.marinas@arm.com: remove duplicate pmd_page definition] Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-12-20 00:49:40 +00:00
#define pud_page(pud) pfn_to_page(__phys_to_pfn(pud_val(pud) & PHYS_MASK))
arm64: mm: enable RCU fast_gup Activate the RCU fast_gup for ARM64. We also need to force THP splits to broadcast an IPI s.t. we block in the fast_gup page walker. As THP splits are comparatively rare, this should not lead to a noticeable performance degradation. Some pre-requisite functions pud_write and pud_page are also added. [akpm@linux-foundation.org: coding-style fixes] Signed-off-by: Steve Capper <steve.capper@linaro.org> Tested-by: Dann Frazier <dann.frazier@canonical.com> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Cc: Hugh Dickins <hughd@google.com> Cc: Russell King <rmk@arm.linux.org.uk> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Mel Gorman <mel@csn.ul.ie> Cc: Will Deacon <will.deacon@arm.com> Cc: Christoffer Dall <christoffer.dall@linaro.org> Cc: Andrea Arcangeli <aarcange@redhat.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2014-10-09 22:29:25 +00:00
arm64: Convert bool ARM64_x_LEVELS to int ARM64_PGTABLE_LEVELS Rather than having several Kconfig options, define int ARM64_PGTABLE_LEVELS which will be also useful in converting some of the pgtable macros. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Tested-by: Jungseok Lee <jungseoklee85@gmail.com>
2014-07-15 14:37:21 +00:00
#endif /* CONFIG_ARM64_PGTABLE_LEVELS > 2 */
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
arm64: Convert bool ARM64_x_LEVELS to int ARM64_PGTABLE_LEVELS Rather than having several Kconfig options, define int ARM64_PGTABLE_LEVELS which will be also useful in converting some of the pgtable macros. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Tested-by: Jungseok Lee <jungseoklee85@gmail.com>
2014-07-15 14:37:21 +00:00
#if CONFIG_ARM64_PGTABLE_LEVELS > 3
arm64: mm: Implement 4 levels of translation tables This patch implements 4 levels of translation tables since 3 levels of page tables with 4KB pages cannot support 40-bit physical address space described in [1] due to the following issue. It is a restriction that kernel logical memory map with 4KB + 3 levels (0xffffffc000000000-0xffffffffffffffff) cannot cover RAM region from 544GB to 1024GB in [1]. Specifically, ARM64 kernel fails to create mapping for this region in map_mem function since __phys_to_virt for this region reaches to address overflow. If SoC design follows the document, [1], over 32GB RAM would be placed from 544GB. Even 64GB system is supposed to use the region from 544GB to 576GB for only 32GB RAM. Naturally, it would reach to enable 4 levels of page tables to avoid hacking __virt_to_phys and __phys_to_virt. However, it is recommended 4 levels of page table should be only enabled if memory map is too sparse or there is about 512GB RAM. References ---------- [1]: Principles of ARM Memory Maps, White Paper, Issue C Signed-off-by: Jungseok Lee <jays.lee@samsung.com> Reviewed-by: Sungjinn Chung <sungjinn.chung@samsung.com> Acked-by: Kukjin Kim <kgene.kim@samsung.com> Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org> Reviewed-by: Steve Capper <steve.capper@linaro.org> [catalin.marinas@arm.com: MEMBLOCK_INITIAL_LIMIT removed, same as PUD_SIZE] [catalin.marinas@arm.com: early_ioremap_init() updated for 4 levels] [catalin.marinas@arm.com: 48-bit VA depends on BROKEN until KVM is fixed] Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Tested-by: Jungseok Lee <jungseoklee85@gmail.com>
2014-05-12 09:40:51 +00:00
arm64: asm/pgtable.h pmd/pud definitions clean-up Non-functional change to group together the pmd/pud definitions and reduce the amount of #if CONFIG_ARM64_PGTABLE_LEVELS. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Tested-by: Jungseok Lee <jungseoklee85@gmail.com>
2014-07-21 13:52:49 +00:00
#define pud_ERROR(pud) __pud_error(__FILE__, __LINE__, pud_val(pud))
arm64: mm: Implement 4 levels of translation tables This patch implements 4 levels of translation tables since 3 levels of page tables with 4KB pages cannot support 40-bit physical address space described in [1] due to the following issue. It is a restriction that kernel logical memory map with 4KB + 3 levels (0xffffffc000000000-0xffffffffffffffff) cannot cover RAM region from 544GB to 1024GB in [1]. Specifically, ARM64 kernel fails to create mapping for this region in map_mem function since __phys_to_virt for this region reaches to address overflow. If SoC design follows the document, [1], over 32GB RAM would be placed from 544GB. Even 64GB system is supposed to use the region from 544GB to 576GB for only 32GB RAM. Naturally, it would reach to enable 4 levels of page tables to avoid hacking __virt_to_phys and __phys_to_virt. However, it is recommended 4 levels of page table should be only enabled if memory map is too sparse or there is about 512GB RAM. References ---------- [1]: Principles of ARM Memory Maps, White Paper, Issue C Signed-off-by: Jungseok Lee <jays.lee@samsung.com> Reviewed-by: Sungjinn Chung <sungjinn.chung@samsung.com> Acked-by: Kukjin Kim <kgene.kim@samsung.com> Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org> Reviewed-by: Steve Capper <steve.capper@linaro.org> [catalin.marinas@arm.com: MEMBLOCK_INITIAL_LIMIT removed, same as PUD_SIZE] [catalin.marinas@arm.com: early_ioremap_init() updated for 4 levels] [catalin.marinas@arm.com: 48-bit VA depends on BROKEN until KVM is fixed] Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Tested-by: Jungseok Lee <jungseoklee85@gmail.com>
2014-05-12 09:40:51 +00:00
#define pgd_none(pgd) (!pgd_val(pgd))
#define pgd_bad(pgd) (!(pgd_val(pgd) & 2))
#define pgd_present(pgd) (pgd_val(pgd))
static inline void set_pgd(pgd_t *pgdp, pgd_t pgd)
{
*pgdp = pgd;
dsb(ishst);
}
static inline void pgd_clear(pgd_t *pgdp)
{
set_pgd(pgdp, __pgd(0));
}
static inline pud_t *pgd_page_vaddr(pgd_t pgd)
{
return __va(pgd_val(pgd) & PHYS_MASK & (s32)PAGE_MASK);
}
arm64: asm/pgtable.h pmd/pud definitions clean-up Non-functional change to group together the pmd/pud definitions and reduce the amount of #if CONFIG_ARM64_PGTABLE_LEVELS. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Tested-by: Jungseok Lee <jungseoklee85@gmail.com>
2014-07-21 13:52:49 +00:00
/* Find an entry in the frst-level page table. */
#define pud_index(addr) (((addr) >> PUD_SHIFT) & (PTRS_PER_PUD - 1))
static inline pud_t *pud_offset(pgd_t *pgd, unsigned long addr)
{
return (pud_t *)pgd_page_vaddr(*pgd) + pud_index(addr);
}
arm64: mm: Add pgd_page to support RCU fast_gup This patch adds pgd_page definition in order to keep supporting HAVE_GENERIC_RCU_GUP configuration. In addition, it changes pud_page expression to align with pmd_page for readability. An introduction of pgd_page resolves the following build breakage under 4KB + 4Level memory management combo. mm/gup.c: In function 'gup_huge_pgd': mm/gup.c:889:2: error: implicit declaration of function 'pgd_page' [-Werror=implicit-function-declaration] head = pgd_page(orig); ^ mm/gup.c:889:7: warning: assignment makes pointer from integer without a cast head = pgd_page(orig); Cc: Will Deacon <will.deacon@arm.com> Cc: Steve Capper <steve.capper@linaro.org> Signed-off-by: Jungseok Lee <jungseoklee85@gmail.com> [catalin.marinas@arm.com: remove duplicate pmd_page definition] Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-12-20 00:49:40 +00:00
#define pgd_page(pgd) pfn_to_page(__phys_to_pfn(pgd_val(pgd) & PHYS_MASK))
arm64: Convert bool ARM64_x_LEVELS to int ARM64_PGTABLE_LEVELS Rather than having several Kconfig options, define int ARM64_PGTABLE_LEVELS which will be also useful in converting some of the pgtable macros. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Tested-by: Jungseok Lee <jungseoklee85@gmail.com>
2014-07-15 14:37:21 +00:00
#endif /* CONFIG_ARM64_PGTABLE_LEVELS > 3 */
arm64: mm: Implement 4 levels of translation tables This patch implements 4 levels of translation tables since 3 levels of page tables with 4KB pages cannot support 40-bit physical address space described in [1] due to the following issue. It is a restriction that kernel logical memory map with 4KB + 3 levels (0xffffffc000000000-0xffffffffffffffff) cannot cover RAM region from 544GB to 1024GB in [1]. Specifically, ARM64 kernel fails to create mapping for this region in map_mem function since __phys_to_virt for this region reaches to address overflow. If SoC design follows the document, [1], over 32GB RAM would be placed from 544GB. Even 64GB system is supposed to use the region from 544GB to 576GB for only 32GB RAM. Naturally, it would reach to enable 4 levels of page tables to avoid hacking __virt_to_phys and __phys_to_virt. However, it is recommended 4 levels of page table should be only enabled if memory map is too sparse or there is about 512GB RAM. References ---------- [1]: Principles of ARM Memory Maps, White Paper, Issue C Signed-off-by: Jungseok Lee <jays.lee@samsung.com> Reviewed-by: Sungjinn Chung <sungjinn.chung@samsung.com> Acked-by: Kukjin Kim <kgene.kim@samsung.com> Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org> Reviewed-by: Steve Capper <steve.capper@linaro.org> [catalin.marinas@arm.com: MEMBLOCK_INITIAL_LIMIT removed, same as PUD_SIZE] [catalin.marinas@arm.com: early_ioremap_init() updated for 4 levels] [catalin.marinas@arm.com: 48-bit VA depends on BROKEN until KVM is fixed] Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Tested-by: Jungseok Lee <jungseoklee85@gmail.com>
2014-05-12 09:40:51 +00:00
arm64: asm/pgtable.h pmd/pud definitions clean-up Non-functional change to group together the pmd/pud definitions and reduce the amount of #if CONFIG_ARM64_PGTABLE_LEVELS. Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Tested-by: Jungseok Lee <jungseoklee85@gmail.com>
2014-07-21 13:52:49 +00:00
#define pgd_ERROR(pgd) __pgd_error(__FILE__, __LINE__, pgd_val(pgd))
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
/* to find an entry in a page-table-directory */
#define pgd_index(addr) (((addr) >> PGDIR_SHIFT) & (PTRS_PER_PGD - 1))
#define pgd_offset(mm, addr) ((mm)->pgd+pgd_index(addr))
/* to find an entry in a kernel page-table-directory */
#define pgd_offset_k(addr) pgd_offset(&init_mm, addr)
static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)
{
arm64: mm: introduce present, faulting entries for PAGE_NONE This is mostly a port of dbf62d50067e ("ARM: mm: introduce L_PTE_VALID for page table entries") and 26ffd0d43b18 ("ARM: mm: introduce present, faulting entries for PAGE_NONE") from ARM, which makes use of present, faulting page table entries for page table entries mapped as PROT_NONE. The main difference with this implementation is that we can make use of the two pte type bits in order to avoid allocating a software bit for identifying PROT_NONE pages, instead reserving the 10b suffix for these types of mappings. This is required to prevent users from accessing such pages via syscalls such as read/write over a pipe. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2012-12-18 14:15:15 +00:00
const pteval_t mask = PTE_USER | PTE_PXN | PTE_UXN | PTE_RDONLY |
arm64: enable PTE type bit in the mask for pte_modify Caught during Trinity testing. The pte_modify does not allow modification for PTE type bit. This cause the test to hang the system. It is found that the PTE can't transit from an inaccessible page (b00) to a valid page (b11) because the mask does not allow it. This happens when a big block of mmaped memory is set the PROT_NONE, then the a small piece is broken off and set to PROT_WRITE | PROT_READ cause a huge page split. Signed-off-by: Feng Kan <fkan@apm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-02-24 23:40:21 +00:00
PTE_PROT_NONE | PTE_WRITE | PTE_TYPE_MASK;
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
pte_val(pte) = (pte_val(pte) & ~mask) | (pgprot_val(newprot) & mask);
return pte;
}
arm64: mm: Route pmd thp functions through pte equivalents Rather than have separate hugetlb and transparent huge page pmd manipulation functions, re-wire our thp functions to simply call the pte equivalents. This allows THP to take advantage of the new PTE_WRITE logic introduced in: c2c93e5 arm64: mm: Introduce PTE_WRITE To represent splitting THPs we use the PTE_SPECIAL bit as this is not used for pmds. Signed-off-by: Steve Capper <steve.capper@linaro.org> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-02-25 10:02:13 +00:00
static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot)
{
return pte_pmd(pte_modify(pmd_pte(pmd), newprot));
}
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
extern pgd_t swapper_pg_dir[PTRS_PER_PGD];
extern pgd_t idmap_pg_dir[PTRS_PER_PGD];
/*
* Encode and decode a swap entry:
arm64: Move PTE_PROT_NONE higher up PTE_PROT_NONE means that a pte is present but does not have any read/write attributes. However, setting the memory type like pgprot_writecombine() is allowed and such bits overlap with PTE_PROT_NONE. This causes mmap/munmap issues in drivers that change the vma->vm_pg_prot on PROT_NONE mappings. This patch reverts the PTE_FILE/PTE_PROT_NONE shift in commit 59911ca4325d (ARM64: mm: Move PTE_PROT_NONE bit) and moves PTE_PROT_NONE together with the other software bits. Signed-off-by: Steve Capper <steve.capper@linaro.org> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Tested-by: Steve Capper <steve.capper@linaro.org> Cc: <stable@vger.kernel.org> # 3.11+
2013-11-27 16:59:27 +00:00
* bits 0-1: present (must be zero)
arm64: drop PTE_FILE and pte_file()-related helpers We've replaced remap_file_pages(2) implementation with emulation. Nobody creates non-linear mapping anymore. This patch also adjust __SWP_TYPE_SHIFT and increase number of bits availble for swap offset. Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Cc: Will Deacon <will.deacon@arm.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2015-02-10 22:10:15 +00:00
* bits 2-7: swap type
* bits 8-57: swap offset
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
*/
arm64: drop PTE_FILE and pte_file()-related helpers We've replaced remap_file_pages(2) implementation with emulation. Nobody creates non-linear mapping anymore. This patch also adjust __SWP_TYPE_SHIFT and increase number of bits availble for swap offset. Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Cc: Will Deacon <will.deacon@arm.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2015-02-10 22:10:15 +00:00
#define __SWP_TYPE_SHIFT 2
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
#define __SWP_TYPE_BITS 6
arm64: drop PTE_FILE and pte_file()-related helpers We've replaced remap_file_pages(2) implementation with emulation. Nobody creates non-linear mapping anymore. This patch also adjust __SWP_TYPE_SHIFT and increase number of bits availble for swap offset. Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Cc: Will Deacon <will.deacon@arm.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2015-02-10 22:10:15 +00:00
#define __SWP_OFFSET_BITS 50
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
#define __SWP_TYPE_MASK ((1 << __SWP_TYPE_BITS) - 1)
#define __SWP_OFFSET_SHIFT (__SWP_TYPE_BITS + __SWP_TYPE_SHIFT)
arm64: Move PTE_PROT_NONE higher up PTE_PROT_NONE means that a pte is present but does not have any read/write attributes. However, setting the memory type like pgprot_writecombine() is allowed and such bits overlap with PTE_PROT_NONE. This causes mmap/munmap issues in drivers that change the vma->vm_pg_prot on PROT_NONE mappings. This patch reverts the PTE_FILE/PTE_PROT_NONE shift in commit 59911ca4325d (ARM64: mm: Move PTE_PROT_NONE bit) and moves PTE_PROT_NONE together with the other software bits. Signed-off-by: Steve Capper <steve.capper@linaro.org> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Tested-by: Steve Capper <steve.capper@linaro.org> Cc: <stable@vger.kernel.org> # 3.11+
2013-11-27 16:59:27 +00:00
#define __SWP_OFFSET_MASK ((1UL << __SWP_OFFSET_BITS) - 1)
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
#define __swp_type(x) (((x).val >> __SWP_TYPE_SHIFT) & __SWP_TYPE_MASK)
arm64: Move PTE_PROT_NONE higher up PTE_PROT_NONE means that a pte is present but does not have any read/write attributes. However, setting the memory type like pgprot_writecombine() is allowed and such bits overlap with PTE_PROT_NONE. This causes mmap/munmap issues in drivers that change the vma->vm_pg_prot on PROT_NONE mappings. This patch reverts the PTE_FILE/PTE_PROT_NONE shift in commit 59911ca4325d (ARM64: mm: Move PTE_PROT_NONE bit) and moves PTE_PROT_NONE together with the other software bits. Signed-off-by: Steve Capper <steve.capper@linaro.org> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Tested-by: Steve Capper <steve.capper@linaro.org> Cc: <stable@vger.kernel.org> # 3.11+
2013-11-27 16:59:27 +00:00
#define __swp_offset(x) (((x).val >> __SWP_OFFSET_SHIFT) & __SWP_OFFSET_MASK)
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
#define __swp_entry(type,offset) ((swp_entry_t) { ((type) << __SWP_TYPE_SHIFT) | ((offset) << __SWP_OFFSET_SHIFT) })
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
#define __swp_entry_to_pte(swp) ((pte_t) { (swp).val })
/*
* Ensure that there are not more swap files than can be encoded in the kernel
arm64: mm: Remove superfluous "the" in comment Signed-off-by: Geert Uytterhoeven <geert+renesas@linux-m68k.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: linux-arm-kernel@lists.infradead.org Signed-off-by: Jiri Kosina <jkosina@suse.cz>
2014-03-11 10:23:39 +00:00
* PTEs.
arm64: MMU definitions The virtual memory layout is described in Documentation/arm64/memory.txt. This patch adds the MMU definitions for the 4KB and 64KB translation table configurations. The SECTION_SIZE is 2MB with 4KB page and 512MB with 64KB page configuration. PHYS_OFFSET is calculated at run-time and stored in a variable (no run-time code patching at this stage). On the current implementation, both user and kernel address spaces are 512G (39-bit) each with a maximum of 256G for the RAM linear mapping. Linux uses 3 levels of translation tables with the 4K page configuration and 2 levels with the 64K configuration. Extending the memory space beyond 39-bit with the 4K pages or 42-bit with 64K pages requires an additional level of translation tables. The SPARSEMEM configuration is global to all AArch64 platforms and allows for 1GB sections with SPARSEMEM_VMEMMAP enabled by default. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com> Acked-by: Arnd Bergmann <arnd@arndb.de>
2012-03-05 11:49:27 +00:00
*/
#define MAX_SWAPFILES_CHECK() BUILD_BUG_ON(MAX_SWAPFILES_SHIFT > __SWP_TYPE_BITS)
extern int kern_addr_valid(unsigned long addr);
#include <asm-generic/pgtable.h>
#define pgtable_cache_init() do { } while (0)
#endif /* !__ASSEMBLY__ */
#endif /* __ASM_PGTABLE_H */
Reference in New Issue Copy Permalink