2005-04-16 22:20:36 +00:00
|
|
|
/*
|
2011-06-28 09:54:48 +00:00
|
|
|
* PPC Huge TLB Page Support for Kernel.
|
2005-04-16 22:20:36 +00:00
|
|
|
*
|
|
|
|
* Copyright (C) 2003 David Gibson, IBM Corporation.
|
2011-06-28 09:54:48 +00:00
|
|
|
* Copyright (C) 2011 Becky Bruce, Freescale Semiconductor
|
2005-04-16 22:20:36 +00:00
|
|
|
*
|
|
|
|
* Based on the IA-32 version:
|
|
|
|
* Copyright (C) 2002, Rohit Seth <rohit.seth@intel.com>
|
|
|
|
*/
|
|
|
|
|
|
|
|
#include <linux/mm.h>
|
2009-10-26 19:24:31 +00:00
|
|
|
#include <linux/io.h>
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 08:04:11 +00:00
|
|
|
#include <linux/slab.h>
|
2005-04-16 22:20:36 +00:00
|
|
|
#include <linux/hugetlb.h>
|
KVM: PPC: Implement MMU notifiers for Book3S HV guests
This adds the infrastructure to enable us to page out pages underneath
a Book3S HV guest, on processors that support virtualized partition
memory, that is, POWER7. Instead of pinning all the guest's pages,
we now look in the host userspace Linux page tables to find the
mapping for a given guest page. Then, if the userspace Linux PTE
gets invalidated, kvm_unmap_hva() gets called for that address, and
we replace all the guest HPTEs that refer to that page with absent
HPTEs, i.e. ones with the valid bit clear and the HPTE_V_ABSENT bit
set, which will cause an HDSI when the guest tries to access them.
Finally, the page fault handler is extended to reinstantiate the
guest HPTE when the guest tries to access a page which has been paged
out.
Since we can't intercept the guest DSI and ISI interrupts on PPC970,
we still have to pin all the guest pages on PPC970. We have a new flag,
kvm->arch.using_mmu_notifiers, that indicates whether we can page
guest pages out. If it is not set, the MMU notifier callbacks do
nothing and everything operates as before.
Signed-off-by: Paul Mackerras <paulus@samba.org>
Signed-off-by: Alexander Graf <agraf@suse.de>
Signed-off-by: Avi Kivity <avi@redhat.com>
2011-12-12 12:38:05 +00:00
|
|
|
#include <linux/export.h>
|
2011-06-28 09:54:48 +00:00
|
|
|
#include <linux/of_fdt.h>
|
|
|
|
#include <linux/memblock.h>
|
2011-11-24 09:40:07 +00:00
|
|
|
#include <linux/moduleparam.h>
|
2017-07-06 22:38:59 +00:00
|
|
|
#include <linux/swap.h>
|
|
|
|
#include <linux/swapops.h>
|
2018-08-13 13:19:52 +00:00
|
|
|
#include <linux/kmemleak.h>
|
2005-04-16 22:20:36 +00:00
|
|
|
#include <asm/pgalloc.h>
|
|
|
|
#include <asm/tlb.h>
|
2011-06-28 09:54:48 +00:00
|
|
|
#include <asm/setup.h>
|
2013-06-20 09:00:16 +00:00
|
|
|
#include <asm/hugetlb.h>
|
2017-07-27 06:24:53 +00:00
|
|
|
#include <asm/pte-walk.h>
|
|
|
|
|
2018-04-10 13:41:31 +00:00
|
|
|
bool hugetlb_disabled = false;
|
|
|
|
|
2016-12-14 04:37:53 +00:00
|
|
|
#define hugepd_none(hpd) (hpd_val(hpd) == 0)
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 19:24:31 +00:00
|
|
|
|
2020-05-19 05:49:06 +00:00
|
|
|
#define PTE_T_ORDER (__builtin_ffs(sizeof(pte_basic_t)) - \
|
|
|
|
__builtin_ffs(sizeof(void *)))
|
2018-11-29 14:07:05 +00:00
|
|
|
|
2017-07-06 22:39:42 +00:00
|
|
|
pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr, unsigned long sz)
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 19:24:31 +00:00
|
|
|
{
|
2017-07-27 06:24:53 +00:00
|
|
|
/*
|
|
|
|
* Only called for hugetlbfs pages, hence can ignore THP and the
|
|
|
|
* irq disabled walk.
|
|
|
|
*/
|
|
|
|
return __find_linux_pte(mm->pgd, addr, NULL, NULL);
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 19:24:31 +00:00
|
|
|
}
|
|
|
|
|
2006-04-28 05:02:51 +00:00
|
|
|
static int __hugepte_alloc(struct mm_struct *mm, hugepd_t *hpdp,
|
2018-06-01 08:24:24 +00:00
|
|
|
unsigned long address, unsigned int pdshift,
|
|
|
|
unsigned int pshift, spinlock_t *ptl)
|
2006-04-28 05:02:51 +00:00
|
|
|
{
|
2011-06-28 09:54:48 +00:00
|
|
|
struct kmem_cache *cachep;
|
|
|
|
pte_t *new;
|
|
|
|
int i;
|
2016-12-07 07:47:26 +00:00
|
|
|
int num_hugepd;
|
|
|
|
|
|
|
|
if (pshift >= pdshift) {
|
2018-11-29 14:07:05 +00:00
|
|
|
cachep = PGT_CACHE(PTE_T_ORDER);
|
2016-12-07 07:47:26 +00:00
|
|
|
num_hugepd = 1 << (pshift - pdshift);
|
|
|
|
} else {
|
|
|
|
cachep = PGT_CACHE(pdshift - pshift);
|
|
|
|
num_hugepd = 1;
|
|
|
|
}
|
2011-06-28 09:54:48 +00:00
|
|
|
|
2020-05-19 05:49:10 +00:00
|
|
|
if (!cachep) {
|
2019-05-28 05:36:25 +00:00
|
|
|
WARN_ONCE(1, "No page table cache created for hugetlb tables");
|
|
|
|
return -ENOMEM;
|
|
|
|
}
|
|
|
|
|
2020-05-19 05:49:10 +00:00
|
|
|
new = kmem_cache_alloc(cachep, pgtable_gfp_flags(mm, GFP_KERNEL));
|
2006-04-28 05:02:51 +00:00
|
|
|
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 19:24:31 +00:00
|
|
|
BUG_ON(pshift > HUGEPD_SHIFT_MASK);
|
|
|
|
BUG_ON((unsigned long)new & HUGEPD_SHIFT_MASK);
|
|
|
|
|
2019-05-28 05:36:25 +00:00
|
|
|
if (!new)
|
2006-04-28 05:02:51 +00:00
|
|
|
return -ENOMEM;
|
|
|
|
|
2016-03-24 06:07:57 +00:00
|
|
|
/*
|
|
|
|
* Make sure other cpus find the hugepd set only after a
|
|
|
|
* properly initialized page table is visible to them.
|
|
|
|
* For more details look for comment in __pte_alloc().
|
|
|
|
*/
|
|
|
|
smp_wmb();
|
|
|
|
|
2018-06-01 08:24:24 +00:00
|
|
|
spin_lock(ptl);
|
2011-06-28 09:54:48 +00:00
|
|
|
/*
|
|
|
|
* We have multiple higher-level entries that point to the same
|
|
|
|
* actual pte location. Fill in each as we go and backtrack on error.
|
|
|
|
* We need all of these so the DTLB pgtable walk code can find the
|
|
|
|
* right higher-level entry without knowing if it's a hugepage or not.
|
|
|
|
*/
|
|
|
|
for (i = 0; i < num_hugepd; i++, hpdp++) {
|
|
|
|
if (unlikely(!hugepd_none(*hpdp)))
|
|
|
|
break;
|
2019-04-26 05:59:45 +00:00
|
|
|
hugepd_populate(hpdp, new, pshift);
|
2011-06-28 09:54:48 +00:00
|
|
|
}
|
|
|
|
/* If we bailed from the for loop early, an error occurred, clean up */
|
|
|
|
if (i < num_hugepd) {
|
|
|
|
for (i = i - 1 ; i >= 0; i--, hpdp--)
|
2016-12-14 04:37:53 +00:00
|
|
|
*hpdp = __hugepd(0);
|
2020-05-19 05:49:10 +00:00
|
|
|
kmem_cache_free(cachep, new);
|
2018-08-13 13:19:52 +00:00
|
|
|
} else {
|
|
|
|
kmemleak_ignore(new);
|
2011-06-28 09:54:48 +00:00
|
|
|
}
|
2018-06-01 08:24:24 +00:00
|
|
|
spin_unlock(ptl);
|
2006-04-28 05:02:51 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2013-04-28 09:37:30 +00:00
|
|
|
/*
|
|
|
|
* At this point we do the placement change only for BOOK3S 64. This would
|
|
|
|
* possibly work on other subarchs.
|
|
|
|
*/
|
|
|
|
pte_t *huge_pte_alloc(struct mm_struct *mm, unsigned long addr, unsigned long sz)
|
|
|
|
{
|
|
|
|
pgd_t *pg;
|
2020-06-04 23:46:44 +00:00
|
|
|
p4d_t *p4;
|
2013-04-28 09:37:30 +00:00
|
|
|
pud_t *pu;
|
|
|
|
pmd_t *pm;
|
|
|
|
hugepd_t *hpdp = NULL;
|
|
|
|
unsigned pshift = __ffs(sz);
|
|
|
|
unsigned pdshift = PGDIR_SHIFT;
|
2018-06-01 08:24:24 +00:00
|
|
|
spinlock_t *ptl;
|
2013-04-28 09:37:30 +00:00
|
|
|
|
|
|
|
addr &= ~(sz-1);
|
|
|
|
pg = pgd_offset(mm, addr);
|
2020-06-04 23:46:44 +00:00
|
|
|
p4 = p4d_offset(pg, addr);
|
2013-04-28 09:37:30 +00:00
|
|
|
|
2016-12-07 07:47:26 +00:00
|
|
|
#ifdef CONFIG_PPC_BOOK3S_64
|
2013-04-28 09:37:30 +00:00
|
|
|
if (pshift == PGDIR_SHIFT)
|
|
|
|
/* 16GB huge page */
|
2020-06-04 23:46:44 +00:00
|
|
|
return (pte_t *) p4;
|
2018-06-01 08:24:24 +00:00
|
|
|
else if (pshift > PUD_SHIFT) {
|
2013-04-28 09:37:30 +00:00
|
|
|
/*
|
|
|
|
* We need to use hugepd table
|
|
|
|
*/
|
2018-06-01 08:24:24 +00:00
|
|
|
ptl = &mm->page_table_lock;
|
2020-06-04 23:46:44 +00:00
|
|
|
hpdp = (hugepd_t *)p4;
|
2018-06-01 08:24:24 +00:00
|
|
|
} else {
|
2013-04-28 09:37:30 +00:00
|
|
|
pdshift = PUD_SHIFT;
|
2020-06-04 23:46:44 +00:00
|
|
|
pu = pud_alloc(mm, p4, addr);
|
2019-05-28 05:36:24 +00:00
|
|
|
if (!pu)
|
|
|
|
return NULL;
|
2013-04-28 09:37:30 +00:00
|
|
|
if (pshift == PUD_SHIFT)
|
|
|
|
return (pte_t *)pu;
|
2018-06-01 08:24:24 +00:00
|
|
|
else if (pshift > PMD_SHIFT) {
|
|
|
|
ptl = pud_lockptr(mm, pu);
|
2013-04-28 09:37:30 +00:00
|
|
|
hpdp = (hugepd_t *)pu;
|
2018-06-01 08:24:24 +00:00
|
|
|
} else {
|
2013-04-28 09:37:30 +00:00
|
|
|
pdshift = PMD_SHIFT;
|
|
|
|
pm = pmd_alloc(mm, pu, addr);
|
2019-05-28 05:36:24 +00:00
|
|
|
if (!pm)
|
|
|
|
return NULL;
|
2013-04-28 09:37:30 +00:00
|
|
|
if (pshift == PMD_SHIFT)
|
|
|
|
/* 16MB hugepage */
|
|
|
|
return (pte_t *)pm;
|
2018-06-01 08:24:24 +00:00
|
|
|
else {
|
|
|
|
ptl = pmd_lockptr(mm, pm);
|
2013-04-28 09:37:30 +00:00
|
|
|
hpdp = (hugepd_t *)pm;
|
2018-06-01 08:24:24 +00:00
|
|
|
}
|
2013-04-28 09:37:30 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
#else
|
2018-07-17 04:24:30 +00:00
|
|
|
if (pshift >= PGDIR_SHIFT) {
|
2018-06-01 08:24:24 +00:00
|
|
|
ptl = &mm->page_table_lock;
|
2020-06-04 23:46:44 +00:00
|
|
|
hpdp = (hugepd_t *)p4;
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 19:24:31 +00:00
|
|
|
} else {
|
|
|
|
pdshift = PUD_SHIFT;
|
2020-06-04 23:46:44 +00:00
|
|
|
pu = pud_alloc(mm, p4, addr);
|
2019-05-28 05:36:24 +00:00
|
|
|
if (!pu)
|
|
|
|
return NULL;
|
2018-07-17 04:24:30 +00:00
|
|
|
if (pshift >= PUD_SHIFT) {
|
2018-06-01 08:24:24 +00:00
|
|
|
ptl = pud_lockptr(mm, pu);
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 19:24:31 +00:00
|
|
|
hpdp = (hugepd_t *)pu;
|
|
|
|
} else {
|
|
|
|
pdshift = PMD_SHIFT;
|
|
|
|
pm = pmd_alloc(mm, pu, addr);
|
2019-05-28 05:36:24 +00:00
|
|
|
if (!pm)
|
|
|
|
return NULL;
|
2018-06-01 08:24:24 +00:00
|
|
|
ptl = pmd_lockptr(mm, pm);
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 19:24:31 +00:00
|
|
|
hpdp = (hugepd_t *)pm;
|
|
|
|
}
|
|
|
|
}
|
2016-12-07 07:47:26 +00:00
|
|
|
#endif
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 19:24:31 +00:00
|
|
|
if (!hpdp)
|
|
|
|
return NULL;
|
|
|
|
|
powerpc/8xx: Manage 512k huge pages as standard pages.
At the time being, 512k huge pages are handled through hugepd page
tables. The PMD entry is flagged as a hugepd pointer and it
means that only 512k hugepages can be managed in that 4M block.
However, the hugepd table has the same size as a normal page
table, and 512k entries can therefore be nested with normal pages.
On the 8xx, TLB loading is performed by software and allthough the
page tables are organised to match the L1 and L2 level defined by
the HW, all TLB entries have both L1 and L2 independent entries.
It means that even if two TLB entries are associated with the same
PMD entry, they can be loaded with different values in L1 part.
The L1 entry contains the page size (PS field):
- 00 for 4k and 16 pages
- 01 for 512k pages
- 11 for 8M pages
By adding a flag for hugepages in the PTE (_PAGE_HUGE) and copying it
into the lower bit of PS, we can then manage 512k pages with normal
page tables:
- PMD entry has PS=11 for 8M pages
- PMD entry has PS=00 for other pages.
As a PMD entry covers 4M areas, a PMD will either point to a hugepd
table having a single entry to an 8M page, or the PMD will point to
a standard page table which will have either entries to 4k or 16k or
512k pages. For 512k pages, as the L1 entry will not know it is a
512k page before the PTE is read, there will be 128 entries in the
PTE as if it was 4k pages. But when loading the TLB, it will be
flagged as a 512k page.
Note that we can't use pmd_ptr() in asm/nohash/32/pgtable.h because
it is not defined yet.
In ITLB miss, we keep the possibility to opt it out as when kernel
text is pinned and no user hugepages are used, we can save several
instruction by not using r11.
In DTLB miss, that's just one instruction so it's not worth bothering
with it.
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Link: https://lore.kernel.org/r/002819e8e166bf81d24b24782d98de7c40905d8f.1589866984.git.christophe.leroy@csgroup.eu
2020-05-19 05:49:09 +00:00
|
|
|
if (IS_ENABLED(CONFIG_PPC_8xx) && sz == SZ_512K)
|
|
|
|
return pte_alloc_map(mm, (pmd_t *)hpdp, addr);
|
|
|
|
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 19:24:31 +00:00
|
|
|
BUG_ON(!hugepd_none(*hpdp) && !hugepd_ok(*hpdp));
|
|
|
|
|
2018-06-01 08:24:24 +00:00
|
|
|
if (hugepd_none(*hpdp) && __hugepte_alloc(mm, hpdp, addr,
|
|
|
|
pdshift, pshift, ptl))
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 19:24:31 +00:00
|
|
|
return NULL;
|
|
|
|
|
2014-11-05 16:27:41 +00:00
|
|
|
return hugepte_offset(*hpdp, addr, pdshift);
|
2008-01-03 22:59:50 +00:00
|
|
|
}
|
|
|
|
|
2017-07-28 05:01:26 +00:00
|
|
|
#ifdef CONFIG_PPC_BOOK3S_64
|
2011-06-28 09:54:48 +00:00
|
|
|
/*
|
2017-07-28 05:01:26 +00:00
|
|
|
* Tracks gpages after the device tree is scanned and before the
|
|
|
|
* huge_boot_pages list is ready on pseries.
|
2011-06-28 09:54:48 +00:00
|
|
|
*/
|
2017-07-28 05:01:26 +00:00
|
|
|
#define MAX_NUMBER_GPAGES 1024
|
|
|
|
__initdata static u64 gpage_freearray[MAX_NUMBER_GPAGES];
|
|
|
|
__initdata static unsigned nr_gpages;
|
2011-06-28 09:54:48 +00:00
|
|
|
|
|
|
|
/*
|
2017-07-28 05:01:26 +00:00
|
|
|
* Build list of addresses of gigantic pages. This function is used in early
|
2014-09-17 12:15:34 +00:00
|
|
|
* boot before the buddy allocator is setup.
|
2011-06-28 09:54:48 +00:00
|
|
|
*/
|
2017-07-28 05:01:26 +00:00
|
|
|
void __init pseries_add_gpage(u64 addr, u64 page_size, unsigned long number_of_pages)
|
2008-07-24 04:27:54 +00:00
|
|
|
{
|
|
|
|
if (!addr)
|
|
|
|
return;
|
|
|
|
while (number_of_pages > 0) {
|
|
|
|
gpage_freearray[nr_gpages] = addr;
|
|
|
|
nr_gpages++;
|
|
|
|
number_of_pages--;
|
|
|
|
addr += page_size;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2017-07-28 05:01:26 +00:00
|
|
|
int __init pseries_alloc_bootmem_huge_page(struct hstate *hstate)
|
2008-07-24 04:27:53 +00:00
|
|
|
{
|
|
|
|
struct huge_bootmem_page *m;
|
|
|
|
if (nr_gpages == 0)
|
|
|
|
return 0;
|
|
|
|
m = phys_to_virt(gpage_freearray[--nr_gpages]);
|
|
|
|
gpage_freearray[nr_gpages] = 0;
|
|
|
|
list_add(&m->list, &huge_boot_pages);
|
2008-07-24 04:27:56 +00:00
|
|
|
m->hstate = hstate;
|
2008-07-24 04:27:53 +00:00
|
|
|
return 1;
|
|
|
|
}
|
2011-06-28 09:54:48 +00:00
|
|
|
#endif
|
2008-07-24 04:27:53 +00:00
|
|
|
|
2017-07-28 05:01:26 +00:00
|
|
|
|
|
|
|
int __init alloc_bootmem_huge_page(struct hstate *h)
|
|
|
|
{
|
|
|
|
|
|
|
|
#ifdef CONFIG_PPC_BOOK3S_64
|
|
|
|
if (firmware_has_feature(FW_FEATURE_LPAR) && !radix_enabled())
|
|
|
|
return pseries_alloc_bootmem_huge_page(h);
|
|
|
|
#endif
|
|
|
|
return __alloc_bootmem_huge_page(h);
|
|
|
|
}
|
|
|
|
|
2019-04-26 05:59:49 +00:00
|
|
|
#ifndef CONFIG_PPC_BOOK3S_64
|
2011-06-28 09:54:48 +00:00
|
|
|
#define HUGEPD_FREELIST_SIZE \
|
|
|
|
((PAGE_SIZE - sizeof(struct hugepd_freelist)) / sizeof(pte_t))
|
|
|
|
|
|
|
|
struct hugepd_freelist {
|
|
|
|
struct rcu_head rcu;
|
|
|
|
unsigned int index;
|
2020-05-07 18:57:55 +00:00
|
|
|
void *ptes[];
|
2011-06-28 09:54:48 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
static DEFINE_PER_CPU(struct hugepd_freelist *, hugepd_freelist_cur);
|
|
|
|
|
|
|
|
static void hugepd_free_rcu_callback(struct rcu_head *head)
|
|
|
|
{
|
|
|
|
struct hugepd_freelist *batch =
|
|
|
|
container_of(head, struct hugepd_freelist, rcu);
|
|
|
|
unsigned int i;
|
|
|
|
|
|
|
|
for (i = 0; i < batch->index; i++)
|
2018-11-29 14:07:05 +00:00
|
|
|
kmem_cache_free(PGT_CACHE(PTE_T_ORDER), batch->ptes[i]);
|
2011-06-28 09:54:48 +00:00
|
|
|
|
|
|
|
free_page((unsigned long)batch);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void hugepd_free(struct mmu_gather *tlb, void *hugepte)
|
|
|
|
{
|
|
|
|
struct hugepd_freelist **batchp;
|
|
|
|
|
2016-03-08 09:03:56 +00:00
|
|
|
batchp = &get_cpu_var(hugepd_freelist_cur);
|
2011-06-28 09:54:48 +00:00
|
|
|
|
|
|
|
if (atomic_read(&tlb->mm->mm_users) < 2 ||
|
2017-07-24 04:28:01 +00:00
|
|
|
mm_is_thread_local(tlb->mm)) {
|
2018-11-29 14:07:05 +00:00
|
|
|
kmem_cache_free(PGT_CACHE(PTE_T_ORDER), hugepte);
|
2016-03-08 09:03:56 +00:00
|
|
|
put_cpu_var(hugepd_freelist_cur);
|
2011-06-28 09:54:48 +00:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (*batchp == NULL) {
|
|
|
|
*batchp = (struct hugepd_freelist *)__get_free_page(GFP_ATOMIC);
|
|
|
|
(*batchp)->index = 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
(*batchp)->ptes[(*batchp)->index++] = hugepte;
|
|
|
|
if ((*batchp)->index == HUGEPD_FREELIST_SIZE) {
|
2018-11-06 00:53:13 +00:00
|
|
|
call_rcu(&(*batchp)->rcu, hugepd_free_rcu_callback);
|
2011-06-28 09:54:48 +00:00
|
|
|
*batchp = NULL;
|
|
|
|
}
|
2014-01-20 08:39:34 +00:00
|
|
|
put_cpu_var(hugepd_freelist_cur);
|
2011-06-28 09:54:48 +00:00
|
|
|
}
|
2016-12-07 07:47:26 +00:00
|
|
|
#else
|
|
|
|
static inline void hugepd_free(struct mmu_gather *tlb, void *hugepte) {}
|
2011-06-28 09:54:48 +00:00
|
|
|
#endif
|
|
|
|
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 19:24:31 +00:00
|
|
|
static void free_hugepd_range(struct mmu_gather *tlb, hugepd_t *hpdp, int pdshift,
|
|
|
|
unsigned long start, unsigned long end,
|
|
|
|
unsigned long floor, unsigned long ceiling)
|
2006-04-28 05:02:51 +00:00
|
|
|
{
|
|
|
|
pte_t *hugepte = hugepd_page(*hpdp);
|
2011-06-28 09:54:48 +00:00
|
|
|
int i;
|
|
|
|
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 19:24:31 +00:00
|
|
|
unsigned long pdmask = ~((1UL << pdshift) - 1);
|
2011-06-28 09:54:48 +00:00
|
|
|
unsigned int num_hugepd = 1;
|
2016-12-07 07:47:26 +00:00
|
|
|
unsigned int shift = hugepd_shift(*hpdp);
|
2011-06-28 09:54:48 +00:00
|
|
|
|
2011-10-10 10:50:40 +00:00
|
|
|
/* Note: On fsl the hpdp may be the first of several */
|
2016-12-07 07:47:26 +00:00
|
|
|
if (shift > pdshift)
|
|
|
|
num_hugepd = 1 << (shift - pdshift);
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 19:24:31 +00:00
|
|
|
|
|
|
|
start &= pdmask;
|
|
|
|
if (start < floor)
|
|
|
|
return;
|
|
|
|
if (ceiling) {
|
|
|
|
ceiling &= pdmask;
|
|
|
|
if (! ceiling)
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
if (end - 1 > ceiling - 1)
|
|
|
|
return;
|
2006-04-28 05:02:51 +00:00
|
|
|
|
2011-06-28 09:54:48 +00:00
|
|
|
for (i = 0; i < num_hugepd; i++, hpdp++)
|
2016-12-14 04:37:53 +00:00
|
|
|
*hpdp = __hugepd(0);
|
2011-06-28 09:54:48 +00:00
|
|
|
|
2016-12-07 07:47:26 +00:00
|
|
|
if (shift >= pdshift)
|
|
|
|
hugepd_free(tlb, hugepte);
|
|
|
|
else
|
2018-06-14 10:31:52 +00:00
|
|
|
pgtable_free_tlb(tlb, hugepte,
|
|
|
|
get_hugepd_cache_index(pdshift - shift));
|
2006-04-28 05:02:51 +00:00
|
|
|
}
|
|
|
|
|
powerpc/8xx: Manage 512k huge pages as standard pages.
At the time being, 512k huge pages are handled through hugepd page
tables. The PMD entry is flagged as a hugepd pointer and it
means that only 512k hugepages can be managed in that 4M block.
However, the hugepd table has the same size as a normal page
table, and 512k entries can therefore be nested with normal pages.
On the 8xx, TLB loading is performed by software and allthough the
page tables are organised to match the L1 and L2 level defined by
the HW, all TLB entries have both L1 and L2 independent entries.
It means that even if two TLB entries are associated with the same
PMD entry, they can be loaded with different values in L1 part.
The L1 entry contains the page size (PS field):
- 00 for 4k and 16 pages
- 01 for 512k pages
- 11 for 8M pages
By adding a flag for hugepages in the PTE (_PAGE_HUGE) and copying it
into the lower bit of PS, we can then manage 512k pages with normal
page tables:
- PMD entry has PS=11 for 8M pages
- PMD entry has PS=00 for other pages.
As a PMD entry covers 4M areas, a PMD will either point to a hugepd
table having a single entry to an 8M page, or the PMD will point to
a standard page table which will have either entries to 4k or 16k or
512k pages. For 512k pages, as the L1 entry will not know it is a
512k page before the PTE is read, there will be 128 entries in the
PTE as if it was 4k pages. But when loading the TLB, it will be
flagged as a 512k page.
Note that we can't use pmd_ptr() in asm/nohash/32/pgtable.h because
it is not defined yet.
In ITLB miss, we keep the possibility to opt it out as when kernel
text is pinned and no user hugepages are used, we can save several
instruction by not using r11.
In DTLB miss, that's just one instruction so it's not worth bothering
with it.
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Link: https://lore.kernel.org/r/002819e8e166bf81d24b24782d98de7c40905d8f.1589866984.git.christophe.leroy@csgroup.eu
2020-05-19 05:49:09 +00:00
|
|
|
static void hugetlb_free_pte_range(struct mmu_gather *tlb, pmd_t *pmd, unsigned long addr)
|
|
|
|
{
|
|
|
|
pgtable_t token = pmd_pgtable(*pmd);
|
|
|
|
|
|
|
|
pmd_clear(pmd);
|
|
|
|
pte_free_tlb(tlb, token, addr);
|
|
|
|
mm_dec_nr_ptes(tlb->mm);
|
|
|
|
}
|
|
|
|
|
2006-04-28 05:02:51 +00:00
|
|
|
static void hugetlb_free_pmd_range(struct mmu_gather *tlb, pud_t *pud,
|
|
|
|
unsigned long addr, unsigned long end,
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 19:24:31 +00:00
|
|
|
unsigned long floor, unsigned long ceiling)
|
2006-04-28 05:02:51 +00:00
|
|
|
{
|
|
|
|
pmd_t *pmd;
|
|
|
|
unsigned long next;
|
|
|
|
unsigned long start;
|
|
|
|
|
|
|
|
start = addr;
|
|
|
|
do {
|
2016-12-07 07:47:26 +00:00
|
|
|
unsigned long more;
|
|
|
|
|
2011-10-10 10:50:39 +00:00
|
|
|
pmd = pmd_offset(pud, addr);
|
2006-04-28 05:02:51 +00:00
|
|
|
next = pmd_addr_end(addr, end);
|
2014-11-05 16:27:41 +00:00
|
|
|
if (!is_hugepd(__hugepd(pmd_val(*pmd)))) {
|
powerpc/8xx: Manage 512k huge pages as standard pages.
At the time being, 512k huge pages are handled through hugepd page
tables. The PMD entry is flagged as a hugepd pointer and it
means that only 512k hugepages can be managed in that 4M block.
However, the hugepd table has the same size as a normal page
table, and 512k entries can therefore be nested with normal pages.
On the 8xx, TLB loading is performed by software and allthough the
page tables are organised to match the L1 and L2 level defined by
the HW, all TLB entries have both L1 and L2 independent entries.
It means that even if two TLB entries are associated with the same
PMD entry, they can be loaded with different values in L1 part.
The L1 entry contains the page size (PS field):
- 00 for 4k and 16 pages
- 01 for 512k pages
- 11 for 8M pages
By adding a flag for hugepages in the PTE (_PAGE_HUGE) and copying it
into the lower bit of PS, we can then manage 512k pages with normal
page tables:
- PMD entry has PS=11 for 8M pages
- PMD entry has PS=00 for other pages.
As a PMD entry covers 4M areas, a PMD will either point to a hugepd
table having a single entry to an 8M page, or the PMD will point to
a standard page table which will have either entries to 4k or 16k or
512k pages. For 512k pages, as the L1 entry will not know it is a
512k page before the PTE is read, there will be 128 entries in the
PTE as if it was 4k pages. But when loading the TLB, it will be
flagged as a 512k page.
Note that we can't use pmd_ptr() in asm/nohash/32/pgtable.h because
it is not defined yet.
In ITLB miss, we keep the possibility to opt it out as when kernel
text is pinned and no user hugepages are used, we can save several
instruction by not using r11.
In DTLB miss, that's just one instruction so it's not worth bothering
with it.
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Link: https://lore.kernel.org/r/002819e8e166bf81d24b24782d98de7c40905d8f.1589866984.git.christophe.leroy@csgroup.eu
2020-05-19 05:49:09 +00:00
|
|
|
if (pmd_none_or_clear_bad(pmd))
|
|
|
|
continue;
|
|
|
|
|
2013-06-19 06:34:26 +00:00
|
|
|
/*
|
|
|
|
* if it is not hugepd pointer, we should already find
|
|
|
|
* it cleared.
|
|
|
|
*/
|
powerpc/8xx: Manage 512k huge pages as standard pages.
At the time being, 512k huge pages are handled through hugepd page
tables. The PMD entry is flagged as a hugepd pointer and it
means that only 512k hugepages can be managed in that 4M block.
However, the hugepd table has the same size as a normal page
table, and 512k entries can therefore be nested with normal pages.
On the 8xx, TLB loading is performed by software and allthough the
page tables are organised to match the L1 and L2 level defined by
the HW, all TLB entries have both L1 and L2 independent entries.
It means that even if two TLB entries are associated with the same
PMD entry, they can be loaded with different values in L1 part.
The L1 entry contains the page size (PS field):
- 00 for 4k and 16 pages
- 01 for 512k pages
- 11 for 8M pages
By adding a flag for hugepages in the PTE (_PAGE_HUGE) and copying it
into the lower bit of PS, we can then manage 512k pages with normal
page tables:
- PMD entry has PS=11 for 8M pages
- PMD entry has PS=00 for other pages.
As a PMD entry covers 4M areas, a PMD will either point to a hugepd
table having a single entry to an 8M page, or the PMD will point to
a standard page table which will have either entries to 4k or 16k or
512k pages. For 512k pages, as the L1 entry will not know it is a
512k page before the PTE is read, there will be 128 entries in the
PTE as if it was 4k pages. But when loading the TLB, it will be
flagged as a 512k page.
Note that we can't use pmd_ptr() in asm/nohash/32/pgtable.h because
it is not defined yet.
In ITLB miss, we keep the possibility to opt it out as when kernel
text is pinned and no user hugepages are used, we can save several
instruction by not using r11.
In DTLB miss, that's just one instruction so it's not worth bothering
with it.
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Link: https://lore.kernel.org/r/002819e8e166bf81d24b24782d98de7c40905d8f.1589866984.git.christophe.leroy@csgroup.eu
2020-05-19 05:49:09 +00:00
|
|
|
WARN_ON(!IS_ENABLED(CONFIG_PPC_8xx));
|
|
|
|
|
|
|
|
hugetlb_free_pte_range(tlb, pmd, addr);
|
|
|
|
|
2006-04-28 05:02:51 +00:00
|
|
|
continue;
|
2013-06-19 06:34:26 +00:00
|
|
|
}
|
2011-10-10 10:50:39 +00:00
|
|
|
/*
|
|
|
|
* Increment next by the size of the huge mapping since
|
|
|
|
* there may be more than one entry at this level for a
|
|
|
|
* single hugepage, but all of them point to
|
|
|
|
* the same kmem cache that holds the hugepte.
|
|
|
|
*/
|
2016-12-07 07:47:26 +00:00
|
|
|
more = addr + (1 << hugepd_shift(*(hugepd_t *)pmd));
|
|
|
|
if (more > next)
|
|
|
|
next = more;
|
|
|
|
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 19:24:31 +00:00
|
|
|
free_hugepd_range(tlb, (hugepd_t *)pmd, PMD_SHIFT,
|
|
|
|
addr, next, floor, ceiling);
|
2011-10-10 10:50:39 +00:00
|
|
|
} while (addr = next, addr != end);
|
2006-04-28 05:02:51 +00:00
|
|
|
|
|
|
|
start &= PUD_MASK;
|
|
|
|
if (start < floor)
|
|
|
|
return;
|
|
|
|
if (ceiling) {
|
|
|
|
ceiling &= PUD_MASK;
|
|
|
|
if (!ceiling)
|
|
|
|
return;
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2006-04-28 05:02:51 +00:00
|
|
|
if (end - 1 > ceiling - 1)
|
|
|
|
return;
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2006-04-28 05:02:51 +00:00
|
|
|
pmd = pmd_offset(pud, start);
|
|
|
|
pud_clear(pud);
|
mm: Pass virtual address to [__]p{te,ud,md}_free_tlb()
mm: Pass virtual address to [__]p{te,ud,md}_free_tlb()
Upcoming paches to support the new 64-bit "BookE" powerpc architecture
will need to have the virtual address corresponding to PTE page when
freeing it, due to the way the HW table walker works.
Basically, the TLB can be loaded with "large" pages that cover the whole
virtual space (well, sort-of, half of it actually) represented by a PTE
page, and which contain an "indirect" bit indicating that this TLB entry
RPN points to an array of PTEs from which the TLB can then create direct
entries. Thus, in order to invalidate those when PTE pages are deleted,
we need the virtual address to pass to tlbilx or tlbivax instructions.
The old trick of sticking it somewhere in the PTE page struct page sucks
too much, the address is almost readily available in all call sites and
almost everybody implemets these as macros, so we may as well add the
argument everywhere. I added it to the pmd and pud variants for consistency.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Acked-by: David Howells <dhowells@redhat.com> [MN10300 & FRV]
Acked-by: Nick Piggin <npiggin@suse.de>
Acked-by: Martin Schwidefsky <schwidefsky@de.ibm.com> [s390]
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2009-07-22 05:44:28 +00:00
|
|
|
pmd_free_tlb(tlb, pmd, start);
|
2015-04-11 00:37:34 +00:00
|
|
|
mm_dec_nr_pmds(tlb->mm);
|
2006-04-28 05:02:51 +00:00
|
|
|
}
|
|
|
|
|
2020-06-04 23:46:44 +00:00
|
|
|
static void hugetlb_free_pud_range(struct mmu_gather *tlb, p4d_t *p4d,
|
2006-04-28 05:02:51 +00:00
|
|
|
unsigned long addr, unsigned long end,
|
|
|
|
unsigned long floor, unsigned long ceiling)
|
|
|
|
{
|
|
|
|
pud_t *pud;
|
|
|
|
unsigned long next;
|
|
|
|
unsigned long start;
|
|
|
|
|
|
|
|
start = addr;
|
|
|
|
do {
|
2020-06-04 23:46:44 +00:00
|
|
|
pud = pud_offset(p4d, addr);
|
2006-04-28 05:02:51 +00:00
|
|
|
next = pud_addr_end(addr, end);
|
2014-11-05 16:27:41 +00:00
|
|
|
if (!is_hugepd(__hugepd(pud_val(*pud)))) {
|
2008-01-03 22:59:50 +00:00
|
|
|
if (pud_none_or_clear_bad(pud))
|
|
|
|
continue;
|
2008-07-24 04:27:56 +00:00
|
|
|
hugetlb_free_pmd_range(tlb, pud, addr, next, floor,
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 19:24:31 +00:00
|
|
|
ceiling);
|
2008-01-03 22:59:50 +00:00
|
|
|
} else {
|
2016-12-07 07:47:26 +00:00
|
|
|
unsigned long more;
|
2011-10-10 10:50:39 +00:00
|
|
|
/*
|
|
|
|
* Increment next by the size of the huge mapping since
|
|
|
|
* there may be more than one entry at this level for a
|
|
|
|
* single hugepage, but all of them point to
|
|
|
|
* the same kmem cache that holds the hugepte.
|
|
|
|
*/
|
2016-12-07 07:47:26 +00:00
|
|
|
more = addr + (1 << hugepd_shift(*(hugepd_t *)pud));
|
|
|
|
if (more > next)
|
|
|
|
next = more;
|
|
|
|
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 19:24:31 +00:00
|
|
|
free_hugepd_range(tlb, (hugepd_t *)pud, PUD_SHIFT,
|
|
|
|
addr, next, floor, ceiling);
|
2008-01-03 22:59:50 +00:00
|
|
|
}
|
2011-10-10 10:50:39 +00:00
|
|
|
} while (addr = next, addr != end);
|
2006-04-28 05:02:51 +00:00
|
|
|
|
|
|
|
start &= PGDIR_MASK;
|
|
|
|
if (start < floor)
|
|
|
|
return;
|
|
|
|
if (ceiling) {
|
|
|
|
ceiling &= PGDIR_MASK;
|
|
|
|
if (!ceiling)
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
if (end - 1 > ceiling - 1)
|
|
|
|
return;
|
|
|
|
|
2020-06-04 23:46:44 +00:00
|
|
|
pud = pud_offset(p4d, start);
|
|
|
|
p4d_clear(p4d);
|
mm: Pass virtual address to [__]p{te,ud,md}_free_tlb()
mm: Pass virtual address to [__]p{te,ud,md}_free_tlb()
Upcoming paches to support the new 64-bit "BookE" powerpc architecture
will need to have the virtual address corresponding to PTE page when
freeing it, due to the way the HW table walker works.
Basically, the TLB can be loaded with "large" pages that cover the whole
virtual space (well, sort-of, half of it actually) represented by a PTE
page, and which contain an "indirect" bit indicating that this TLB entry
RPN points to an array of PTEs from which the TLB can then create direct
entries. Thus, in order to invalidate those when PTE pages are deleted,
we need the virtual address to pass to tlbilx or tlbivax instructions.
The old trick of sticking it somewhere in the PTE page struct page sucks
too much, the address is almost readily available in all call sites and
almost everybody implemets these as macros, so we may as well add the
argument everywhere. I added it to the pmd and pud variants for consistency.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Acked-by: David Howells <dhowells@redhat.com> [MN10300 & FRV]
Acked-by: Nick Piggin <npiggin@suse.de>
Acked-by: Martin Schwidefsky <schwidefsky@de.ibm.com> [s390]
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2009-07-22 05:44:28 +00:00
|
|
|
pud_free_tlb(tlb, pud, start);
|
2017-11-16 01:35:33 +00:00
|
|
|
mm_dec_nr_puds(tlb->mm);
|
2006-04-28 05:02:51 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This function frees user-level page tables of a process.
|
|
|
|
*/
|
2008-07-24 04:27:10 +00:00
|
|
|
void hugetlb_free_pgd_range(struct mmu_gather *tlb,
|
2006-04-28 05:02:51 +00:00
|
|
|
unsigned long addr, unsigned long end,
|
|
|
|
unsigned long floor, unsigned long ceiling)
|
|
|
|
{
|
|
|
|
pgd_t *pgd;
|
2020-06-04 23:46:44 +00:00
|
|
|
p4d_t *p4d;
|
2006-04-28 05:02:51 +00:00
|
|
|
unsigned long next;
|
|
|
|
|
|
|
|
/*
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 19:24:31 +00:00
|
|
|
* Because there are a number of different possible pagetable
|
|
|
|
* layouts for hugepage ranges, we limit knowledge of how
|
|
|
|
* things should be laid out to the allocation path
|
|
|
|
* (huge_pte_alloc(), above). Everything else works out the
|
|
|
|
* structure as it goes from information in the hugepd
|
|
|
|
* pointers. That means that we can't here use the
|
|
|
|
* optimization used in the normal page free_pgd_range(), of
|
|
|
|
* checking whether we're actually covering a large enough
|
|
|
|
* range to have to do anything at the top level of the walk
|
|
|
|
* instead of at the bottom.
|
2006-04-28 05:02:51 +00:00
|
|
|
*
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 19:24:31 +00:00
|
|
|
* To make sense of this, you should probably go read the big
|
|
|
|
* block comment at the top of the normal free_pgd_range(),
|
|
|
|
* too.
|
2006-04-28 05:02:51 +00:00
|
|
|
*/
|
|
|
|
|
|
|
|
do {
|
|
|
|
next = pgd_addr_end(addr, end);
|
2011-06-28 09:54:48 +00:00
|
|
|
pgd = pgd_offset(tlb->mm, addr);
|
2020-06-04 23:46:44 +00:00
|
|
|
p4d = p4d_offset(pgd, addr);
|
2014-11-05 16:27:41 +00:00
|
|
|
if (!is_hugepd(__hugepd(pgd_val(*pgd)))) {
|
2020-06-04 23:46:44 +00:00
|
|
|
if (p4d_none_or_clear_bad(p4d))
|
2008-09-05 01:49:54 +00:00
|
|
|
continue;
|
2020-06-04 23:46:44 +00:00
|
|
|
hugetlb_free_pud_range(tlb, p4d, addr, next, floor, ceiling);
|
2008-09-05 01:49:54 +00:00
|
|
|
} else {
|
2016-12-07 07:47:26 +00:00
|
|
|
unsigned long more;
|
2011-06-28 09:54:48 +00:00
|
|
|
/*
|
|
|
|
* Increment next by the size of the huge mapping since
|
2011-10-10 10:50:40 +00:00
|
|
|
* there may be more than one entry at the pgd level
|
|
|
|
* for a single hugepage, but all of them point to the
|
|
|
|
* same kmem cache that holds the hugepte.
|
2011-06-28 09:54:48 +00:00
|
|
|
*/
|
2016-12-07 07:47:26 +00:00
|
|
|
more = addr + (1 << hugepd_shift(*(hugepd_t *)pgd));
|
|
|
|
if (more > next)
|
|
|
|
next = more;
|
|
|
|
|
2020-06-04 23:46:44 +00:00
|
|
|
free_hugepd_range(tlb, (hugepd_t *)p4d, PGDIR_SHIFT,
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 19:24:31 +00:00
|
|
|
addr, next, floor, ceiling);
|
2008-09-05 01:49:54 +00:00
|
|
|
}
|
2011-06-28 09:54:48 +00:00
|
|
|
} while (addr = next, addr != end);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
|
|
|
|
2017-07-06 22:38:59 +00:00
|
|
|
struct page *follow_huge_pd(struct vm_area_struct *vma,
|
|
|
|
unsigned long address, hugepd_t hpd,
|
|
|
|
int flags, int pdshift)
|
|
|
|
{
|
|
|
|
pte_t *ptep;
|
|
|
|
spinlock_t *ptl;
|
|
|
|
struct page *page = NULL;
|
|
|
|
unsigned long mask;
|
|
|
|
int shift = hugepd_shift(hpd);
|
|
|
|
struct mm_struct *mm = vma->vm_mm;
|
|
|
|
|
|
|
|
retry:
|
2018-06-01 08:24:24 +00:00
|
|
|
/*
|
|
|
|
* hugepage directory entries are protected by mm->page_table_lock
|
|
|
|
* Use this instead of huge_pte_lockptr
|
|
|
|
*/
|
2017-07-06 22:38:59 +00:00
|
|
|
ptl = &mm->page_table_lock;
|
|
|
|
spin_lock(ptl);
|
|
|
|
|
|
|
|
ptep = hugepte_offset(hpd, address, pdshift);
|
|
|
|
if (pte_present(*ptep)) {
|
|
|
|
mask = (1UL << shift) - 1;
|
|
|
|
page = pte_page(*ptep);
|
|
|
|
page += ((address & mask) >> PAGE_SHIFT);
|
|
|
|
if (flags & FOLL_GET)
|
|
|
|
get_page(page);
|
|
|
|
} else {
|
|
|
|
if (is_hugetlb_entry_migration(*ptep)) {
|
|
|
|
spin_unlock(ptl);
|
|
|
|
__migration_entry_wait(mm, ptep, ptl);
|
|
|
|
goto retry;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
spin_unlock(ptl);
|
|
|
|
return page;
|
|
|
|
}
|
|
|
|
|
2011-10-10 10:50:36 +00:00
|
|
|
#ifdef CONFIG_PPC_MM_SLICES
|
2005-04-16 22:20:36 +00:00
|
|
|
unsigned long hugetlb_get_unmapped_area(struct file *file, unsigned long addr,
|
|
|
|
unsigned long len, unsigned long pgoff,
|
|
|
|
unsigned long flags)
|
|
|
|
{
|
2008-07-24 04:27:56 +00:00
|
|
|
struct hstate *hstate = hstate_file(file);
|
|
|
|
int mmu_psize = shift_to_mmu_psize(huge_page_shift(hstate));
|
2008-12-04 04:07:54 +00:00
|
|
|
|
powerpc/mm/slice: Fix hugepage allocation at hint address on 8xx
On the 8xx, the page size is set in the PMD entry and applies to
all pages of the page table pointed by the said PMD entry.
When an app has some regular pages allocated (e.g. see below) and tries
to mmap() a huge page at a hint address covered by the same PMD entry,
the kernel accepts the hint allthough the 8xx cannot handle different
page sizes in the same PMD entry.
10000000-10001000 r-xp 00000000 00:0f 2597 /root/malloc
10010000-10011000 rwxp 00000000 00:0f 2597 /root/malloc
mmap(0x10080000, 524288, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS|0x40000, -1, 0) = 0x10080000
This results the app remaining forever in do_page_fault()/hugetlb_fault()
and when interrupting that app, we get the following warning:
[162980.035629] WARNING: CPU: 0 PID: 2777 at arch/powerpc/mm/hugetlbpage.c:354 hugetlb_free_pgd_range+0xc8/0x1e4
[162980.035699] CPU: 0 PID: 2777 Comm: malloc Tainted: G W 4.14.6 #85
[162980.035744] task: c67e2c00 task.stack: c668e000
[162980.035783] NIP: c000fe18 LR: c00e1eec CTR: c00f90c0
[162980.035830] REGS: c668fc20 TRAP: 0700 Tainted: G W (4.14.6)
[162980.035854] MSR: 00029032 <EE,ME,IR,DR,RI> CR: 24044224 XER: 20000000
[162980.036003]
[162980.036003] GPR00: c00e1eec c668fcd0 c67e2c00 00000010 c6869410 10080000 00000000 77fb4000
[162980.036003] GPR08: ffff0001 0683c001 00000000 ffffff80 44028228 10018a34 00004008 418004fc
[162980.036003] GPR16: c668e000 00040100 c668e000 c06c0000 c668fe78 c668e000 c6835ba0 c668fd48
[162980.036003] GPR24: 00000000 73ffffff 74000000 00000001 77fb4000 100fffff 10100000 10100000
[162980.036743] NIP [c000fe18] hugetlb_free_pgd_range+0xc8/0x1e4
[162980.036839] LR [c00e1eec] free_pgtables+0x12c/0x150
[162980.036861] Call Trace:
[162980.036939] [c668fcd0] [c00f0774] unlink_anon_vmas+0x1c4/0x214 (unreliable)
[162980.037040] [c668fd10] [c00e1eec] free_pgtables+0x12c/0x150
[162980.037118] [c668fd40] [c00eabac] exit_mmap+0xe8/0x1b4
[162980.037210] [c668fda0] [c0019710] mmput.part.9+0x20/0xd8
[162980.037301] [c668fdb0] [c001ecb0] do_exit+0x1f0/0x93c
[162980.037386] [c668fe00] [c001f478] do_group_exit+0x40/0xcc
[162980.037479] [c668fe10] [c002a76c] get_signal+0x47c/0x614
[162980.037570] [c668fe70] [c0007840] do_signal+0x54/0x244
[162980.037654] [c668ff30] [c0007ae8] do_notify_resume+0x34/0x88
[162980.037744] [c668ff40] [c000dae8] do_user_signal+0x74/0xc4
[162980.037781] Instruction dump:
[162980.037821] 7fdff378 81370000 54a3463a 80890020 7d24182e 7c841a14 712a0004 4082ff94
[162980.038014] 2f890000 419e0010 712a0ff0 408200e0 <0fe00000> 54a9000a 7f984840 419d0094
[162980.038216] ---[ end trace c0ceeca8e7a5800a ]---
[162980.038754] BUG: non-zero nr_ptes on freeing mm: 1
[162985.363322] BUG: non-zero nr_ptes on freeing mm: -1
In order to fix this, this patch uses the address space "slices"
implemented for BOOK3S/64 and enhanced to support PPC32 by the
preceding patch.
This patch modifies the context.id on the 8xx to be in the range
[1:16] instead of [0:15] in order to identify context.id == 0 as
not initialised contexts as done on BOOK3S
This patch activates CONFIG_PPC_MM_SLICES when CONFIG_HUGETLB_PAGE is
selected for the 8xx
Alltough we could in theory have as many slices as PMD entries, the
current slices implementation limits the number of low slices to 16.
This limitation is not preventing us to fix the initial issue allthough
it is suboptimal. It will be cured in a subsequent patch.
Fixes: 4b91428699477 ("powerpc/8xx: Implement support of hugepages")
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Reviewed-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2018-02-22 14:27:26 +00:00
|
|
|
#ifdef CONFIG_PPC_RADIX_MMU
|
2016-04-29 13:26:25 +00:00
|
|
|
if (radix_enabled())
|
|
|
|
return radix__hugetlb_get_unmapped_area(file, addr, len,
|
|
|
|
pgoff, flags);
|
powerpc/mm/slice: Fix hugepage allocation at hint address on 8xx
On the 8xx, the page size is set in the PMD entry and applies to
all pages of the page table pointed by the said PMD entry.
When an app has some regular pages allocated (e.g. see below) and tries
to mmap() a huge page at a hint address covered by the same PMD entry,
the kernel accepts the hint allthough the 8xx cannot handle different
page sizes in the same PMD entry.
10000000-10001000 r-xp 00000000 00:0f 2597 /root/malloc
10010000-10011000 rwxp 00000000 00:0f 2597 /root/malloc
mmap(0x10080000, 524288, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS|0x40000, -1, 0) = 0x10080000
This results the app remaining forever in do_page_fault()/hugetlb_fault()
and when interrupting that app, we get the following warning:
[162980.035629] WARNING: CPU: 0 PID: 2777 at arch/powerpc/mm/hugetlbpage.c:354 hugetlb_free_pgd_range+0xc8/0x1e4
[162980.035699] CPU: 0 PID: 2777 Comm: malloc Tainted: G W 4.14.6 #85
[162980.035744] task: c67e2c00 task.stack: c668e000
[162980.035783] NIP: c000fe18 LR: c00e1eec CTR: c00f90c0
[162980.035830] REGS: c668fc20 TRAP: 0700 Tainted: G W (4.14.6)
[162980.035854] MSR: 00029032 <EE,ME,IR,DR,RI> CR: 24044224 XER: 20000000
[162980.036003]
[162980.036003] GPR00: c00e1eec c668fcd0 c67e2c00 00000010 c6869410 10080000 00000000 77fb4000
[162980.036003] GPR08: ffff0001 0683c001 00000000 ffffff80 44028228 10018a34 00004008 418004fc
[162980.036003] GPR16: c668e000 00040100 c668e000 c06c0000 c668fe78 c668e000 c6835ba0 c668fd48
[162980.036003] GPR24: 00000000 73ffffff 74000000 00000001 77fb4000 100fffff 10100000 10100000
[162980.036743] NIP [c000fe18] hugetlb_free_pgd_range+0xc8/0x1e4
[162980.036839] LR [c00e1eec] free_pgtables+0x12c/0x150
[162980.036861] Call Trace:
[162980.036939] [c668fcd0] [c00f0774] unlink_anon_vmas+0x1c4/0x214 (unreliable)
[162980.037040] [c668fd10] [c00e1eec] free_pgtables+0x12c/0x150
[162980.037118] [c668fd40] [c00eabac] exit_mmap+0xe8/0x1b4
[162980.037210] [c668fda0] [c0019710] mmput.part.9+0x20/0xd8
[162980.037301] [c668fdb0] [c001ecb0] do_exit+0x1f0/0x93c
[162980.037386] [c668fe00] [c001f478] do_group_exit+0x40/0xcc
[162980.037479] [c668fe10] [c002a76c] get_signal+0x47c/0x614
[162980.037570] [c668fe70] [c0007840] do_signal+0x54/0x244
[162980.037654] [c668ff30] [c0007ae8] do_notify_resume+0x34/0x88
[162980.037744] [c668ff40] [c000dae8] do_user_signal+0x74/0xc4
[162980.037781] Instruction dump:
[162980.037821] 7fdff378 81370000 54a3463a 80890020 7d24182e 7c841a14 712a0004 4082ff94
[162980.038014] 2f890000 419e0010 712a0ff0 408200e0 <0fe00000> 54a9000a 7f984840 419d0094
[162980.038216] ---[ end trace c0ceeca8e7a5800a ]---
[162980.038754] BUG: non-zero nr_ptes on freeing mm: 1
[162985.363322] BUG: non-zero nr_ptes on freeing mm: -1
In order to fix this, this patch uses the address space "slices"
implemented for BOOK3S/64 and enhanced to support PPC32 by the
preceding patch.
This patch modifies the context.id on the 8xx to be in the range
[1:16] instead of [0:15] in order to identify context.id == 0 as
not initialised contexts as done on BOOK3S
This patch activates CONFIG_PPC_MM_SLICES when CONFIG_HUGETLB_PAGE is
selected for the 8xx
Alltough we could in theory have as many slices as PMD entries, the
current slices implementation limits the number of low slices to 16.
This limitation is not preventing us to fix the initial issue allthough
it is suboptimal. It will be cured in a subsequent patch.
Fixes: 4b91428699477 ("powerpc/8xx: Implement support of hugepages")
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Reviewed-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2018-02-22 14:27:26 +00:00
|
|
|
#endif
|
2013-04-29 18:53:52 +00:00
|
|
|
return slice_get_unmapped_area(addr, len, flags, mmu_psize, 1);
|
2005-04-16 22:20:36 +00:00
|
|
|
}
|
2011-10-10 10:50:36 +00:00
|
|
|
#endif
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2009-01-06 22:38:54 +00:00
|
|
|
unsigned long vma_mmu_pagesize(struct vm_area_struct *vma)
|
|
|
|
{
|
2016-04-29 13:26:24 +00:00
|
|
|
/* With radix we don't use slice, so derive it from vma*/
|
2019-04-25 14:29:35 +00:00
|
|
|
if (IS_ENABLED(CONFIG_PPC_MM_SLICES) && !radix_enabled()) {
|
2018-03-07 01:37:17 +00:00
|
|
|
unsigned int psize = get_slice_psize(vma->vm_mm, vma->vm_start);
|
|
|
|
|
2016-04-29 13:26:24 +00:00
|
|
|
return 1UL << mmu_psize_to_shift(psize);
|
2018-03-07 01:37:17 +00:00
|
|
|
}
|
2018-04-05 23:24:21 +00:00
|
|
|
return vma_kernel_pagesize(vma);
|
2011-06-28 09:54:48 +00:00
|
|
|
}
|
|
|
|
|
2020-06-03 23:00:34 +00:00
|
|
|
bool __init arch_hugetlb_valid_size(unsigned long size)
|
2008-01-03 22:59:50 +00:00
|
|
|
{
|
2009-10-26 19:24:31 +00:00
|
|
|
int shift = __ffs(size);
|
|
|
|
int mmu_psize;
|
powerpc/mm: Allow more flexible layouts for hugepage pagetables
Currently each available hugepage size uses a slightly different
pagetable layout: that is, the bottem level table of pointers to
hugepages is a different size, and may branch off from the normal page
tables at a different level. Every hugepage aware path that needs to
walk the pagetables must therefore look up the hugepage size from the
slice info first, and work out the correct way to walk the pagetables
accordingly. Future hardware is likely to add more possible hugepage
sizes, more layout options and more mess.
This patch, therefore reworks the handling of hugepage pagetables to
reduce this complexity. In the new scheme, instead of having to
consult the slice mask, pagetable walking code can check a flag in the
PGD/PUD/PMD entries to see where to branch off to hugepage pagetables,
and the entry also contains the information (eseentially hugepage
shift) necessary to then interpret that table without recourse to the
slice mask. This scheme can be extended neatly to handle multiple
levels of self-describing "special" hugepage pagetables, although for
now we assume only one level exists.
This approach means that only the pagetable allocation path needs to
know how the pagetables should be set out. All other (hugepage)
pagetable walking paths can just interpret the structure as they go.
There already was a flag bit in PGD/PUD/PMD entries for hugepage
directory pointers, but it was only used for debug. We alter that
flag bit to instead be a 0 in the MSB to indicate a hugepage pagetable
pointer (normally it would be 1 since the pointer lies in the linear
mapping). This means that asm pagetable walking can test for (and
punt on) hugepage pointers with the same test that checks for
unpopulated page directory entries (beq becomes bge), since hugepage
pointers will always be positive, and normal pointers always negative.
While we're at it, we get rid of the confusing (and grep defeating)
#defining of hugepte_shift to be the same thing as mmu_huge_psizes.
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-10-26 19:24:31 +00:00
|
|
|
|
2008-01-03 22:59:50 +00:00
|
|
|
/* Check that it is a page size supported by the hardware and
|
2009-10-26 19:24:31 +00:00
|
|
|
* that it fits within pagetable and slice limits. */
|
2019-04-26 05:59:46 +00:00
|
|
|
if (size <= PAGE_SIZE || !is_power_of_2(size))
|
2020-06-03 23:00:34 +00:00
|
|
|
return false;
|
2008-07-24 04:27:55 +00:00
|
|
|
|
powerpc/mm: Fix crashes with hugepages & 4K pages
The recent commit to cleanup ifdefs in the hugepage initialisation led
to crashes when using 4K pages as reported by Sachin:
BUG: Kernel NULL pointer dereference at 0x0000001c
Faulting instruction address: 0xc000000001d1e58c
Oops: Kernel access of bad area, sig: 11 [#1]
LE PAGE_SIZE=4K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries
...
CPU: 3 PID: 4635 Comm: futex_wake04 Tainted: G W O 5.1.0-next-20190507-autotest #1
NIP: c000000001d1e58c LR: c000000001d1e54c CTR: 0000000000000000
REGS: c000000004937890 TRAP: 0300
MSR: 8000000000009033 <SF,EE,ME,IR,DR,RI,LE> CR: 22424822 XER: 00000000
CFAR: c00000000183e9e0 DAR: 000000000000001c DSISR: 40000000 IRQMASK: 0
...
NIP kmem_cache_alloc+0xbc/0x5a0
LR kmem_cache_alloc+0x7c/0x5a0
Call Trace:
huge_pte_alloc+0x580/0x950
hugetlb_fault+0x9a0/0x1250
handle_mm_fault+0x490/0x4a0
__do_page_fault+0x77c/0x1f00
do_page_fault+0x28/0x50
handle_page_fault+0x18/0x38
This is caused by us trying to allocate from a NULL kmem cache in
__hugepte_alloc(). The kmem cache is NULL because it was never
allocated in hugetlbpage_init(), because add_huge_page_size() returned
an error.
The reason add_huge_page_size() returned an error is a simple typo, we
are calling check_and_get_huge_psize(size) when we should be passing
shift instead.
The fact that we're able to trigger this path when the kmem caches are
NULL is a separate bug, ie. we should not advertise any hugepage sizes
if we haven't setup the required caches for them.
This was only seen with 4K pages, with 64K pages we don't need to
allocate any extra kmem caches because the 16M hugepage just occupies
a single entry at the PMD level.
Fixes: 723f268f19da ("powerpc/mm: cleanup ifdef mess in add_huge_page_size()")
Reported-by: Sachin Sant <sachinp@linux.ibm.com>
Tested-by: Sachin Sant <sachinp@linux.ibm.com>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Reviewed-by: Christophe Leroy <christophe.leroy@c-s.fr>
Reviewed-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
2019-05-14 13:00:58 +00:00
|
|
|
mmu_psize = check_and_get_huge_psize(shift);
|
2019-04-26 05:59:46 +00:00
|
|
|
if (mmu_psize < 0)
|
2020-06-03 23:00:34 +00:00
|
|
|
return false;
|
2009-10-26 19:24:31 +00:00
|
|
|
|
|
|
|
BUG_ON(mmu_psize_defs[mmu_psize].shift != shift);
|
|
|
|
|
2020-06-03 23:00:34 +00:00
|
|
|
return true;
|
|
|
|
}
|
2009-10-26 19:24:31 +00:00
|
|
|
|
2020-06-03 23:00:34 +00:00
|
|
|
static int __init add_huge_page_size(unsigned long long size)
|
|
|
|
{
|
|
|
|
int shift = __ffs(size);
|
|
|
|
|
|
|
|
if (!arch_hugetlb_valid_size((unsigned long)size))
|
|
|
|
return -EINVAL;
|
2009-10-26 19:24:31 +00:00
|
|
|
|
2020-06-03 23:00:42 +00:00
|
|
|
hugetlb_add_hstate(shift - PAGE_SHIFT);
|
2009-10-26 19:24:31 +00:00
|
|
|
return 0;
|
2008-01-03 22:59:50 +00:00
|
|
|
}
|
|
|
|
|
2011-06-28 09:54:48 +00:00
|
|
|
static int __init hugetlbpage_init(void)
|
|
|
|
{
|
2019-05-28 05:36:26 +00:00
|
|
|
bool configured = false;
|
2011-06-28 09:54:48 +00:00
|
|
|
int psize;
|
|
|
|
|
2018-04-10 13:41:31 +00:00
|
|
|
if (hugetlb_disabled) {
|
|
|
|
pr_info("HugeTLB support is disabled!\n");
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2019-04-26 05:59:49 +00:00
|
|
|
if (IS_ENABLED(CONFIG_PPC_BOOK3S_64) && !radix_enabled() &&
|
|
|
|
!mmu_has_feature(MMU_FTR_16M_PAGE))
|
2006-04-28 05:02:51 +00:00
|
|
|
return -ENODEV;
|
2019-04-26 05:59:49 +00:00
|
|
|
|
2009-10-26 19:24:31 +00:00
|
|
|
for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) {
|
|
|
|
unsigned shift;
|
|
|
|
unsigned pdshift;
|
2008-07-24 04:27:56 +00:00
|
|
|
|
2009-10-26 19:24:31 +00:00
|
|
|
if (!mmu_psize_defs[psize].shift)
|
|
|
|
continue;
|
2008-07-28 06:13:18 +00:00
|
|
|
|
2009-10-26 19:24:31 +00:00
|
|
|
shift = mmu_psize_to_shift(psize);
|
|
|
|
|
2018-03-30 12:04:08 +00:00
|
|
|
#ifdef CONFIG_PPC_BOOK3S_64
|
|
|
|
if (shift > PGDIR_SHIFT)
|
2009-10-26 19:24:31 +00:00
|
|
|
continue;
|
2018-03-30 12:04:08 +00:00
|
|
|
else if (shift > PUD_SHIFT)
|
|
|
|
pdshift = PGDIR_SHIFT;
|
|
|
|
else if (shift > PMD_SHIFT)
|
|
|
|
pdshift = PUD_SHIFT;
|
|
|
|
else
|
|
|
|
pdshift = PMD_SHIFT;
|
|
|
|
#else
|
2018-07-17 04:24:30 +00:00
|
|
|
if (shift < PUD_SHIFT)
|
2009-10-26 19:24:31 +00:00
|
|
|
pdshift = PMD_SHIFT;
|
2018-07-17 04:24:30 +00:00
|
|
|
else if (shift < PGDIR_SHIFT)
|
2009-10-26 19:24:31 +00:00
|
|
|
pdshift = PUD_SHIFT;
|
|
|
|
else
|
|
|
|
pdshift = PGDIR_SHIFT;
|
2018-03-30 12:04:08 +00:00
|
|
|
#endif
|
|
|
|
|
|
|
|
if (add_huge_page_size(1ULL << shift) < 0)
|
|
|
|
continue;
|
2013-04-28 09:37:30 +00:00
|
|
|
/*
|
|
|
|
* if we have pdshift and shift value same, we don't
|
|
|
|
* use pgt cache for hugepd.
|
|
|
|
*/
|
2020-02-06 13:50:28 +00:00
|
|
|
if (pdshift > shift) {
|
|
|
|
if (!IS_ENABLED(CONFIG_PPC_8xx))
|
|
|
|
pgtable_cache_add(pdshift - shift);
|
|
|
|
} else if (IS_ENABLED(CONFIG_PPC_FSL_BOOK3E) ||
|
|
|
|
IS_ENABLED(CONFIG_PPC_8xx)) {
|
2018-11-29 14:07:07 +00:00
|
|
|
pgtable_cache_add(PTE_T_ORDER);
|
2020-02-06 13:50:28 +00:00
|
|
|
}
|
2019-05-28 05:36:26 +00:00
|
|
|
|
|
|
|
configured = true;
|
2008-07-24 04:27:56 +00:00
|
|
|
}
|
2006-04-28 05:02:51 +00:00
|
|
|
|
2019-05-28 05:36:26 +00:00
|
|
|
if (configured) {
|
|
|
|
if (IS_ENABLED(CONFIG_HUGETLB_PAGE_SIZE_VARIABLE))
|
|
|
|
hugetlbpage_init_default();
|
|
|
|
} else
|
|
|
|
pr_info("Failed to initialize. Disabling HugeTLB");
|
2019-04-26 05:59:48 +00:00
|
|
|
|
2006-04-28 05:02:51 +00:00
|
|
|
return 0;
|
|
|
|
}
|
2016-12-07 07:47:26 +00:00
|
|
|
|
2015-05-02 00:08:21 +00:00
|
|
|
arch_initcall(hugetlbpage_init);
|
2009-10-26 19:24:31 +00:00
|
|
|
|
|
|
|
void flush_dcache_icache_hugepage(struct page *page)
|
|
|
|
{
|
|
|
|
int i;
|
2011-06-28 09:54:48 +00:00
|
|
|
void *start;
|
2009-10-26 19:24:31 +00:00
|
|
|
|
|
|
|
BUG_ON(!PageCompound(page));
|
|
|
|
|
2019-09-23 22:34:30 +00:00
|
|
|
for (i = 0; i < compound_nr(page); i++) {
|
2011-06-28 09:54:48 +00:00
|
|
|
if (!PageHighMem(page)) {
|
|
|
|
__flush_dcache_icache(page_address(page+i));
|
|
|
|
} else {
|
2011-11-25 15:14:16 +00:00
|
|
|
start = kmap_atomic(page+i);
|
2011-06-28 09:54:48 +00:00
|
|
|
__flush_dcache_icache(start);
|
2011-11-25 15:14:16 +00:00
|
|
|
kunmap_atomic(start);
|
2011-06-28 09:54:48 +00:00
|
|
|
}
|
|
|
|
}
|
2009-10-26 19:24:31 +00:00
|
|
|
}
|