2013-01-20 23:28:06 +00:00
|
|
|
/*
|
|
|
|
* Copyright (C) 2012 - Virtual Open Systems and Columbia University
|
|
|
|
* Author: Christoffer Dall <c.dall@virtualopensystems.com>
|
|
|
|
*
|
|
|
|
* This program is free software; you can redistribute it and/or modify
|
|
|
|
* it under the terms of the GNU General Public License, version 2, as
|
|
|
|
* published by the Free Software Foundation.
|
|
|
|
*
|
|
|
|
* This program is distributed in the hope that it will be useful,
|
|
|
|
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
|
|
* GNU General Public License for more details.
|
|
|
|
*
|
|
|
|
* You should have received a copy of the GNU General Public License
|
|
|
|
* along with this program; if not, write to the Free Software
|
|
|
|
* Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
|
|
|
*/
|
|
|
|
|
|
|
|
#ifndef __ARM_KVM_MMU_H__
|
|
|
|
#define __ARM_KVM_MMU_H__
|
|
|
|
|
ARM: KVM: switch to a dual-step HYP init code
Our HYP init code suffers from two major design issues:
- it cannot support CPU hotplug, as we tear down the idmap very early
- it cannot perform a TLB invalidation when switching from init to
runtime mappings, as pages are manipulated from PL1 exclusively
The hotplug problem mandates that we keep two sets of page tables
(boot and runtime). The TLB problem mandates that we're able to
transition from one PGD to another while in HYP, invalidating the TLBs
in the process.
To be able to do this, we need to share a page between the two page
tables. A page that will have the same VA in both configurations. All we
need is a VA that has the following properties:
- This VA can't be used to represent a kernel mapping.
- This VA will not conflict with the physical address of the kernel text
The vectors page seems to satisfy this requirement:
- The kernel never maps anything else there
- The kernel text being copied at the beginning of the physical memory,
it is unlikely to use the last 64kB (I doubt we'll ever support KVM
on a system with something like 4MB of RAM, but patches are very
welcome).
Let's call this VA the trampoline VA.
Now, we map our init page at 3 locations:
- idmap in the boot pgd
- trampoline VA in the boot pgd
- trampoline VA in the runtime pgd
The init scenario is now the following:
- We jump in HYP with four parameters: boot HYP pgd, runtime HYP pgd,
runtime stack, runtime vectors
- Enable the MMU with the boot pgd
- Jump to a target into the trampoline page (remember, this is the same
physical page!)
- Now switch to the runtime pgd (same VA, and still the same physical
page!)
- Invalidate TLBs
- Set stack and vectors
- Profit! (or eret, if you only care about the code).
Note that we keep the boot mapping permanently (it is not strictly an
idmap anymore) to allow for CPU hotplug in later patches.
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
Signed-off-by: Christoffer Dall <cdall@cs.columbia.edu>
2013-04-12 18:12:06 +00:00
|
|
|
#include <asm/memory.h>
|
|
|
|
#include <asm/page.h>
|
2012-10-15 10:27:37 +00:00
|
|
|
|
2012-10-28 00:09:14 +00:00
|
|
|
/*
|
|
|
|
* We directly use the kernel VA for the HYP, as we can directly share
|
|
|
|
* the mapping (HTTBR "covers" TTBR1).
|
|
|
|
*/
|
ARM: KVM: switch to a dual-step HYP init code
Our HYP init code suffers from two major design issues:
- it cannot support CPU hotplug, as we tear down the idmap very early
- it cannot perform a TLB invalidation when switching from init to
runtime mappings, as pages are manipulated from PL1 exclusively
The hotplug problem mandates that we keep two sets of page tables
(boot and runtime). The TLB problem mandates that we're able to
transition from one PGD to another while in HYP, invalidating the TLBs
in the process.
To be able to do this, we need to share a page between the two page
tables. A page that will have the same VA in both configurations. All we
need is a VA that has the following properties:
- This VA can't be used to represent a kernel mapping.
- This VA will not conflict with the physical address of the kernel text
The vectors page seems to satisfy this requirement:
- The kernel never maps anything else there
- The kernel text being copied at the beginning of the physical memory,
it is unlikely to use the last 64kB (I doubt we'll ever support KVM
on a system with something like 4MB of RAM, but patches are very
welcome).
Let's call this VA the trampoline VA.
Now, we map our init page at 3 locations:
- idmap in the boot pgd
- trampoline VA in the boot pgd
- trampoline VA in the runtime pgd
The init scenario is now the following:
- We jump in HYP with four parameters: boot HYP pgd, runtime HYP pgd,
runtime stack, runtime vectors
- Enable the MMU with the boot pgd
- Jump to a target into the trampoline page (remember, this is the same
physical page!)
- Now switch to the runtime pgd (same VA, and still the same physical
page!)
- Invalidate TLBs
- Set stack and vectors
- Profit! (or eret, if you only care about the code).
Note that we keep the boot mapping permanently (it is not strictly an
idmap anymore) to allow for CPU hotplug in later patches.
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
Signed-off-by: Christoffer Dall <cdall@cs.columbia.edu>
2013-04-12 18:12:06 +00:00
|
|
|
#define HYP_PAGE_OFFSET_MASK UL(~0)
|
2012-10-28 00:09:14 +00:00
|
|
|
#define HYP_PAGE_OFFSET PAGE_OFFSET
|
|
|
|
#define KERN_TO_HYP(kva) (kva)
|
|
|
|
|
ARM: KVM: switch to a dual-step HYP init code
Our HYP init code suffers from two major design issues:
- it cannot support CPU hotplug, as we tear down the idmap very early
- it cannot perform a TLB invalidation when switching from init to
runtime mappings, as pages are manipulated from PL1 exclusively
The hotplug problem mandates that we keep two sets of page tables
(boot and runtime). The TLB problem mandates that we're able to
transition from one PGD to another while in HYP, invalidating the TLBs
in the process.
To be able to do this, we need to share a page between the two page
tables. A page that will have the same VA in both configurations. All we
need is a VA that has the following properties:
- This VA can't be used to represent a kernel mapping.
- This VA will not conflict with the physical address of the kernel text
The vectors page seems to satisfy this requirement:
- The kernel never maps anything else there
- The kernel text being copied at the beginning of the physical memory,
it is unlikely to use the last 64kB (I doubt we'll ever support KVM
on a system with something like 4MB of RAM, but patches are very
welcome).
Let's call this VA the trampoline VA.
Now, we map our init page at 3 locations:
- idmap in the boot pgd
- trampoline VA in the boot pgd
- trampoline VA in the runtime pgd
The init scenario is now the following:
- We jump in HYP with four parameters: boot HYP pgd, runtime HYP pgd,
runtime stack, runtime vectors
- Enable the MMU with the boot pgd
- Jump to a target into the trampoline page (remember, this is the same
physical page!)
- Now switch to the runtime pgd (same VA, and still the same physical
page!)
- Invalidate TLBs
- Set stack and vectors
- Profit! (or eret, if you only care about the code).
Note that we keep the boot mapping permanently (it is not strictly an
idmap anymore) to allow for CPU hotplug in later patches.
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
Signed-off-by: Christoffer Dall <cdall@cs.columbia.edu>
2013-04-12 18:12:06 +00:00
|
|
|
/*
|
|
|
|
* Our virtual mapping for the boot-time MMU-enable code. Must be
|
|
|
|
* shared across all the page-tables. Conveniently, we use the vectors
|
|
|
|
* page, where no kernel data will ever be shared with HYP.
|
|
|
|
*/
|
|
|
|
#define TRAMPOLINE_VA UL(CONFIG_VECTORS_BASE)
|
|
|
|
|
|
|
|
#ifndef __ASSEMBLY__
|
|
|
|
|
|
|
|
#include <asm/cacheflush.h>
|
|
|
|
#include <asm/pgalloc.h>
|
|
|
|
|
2013-01-20 23:28:06 +00:00
|
|
|
int create_hyp_mappings(void *from, void *to);
|
|
|
|
int create_hyp_io_mappings(void *from, void *to, phys_addr_t);
|
2013-04-12 18:12:07 +00:00
|
|
|
void free_boot_hyp_pgd(void);
|
2013-04-12 18:12:05 +00:00
|
|
|
void free_hyp_pgds(void);
|
2013-01-20 23:28:06 +00:00
|
|
|
|
2013-01-20 23:28:07 +00:00
|
|
|
int kvm_alloc_stage2_pgd(struct kvm *kvm);
|
|
|
|
void kvm_free_stage2_pgd(struct kvm *kvm);
|
|
|
|
int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa,
|
|
|
|
phys_addr_t pa, unsigned long size);
|
|
|
|
|
|
|
|
int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run);
|
|
|
|
|
|
|
|
void kvm_mmu_free_memory_caches(struct kvm_vcpu *vcpu);
|
|
|
|
|
2013-01-20 23:28:06 +00:00
|
|
|
phys_addr_t kvm_mmu_get_httbr(void);
|
ARM: KVM: switch to a dual-step HYP init code
Our HYP init code suffers from two major design issues:
- it cannot support CPU hotplug, as we tear down the idmap very early
- it cannot perform a TLB invalidation when switching from init to
runtime mappings, as pages are manipulated from PL1 exclusively
The hotplug problem mandates that we keep two sets of page tables
(boot and runtime). The TLB problem mandates that we're able to
transition from one PGD to another while in HYP, invalidating the TLBs
in the process.
To be able to do this, we need to share a page between the two page
tables. A page that will have the same VA in both configurations. All we
need is a VA that has the following properties:
- This VA can't be used to represent a kernel mapping.
- This VA will not conflict with the physical address of the kernel text
The vectors page seems to satisfy this requirement:
- The kernel never maps anything else there
- The kernel text being copied at the beginning of the physical memory,
it is unlikely to use the last 64kB (I doubt we'll ever support KVM
on a system with something like 4MB of RAM, but patches are very
welcome).
Let's call this VA the trampoline VA.
Now, we map our init page at 3 locations:
- idmap in the boot pgd
- trampoline VA in the boot pgd
- trampoline VA in the runtime pgd
The init scenario is now the following:
- We jump in HYP with four parameters: boot HYP pgd, runtime HYP pgd,
runtime stack, runtime vectors
- Enable the MMU with the boot pgd
- Jump to a target into the trampoline page (remember, this is the same
physical page!)
- Now switch to the runtime pgd (same VA, and still the same physical
page!)
- Invalidate TLBs
- Set stack and vectors
- Profit! (or eret, if you only care about the code).
Note that we keep the boot mapping permanently (it is not strictly an
idmap anymore) to allow for CPU hotplug in later patches.
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
Signed-off-by: Christoffer Dall <cdall@cs.columbia.edu>
2013-04-12 18:12:06 +00:00
|
|
|
phys_addr_t kvm_mmu_get_boot_httbr(void);
|
|
|
|
phys_addr_t kvm_get_idmap_vector(void);
|
2013-01-20 23:28:06 +00:00
|
|
|
int kvm_mmu_init(void);
|
|
|
|
void kvm_clear_hyp_idmap(void);
|
2013-01-20 23:28:12 +00:00
|
|
|
|
2012-10-15 10:27:37 +00:00
|
|
|
static inline void kvm_set_pte(pte_t *pte, pte_t new_pte)
|
|
|
|
{
|
2013-08-09 03:35:07 +00:00
|
|
|
*pte = new_pte;
|
2012-10-15 10:27:37 +00:00
|
|
|
/*
|
|
|
|
* flush_pmd_entry just takes a void pointer and cleans the necessary
|
|
|
|
* cache entries, so we can reuse the function for ptes.
|
|
|
|
*/
|
|
|
|
flush_pmd_entry(pte);
|
|
|
|
}
|
|
|
|
|
2013-01-20 23:28:12 +00:00
|
|
|
static inline bool kvm_is_write_fault(unsigned long hsr)
|
|
|
|
{
|
|
|
|
unsigned long hsr_ec = hsr >> HSR_EC_SHIFT;
|
|
|
|
if (hsr_ec == HSR_EC_IABT)
|
|
|
|
return false;
|
|
|
|
else if ((hsr & HSR_ISV) && !(hsr & HSR_WNR))
|
|
|
|
return false;
|
|
|
|
else
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
2012-10-15 10:27:37 +00:00
|
|
|
static inline void kvm_clean_pgd(pgd_t *pgd)
|
|
|
|
{
|
|
|
|
clean_dcache_area(pgd, PTRS_PER_S2_PGD * sizeof(pgd_t));
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void kvm_clean_pmd_entry(pmd_t *pmd)
|
|
|
|
{
|
|
|
|
clean_pmd_entry(pmd);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void kvm_clean_pte(pte_t *pte)
|
|
|
|
{
|
|
|
|
clean_pte_table(pte);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void kvm_set_s2pte_writable(pte_t *pte)
|
|
|
|
{
|
|
|
|
pte_val(*pte) |= L_PTE_S2_RDWR;
|
|
|
|
}
|
|
|
|
|
|
|
|
struct kvm;
|
|
|
|
|
|
|
|
static inline void coherent_icache_guest_page(struct kvm *kvm, gfn_t gfn)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* If we are going to insert an instruction page and the icache is
|
|
|
|
* either VIPT or PIPT, there is a potential problem where the host
|
|
|
|
* (or another VM) may have used the same page as this guest, and we
|
|
|
|
* read incorrect data from the icache. If we're using a PIPT cache,
|
|
|
|
* we can invalidate just that page, but if we are using a VIPT cache
|
|
|
|
* we need to invalidate the entire icache - damn shame - as written
|
|
|
|
* in the ARM ARM (DDI 0406C.b - Page B3-1393).
|
|
|
|
*
|
|
|
|
* VIVT caches are tagged using both the ASID and the VMID and doesn't
|
|
|
|
* need any kind of flushing (DDI 0406C.b - Page B3-1392).
|
|
|
|
*/
|
|
|
|
if (icache_is_pipt()) {
|
|
|
|
unsigned long hva = gfn_to_hva(kvm, gfn);
|
|
|
|
__cpuc_coherent_user_range(hva, hva + PAGE_SIZE);
|
|
|
|
} else if (!icache_is_vivt_asid_tagged()) {
|
|
|
|
/* any kind of VIPT cache */
|
|
|
|
__flush_icache_all();
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
ARM: KVM: switch to a dual-step HYP init code
Our HYP init code suffers from two major design issues:
- it cannot support CPU hotplug, as we tear down the idmap very early
- it cannot perform a TLB invalidation when switching from init to
runtime mappings, as pages are manipulated from PL1 exclusively
The hotplug problem mandates that we keep two sets of page tables
(boot and runtime). The TLB problem mandates that we're able to
transition from one PGD to another while in HYP, invalidating the TLBs
in the process.
To be able to do this, we need to share a page between the two page
tables. A page that will have the same VA in both configurations. All we
need is a VA that has the following properties:
- This VA can't be used to represent a kernel mapping.
- This VA will not conflict with the physical address of the kernel text
The vectors page seems to satisfy this requirement:
- The kernel never maps anything else there
- The kernel text being copied at the beginning of the physical memory,
it is unlikely to use the last 64kB (I doubt we'll ever support KVM
on a system with something like 4MB of RAM, but patches are very
welcome).
Let's call this VA the trampoline VA.
Now, we map our init page at 3 locations:
- idmap in the boot pgd
- trampoline VA in the boot pgd
- trampoline VA in the runtime pgd
The init scenario is now the following:
- We jump in HYP with four parameters: boot HYP pgd, runtime HYP pgd,
runtime stack, runtime vectors
- Enable the MMU with the boot pgd
- Jump to a target into the trampoline page (remember, this is the same
physical page!)
- Now switch to the runtime pgd (same VA, and still the same physical
page!)
- Invalidate TLBs
- Set stack and vectors
- Profit! (or eret, if you only care about the code).
Note that we keep the boot mapping permanently (it is not strictly an
idmap anymore) to allow for CPU hotplug in later patches.
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
Signed-off-by: Christoffer Dall <cdall@cs.columbia.edu>
2013-04-12 18:12:06 +00:00
|
|
|
#define kvm_flush_dcache_to_poc(a,l) __cpuc_flush_dcache_area((a), (l))
|
|
|
|
|
|
|
|
#endif /* !__ASSEMBLY__ */
|
|
|
|
|
2013-01-20 23:28:06 +00:00
|
|
|
#endif /* __ARM_KVM_MMU_H__ */
|