2019-06-03 05:44:50 +00:00
|
|
|
// SPDX-License-Identifier: GPL-2.0-only
|
2015-10-23 07:26:37 +00:00
|
|
|
/*
|
|
|
|
* Copyright (C) 2015 - ARM Ltd
|
|
|
|
* Author: Marc Zyngier <marc.zyngier@arm.com>
|
|
|
|
*/
|
|
|
|
|
2018-12-06 17:31:19 +00:00
|
|
|
#include <linux/irqflags.h>
|
|
|
|
|
2016-01-28 13:44:07 +00:00
|
|
|
#include <asm/kvm_hyp.h>
|
2017-10-23 16:11:14 +00:00
|
|
|
#include <asm/kvm_mmu.h>
|
2017-01-25 15:52:31 +00:00
|
|
|
#include <asm/tlbflush.h>
|
2015-10-23 07:26:37 +00:00
|
|
|
|
2018-12-06 17:31:25 +00:00
|
|
|
struct tlb_inv_context {
|
2023-10-18 23:32:09 +00:00
|
|
|
struct kvm_s2_mmu *mmu;
|
|
|
|
unsigned long flags;
|
|
|
|
u64 tcr;
|
|
|
|
u64 sctlr;
|
2018-12-06 17:31:25 +00:00
|
|
|
};
|
|
|
|
|
2019-01-04 20:09:05 +00:00
|
|
|
static void __tlb_switch_to_guest(struct kvm_s2_mmu *mmu,
|
|
|
|
struct tlb_inv_context *cxt)
|
2017-02-17 14:32:18 +00:00
|
|
|
{
|
2023-10-18 23:32:09 +00:00
|
|
|
struct kvm_vcpu *vcpu = kvm_get_running_vcpu();
|
2017-02-17 14:32:18 +00:00
|
|
|
u64 val;
|
|
|
|
|
2018-12-06 17:31:25 +00:00
|
|
|
local_irq_save(cxt->flags);
|
|
|
|
|
2023-10-18 23:32:09 +00:00
|
|
|
if (vcpu && mmu != vcpu->arch.hw_mmu)
|
|
|
|
cxt->mmu = vcpu->arch.hw_mmu;
|
|
|
|
else
|
|
|
|
cxt->mmu = NULL;
|
|
|
|
|
2020-05-04 09:48:58 +00:00
|
|
|
if (cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT)) {
|
2018-12-06 17:31:25 +00:00
|
|
|
/*
|
2019-12-16 11:56:31 +00:00
|
|
|
* For CPUs that are affected by ARM errata 1165522 or 1530923,
|
|
|
|
* we cannot trust stage-1 to be in a correct state at that
|
2018-12-06 17:31:25 +00:00
|
|
|
* point. Since we do not want to force a full load of the
|
|
|
|
* vcpu state, we prevent the EL1 page-table walker to
|
|
|
|
* allocate new TLBs. This is done by setting the EPD bits
|
|
|
|
* in the TCR_EL1 register. We also need to prevent it to
|
|
|
|
* allocate IPA->PA walks, so we enable the S1 MMU...
|
|
|
|
*/
|
KVM: arm64: Migrate _elx sysreg accessors to msr_s/mrs_s
Currently, the {read,write}_sysreg_el*() accessors for accessing
particular ELs' sysregs in the presence of VHE rely on some local
hacks and define their system register encodings in a way that is
inconsistent with the core definitions in <asm/sysreg.h>.
As a result, it is necessary to add duplicate definitions for any
system register that already needs a definition in sysreg.h for
other reasons.
This is a bit of a maintenance headache, and the reasons for the
_el*() accessors working the way they do is a bit historical.
This patch gets rid of the shadow sysreg definitions in
<asm/kvm_hyp.h>, converts the _el*() accessors to use the core
__msr_s/__mrs_s interface, and converts all call sites to use the
standard sysreg #define names (i.e., upper case, with SYS_ prefix).
This patch will conflict heavily anyway, so the opportunity
to clean up some bad whitespace in the context of the changes is
taken.
The change exposes a few system registers that have no sysreg.h
definition, due to msr_s/mrs_s being used in place of msr/mrs:
additions are made in order to fill in the gaps.
Signed-off-by: Dave Martin <Dave.Martin@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Christoffer Dall <christoffer.dall@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Link: https://www.spinics.net/lists/kvm-arm/msg31717.html
[Rebased to v4.21-rc1]
Signed-off-by: Sudeep Holla <sudeep.holla@arm.com>
[Rebased to v5.2-rc5, changelog updates]
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2019-04-06 10:29:40 +00:00
|
|
|
val = cxt->tcr = read_sysreg_el1(SYS_TCR);
|
2018-12-06 17:31:25 +00:00
|
|
|
val |= TCR_EPD1_MASK | TCR_EPD0_MASK;
|
KVM: arm64: Migrate _elx sysreg accessors to msr_s/mrs_s
Currently, the {read,write}_sysreg_el*() accessors for accessing
particular ELs' sysregs in the presence of VHE rely on some local
hacks and define their system register encodings in a way that is
inconsistent with the core definitions in <asm/sysreg.h>.
As a result, it is necessary to add duplicate definitions for any
system register that already needs a definition in sysreg.h for
other reasons.
This is a bit of a maintenance headache, and the reasons for the
_el*() accessors working the way they do is a bit historical.
This patch gets rid of the shadow sysreg definitions in
<asm/kvm_hyp.h>, converts the _el*() accessors to use the core
__msr_s/__mrs_s interface, and converts all call sites to use the
standard sysreg #define names (i.e., upper case, with SYS_ prefix).
This patch will conflict heavily anyway, so the opportunity
to clean up some bad whitespace in the context of the changes is
taken.
The change exposes a few system registers that have no sysreg.h
definition, due to msr_s/mrs_s being used in place of msr/mrs:
additions are made in order to fill in the gaps.
Signed-off-by: Dave Martin <Dave.Martin@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Christoffer Dall <christoffer.dall@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Link: https://www.spinics.net/lists/kvm-arm/msg31717.html
[Rebased to v4.21-rc1]
Signed-off-by: Sudeep Holla <sudeep.holla@arm.com>
[Rebased to v5.2-rc5, changelog updates]
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2019-04-06 10:29:40 +00:00
|
|
|
write_sysreg_el1(val, SYS_TCR);
|
|
|
|
val = cxt->sctlr = read_sysreg_el1(SYS_SCTLR);
|
2018-12-06 17:31:25 +00:00
|
|
|
val |= SCTLR_ELx_M;
|
KVM: arm64: Migrate _elx sysreg accessors to msr_s/mrs_s
Currently, the {read,write}_sysreg_el*() accessors for accessing
particular ELs' sysregs in the presence of VHE rely on some local
hacks and define their system register encodings in a way that is
inconsistent with the core definitions in <asm/sysreg.h>.
As a result, it is necessary to add duplicate definitions for any
system register that already needs a definition in sysreg.h for
other reasons.
This is a bit of a maintenance headache, and the reasons for the
_el*() accessors working the way they do is a bit historical.
This patch gets rid of the shadow sysreg definitions in
<asm/kvm_hyp.h>, converts the _el*() accessors to use the core
__msr_s/__mrs_s interface, and converts all call sites to use the
standard sysreg #define names (i.e., upper case, with SYS_ prefix).
This patch will conflict heavily anyway, so the opportunity
to clean up some bad whitespace in the context of the changes is
taken.
The change exposes a few system registers that have no sysreg.h
definition, due to msr_s/mrs_s being used in place of msr/mrs:
additions are made in order to fill in the gaps.
Signed-off-by: Dave Martin <Dave.Martin@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Christoffer Dall <christoffer.dall@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Link: https://www.spinics.net/lists/kvm-arm/msg31717.html
[Rebased to v4.21-rc1]
Signed-off-by: Sudeep Holla <sudeep.holla@arm.com>
[Rebased to v5.2-rc5, changelog updates]
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2019-04-06 10:29:40 +00:00
|
|
|
write_sysreg_el1(val, SYS_SCTLR);
|
2018-12-06 17:31:25 +00:00
|
|
|
}
|
2018-12-06 17:31:19 +00:00
|
|
|
|
2017-02-17 14:32:18 +00:00
|
|
|
/*
|
|
|
|
* With VHE enabled, we have HCR_EL2.{E2H,TGE} = {1,1}, and
|
|
|
|
* most TLB operations target EL2/EL0. In order to affect the
|
|
|
|
* guest TLBs (EL1/EL0), we need to change one of these two
|
|
|
|
* bits. Changing E2H is impossible (goodbye TTBR1_EL2), so
|
|
|
|
* let's flip TGE before executing the TLB operation.
|
2018-12-06 17:31:25 +00:00
|
|
|
*
|
|
|
|
* ARM erratum 1165522 requires some special handling (again),
|
|
|
|
* as we need to make sure both stages of translation are in
|
2021-08-06 11:31:07 +00:00
|
|
|
* place before clearing TGE. __load_stage2() already
|
2018-12-06 17:31:25 +00:00
|
|
|
* has an ISB in order to deal with this.
|
2017-02-17 14:32:18 +00:00
|
|
|
*/
|
2021-08-06 11:31:07 +00:00
|
|
|
__load_stage2(mmu, mmu->arch);
|
2017-02-17 14:32:18 +00:00
|
|
|
val = read_sysreg(hcr_el2);
|
|
|
|
val &= ~HCR_TGE;
|
|
|
|
write_sysreg(val, hcr_el2);
|
|
|
|
isb();
|
|
|
|
}
|
|
|
|
|
2019-01-04 20:09:05 +00:00
|
|
|
static void __tlb_switch_to_host(struct tlb_inv_context *cxt)
|
2017-02-17 14:32:18 +00:00
|
|
|
{
|
|
|
|
/*
|
|
|
|
* We're done with the TLB operation, let's restore the host's
|
|
|
|
* view of HCR_EL2.
|
|
|
|
*/
|
|
|
|
write_sysreg(HCR_HOST_VHE_FLAGS, hcr_el2);
|
2018-12-06 17:31:19 +00:00
|
|
|
isb();
|
2018-12-06 17:31:25 +00:00
|
|
|
|
2023-10-18 23:32:09 +00:00
|
|
|
/* ... and the stage-2 MMU context that we switched away from */
|
|
|
|
if (cxt->mmu)
|
|
|
|
__load_stage2(cxt->mmu, cxt->mmu->arch);
|
|
|
|
|
2020-05-04 09:48:58 +00:00
|
|
|
if (cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT)) {
|
2018-12-06 17:31:25 +00:00
|
|
|
/* Restore the registers to what they were */
|
KVM: arm64: Migrate _elx sysreg accessors to msr_s/mrs_s
Currently, the {read,write}_sysreg_el*() accessors for accessing
particular ELs' sysregs in the presence of VHE rely on some local
hacks and define their system register encodings in a way that is
inconsistent with the core definitions in <asm/sysreg.h>.
As a result, it is necessary to add duplicate definitions for any
system register that already needs a definition in sysreg.h for
other reasons.
This is a bit of a maintenance headache, and the reasons for the
_el*() accessors working the way they do is a bit historical.
This patch gets rid of the shadow sysreg definitions in
<asm/kvm_hyp.h>, converts the _el*() accessors to use the core
__msr_s/__mrs_s interface, and converts all call sites to use the
standard sysreg #define names (i.e., upper case, with SYS_ prefix).
This patch will conflict heavily anyway, so the opportunity
to clean up some bad whitespace in the context of the changes is
taken.
The change exposes a few system registers that have no sysreg.h
definition, due to msr_s/mrs_s being used in place of msr/mrs:
additions are made in order to fill in the gaps.
Signed-off-by: Dave Martin <Dave.Martin@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Christoffer Dall <christoffer.dall@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Link: https://www.spinics.net/lists/kvm-arm/msg31717.html
[Rebased to v4.21-rc1]
Signed-off-by: Sudeep Holla <sudeep.holla@arm.com>
[Rebased to v5.2-rc5, changelog updates]
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2019-04-06 10:29:40 +00:00
|
|
|
write_sysreg_el1(cxt->tcr, SYS_TCR);
|
|
|
|
write_sysreg_el1(cxt->sctlr, SYS_SCTLR);
|
2018-12-06 17:31:25 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
local_irq_restore(cxt->flags);
|
2017-02-17 14:32:18 +00:00
|
|
|
}
|
|
|
|
|
2019-01-02 12:34:25 +00:00
|
|
|
void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu,
|
|
|
|
phys_addr_t ipa, int level)
|
2015-10-23 07:26:37 +00:00
|
|
|
{
|
2018-12-06 17:31:25 +00:00
|
|
|
struct tlb_inv_context cxt;
|
2018-12-06 17:31:19 +00:00
|
|
|
|
2015-10-23 07:26:37 +00:00
|
|
|
dsb(ishst);
|
|
|
|
|
|
|
|
/* Switch to requested VMID */
|
2019-01-04 20:09:05 +00:00
|
|
|
__tlb_switch_to_guest(mmu, &cxt);
|
2015-10-23 07:26:37 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* We could do so much better if we had the VA as well.
|
|
|
|
* Instead, we invalidate Stage-2 for this IPA, and the
|
|
|
|
* whole of Stage-1. Weep...
|
|
|
|
*/
|
|
|
|
ipa >>= 12;
|
2019-01-02 12:34:25 +00:00
|
|
|
__tlbi_level(ipas2e1is, ipa, level);
|
2015-10-23 07:26:37 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* We have to ensure completion of the invalidation at Stage-2,
|
|
|
|
* since a table walk on another CPU could refill a TLB with a
|
|
|
|
* complete (S1 + S2) walk based on the old Stage-2 mapping if
|
|
|
|
* the Stage-1 invalidation happened first.
|
|
|
|
*/
|
|
|
|
dsb(ish);
|
2017-01-25 15:52:31 +00:00
|
|
|
__tlbi(vmalle1is);
|
2015-10-23 07:26:37 +00:00
|
|
|
dsb(ish);
|
|
|
|
isb();
|
|
|
|
|
2019-01-04 20:09:05 +00:00
|
|
|
__tlb_switch_to_host(&cxt);
|
2015-10-23 07:26:37 +00:00
|
|
|
}
|
|
|
|
|
2023-04-26 17:23:30 +00:00
|
|
|
void __kvm_tlb_flush_vmid_ipa_nsh(struct kvm_s2_mmu *mmu,
|
|
|
|
phys_addr_t ipa, int level)
|
|
|
|
{
|
|
|
|
struct tlb_inv_context cxt;
|
|
|
|
|
|
|
|
dsb(nshst);
|
|
|
|
|
|
|
|
/* Switch to requested VMID */
|
|
|
|
__tlb_switch_to_guest(mmu, &cxt);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We could do so much better if we had the VA as well.
|
|
|
|
* Instead, we invalidate Stage-2 for this IPA, and the
|
|
|
|
* whole of Stage-1. Weep...
|
|
|
|
*/
|
|
|
|
ipa >>= 12;
|
|
|
|
__tlbi_level(ipas2e1, ipa, level);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We have to ensure completion of the invalidation at Stage-2,
|
|
|
|
* since a table walk on another CPU could refill a TLB with a
|
|
|
|
* complete (S1 + S2) walk based on the old Stage-2 mapping if
|
|
|
|
* the Stage-1 invalidation happened first.
|
|
|
|
*/
|
|
|
|
dsb(nsh);
|
|
|
|
__tlbi(vmalle1);
|
|
|
|
dsb(nsh);
|
|
|
|
isb();
|
|
|
|
|
|
|
|
__tlb_switch_to_host(&cxt);
|
|
|
|
}
|
|
|
|
|
2023-08-11 04:51:22 +00:00
|
|
|
void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
|
|
|
|
phys_addr_t start, unsigned long pages)
|
|
|
|
{
|
|
|
|
struct tlb_inv_context cxt;
|
|
|
|
unsigned long stride;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Since the range of addresses may not be mapped at
|
|
|
|
* the same level, assume the worst case as PAGE_SIZE
|
|
|
|
*/
|
|
|
|
stride = PAGE_SIZE;
|
|
|
|
start = round_down(start, stride);
|
|
|
|
|
|
|
|
dsb(ishst);
|
|
|
|
|
|
|
|
/* Switch to requested VMID */
|
|
|
|
__tlb_switch_to_guest(mmu, &cxt);
|
|
|
|
|
|
|
|
__flush_s2_tlb_range_op(ipas2e1is, start, pages, stride, 0);
|
|
|
|
|
|
|
|
dsb(ish);
|
|
|
|
__tlbi(vmalle1is);
|
|
|
|
dsb(ish);
|
|
|
|
isb();
|
|
|
|
|
|
|
|
__tlb_switch_to_host(&cxt);
|
|
|
|
}
|
|
|
|
|
2019-01-04 20:09:05 +00:00
|
|
|
void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu)
|
2015-10-23 07:26:37 +00:00
|
|
|
{
|
2018-12-06 17:31:25 +00:00
|
|
|
struct tlb_inv_context cxt;
|
2018-12-06 17:31:19 +00:00
|
|
|
|
2015-10-23 07:26:37 +00:00
|
|
|
dsb(ishst);
|
|
|
|
|
|
|
|
/* Switch to requested VMID */
|
2019-01-04 20:09:05 +00:00
|
|
|
__tlb_switch_to_guest(mmu, &cxt);
|
2015-10-23 07:26:37 +00:00
|
|
|
|
2017-01-25 15:52:31 +00:00
|
|
|
__tlbi(vmalls12e1is);
|
2015-10-23 07:26:37 +00:00
|
|
|
dsb(ish);
|
|
|
|
isb();
|
|
|
|
|
2019-01-04 20:09:05 +00:00
|
|
|
__tlb_switch_to_host(&cxt);
|
2015-10-23 07:26:37 +00:00
|
|
|
}
|
|
|
|
|
2021-03-03 16:45:05 +00:00
|
|
|
void __kvm_flush_cpu_context(struct kvm_s2_mmu *mmu)
|
2016-10-18 17:37:49 +00:00
|
|
|
{
|
2018-12-06 17:31:25 +00:00
|
|
|
struct tlb_inv_context cxt;
|
2016-10-18 17:37:49 +00:00
|
|
|
|
|
|
|
/* Switch to requested VMID */
|
2019-01-04 20:09:05 +00:00
|
|
|
__tlb_switch_to_guest(mmu, &cxt);
|
2016-10-18 17:37:49 +00:00
|
|
|
|
2017-01-25 15:52:31 +00:00
|
|
|
__tlbi(vmalle1);
|
2021-03-03 16:45:05 +00:00
|
|
|
asm volatile("ic iallu");
|
2016-10-18 17:37:49 +00:00
|
|
|
dsb(nsh);
|
|
|
|
isb();
|
|
|
|
|
2019-01-04 20:09:05 +00:00
|
|
|
__tlb_switch_to_host(&cxt);
|
2016-10-18 17:37:49 +00:00
|
|
|
}
|
|
|
|
|
2020-06-25 13:14:13 +00:00
|
|
|
void __kvm_flush_vm_context(void)
|
2015-10-23 07:26:37 +00:00
|
|
|
{
|
|
|
|
dsb(ishst);
|
2017-01-25 15:52:31 +00:00
|
|
|
__tlbi(alle1is);
|
arm64/kvm: Remove VMID rollover I-cache maintenance
For VPIPT I-caches, we need I-cache maintenance on VMID rollover to
avoid an ABA problem. Consider a single vCPU VM, with a pinned stage-2,
running with an idmap VA->IPA and idmap IPA->PA. If we don't do
maintenance on rollover:
// VMID A
Writes insn X to PA 0xF
Invalidates PA 0xF (for VMID A)
I$ contains [{A,F}->X]
[VMID ROLLOVER]
// VMID B
Writes insn Y to PA 0xF
Invalidates PA 0xF (for VMID B)
I$ contains [{A,F}->X, {B,F}->Y]
[VMID ROLLOVER]
// VMID A
I$ contains [{A,F}->X, {B,F}->Y]
Unexpectedly hits stale I$ line {A,F}->X.
However, for PIPT and VIPT I-caches, the VMID doesn't affect lookup or
constrain maintenance. Given the VMID doesn't affect PIPT and VIPT
I-caches, and given VMID rollover is independent of changes to stage-2
mappings, I-cache maintenance cannot be necessary on VMID rollover for
PIPT or VIPT I-caches.
This patch removes the maintenance on rollover for VIPT and PIPT
I-caches. At the same time, the unnecessary colons are removed from the
asm statement to make it more legible.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Christoffer Dall <christoffer.dall@arm.com>
Reviewed-by: James Morse <james.morse@arm.com>
Cc: Julien Thierry <julien.thierry.kdev@gmail.com>
Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
Cc: kvmarm@lists.cs.columbia.edu
Signed-off-by: Marc Zyngier <maz@kernel.org>
2019-08-06 15:57:37 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* VIPT and PIPT caches are not affected by VMID, so no maintenance
|
|
|
|
* is necessary across a VMID rollover.
|
|
|
|
*
|
|
|
|
* VPIPT caches constrain lookup and maintenance to the active VMID,
|
|
|
|
* so we need to invalidate lines with a stale VMID to avoid an ABA
|
|
|
|
* race after multiple rollovers.
|
|
|
|
*
|
|
|
|
*/
|
|
|
|
if (icache_is_vpipt())
|
|
|
|
asm volatile("ic ialluis");
|
|
|
|
|
2015-10-23 07:26:37 +00:00
|
|
|
dsb(ish);
|
|
|
|
}
|