mirrors
/
linux-stable
mirror of https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
1
0
Fork 0
Code Issues Releases Wiki Activity

ARM: 

* Generalized infrastructure for 'writable' ID registers, effectively
   allowing userspace to opt-out of certain vCPU features for its guest
 
 * Optimization for vSGI injection, opportunistically compressing MPIDR
   to vCPU mapping into a table
 
 * Improvements to KVM's PMU emulation, allowing userspace to select
   the number of PMCs available to a VM
 
 * Guest support for memory operation instructions (FEAT_MOPS)
 
 * Cleanups to handling feature flags in KVM_ARM_VCPU_INIT, squashing
   bugs and getting rid of useless code
 
 * Changes to the way the SMCCC filter is constructed, avoiding wasted
   memory allocations when not in use
 
 * Load the stage-2 MMU context at vcpu_load() for VHE systems, reducing
   the overhead of errata mitigations
 
 * Miscellaneous kernel and selftest fixes
 
 LoongArch:
 
 * New architecture.  The hardware uses the same model as x86, s390
   and RISC-V, where guest/host mode is orthogonal to supervisor/user
   mode.  The virtualization extensions are very similar to MIPS,
   therefore the code also has some similarities but it's been cleaned
   up to avoid some of the historical bogosities that are found in
   arch/mips.  The kernel emulates MMU, timer and CSR accesses, while
   interrupt controllers are only emulated in userspace, at least for
   now.
 
 RISC-V:
 
 * Support for the Smstateen and Zicond extensions
 
 * Support for virtualizing senvcfg
 
 * Support for virtualized SBI debug console (DBCN)
 
 S390:
 
 * Nested page table management can be monitored through tracepoints
   and statistics
 
 x86:
 
 * Fix incorrect handling of VMX posted interrupt descriptor in KVM_SET_LAPIC,
   which could result in a dropped timer IRQ
 
 * Avoid WARN on systems with Intel IPI virtualization
 
 * Add CONFIG_KVM_MAX_NR_VCPUS, to allow supporting up to 4096 vCPUs without
   forcing more common use cases to eat the extra memory overhead.
 
 * Add virtualization support for AMD SRSO mitigation (IBPB_BRTYPE and
   SBPB, aka Selective Branch Predictor Barrier).
 
 * Fix a bug where restoring a vCPU snapshot that was taken within 1 second of
   creating the original vCPU would cause KVM to try to synchronize the vCPU's
   TSC and thus clobber the correct TSC being set by userspace.
 
 * Compute guest wall clock using a single TSC read to avoid generating an
   inaccurate time, e.g. if the vCPU is preempted between multiple TSC reads.
 
 * "Virtualize" HWCR.TscFreqSel to make Linux guests happy, which complain
   about a "Firmware Bug" if the bit isn't set for select F/M/S combos.
   Likewise "virtualize" (ignore) MSR_AMD64_TW_CFG to appease Windows Server
   2022.
 
 * Don't apply side effects to Hyper-V's synthetic timer on writes from
   userspace to fix an issue where the auto-enable behavior can trigger
   spurious interrupts, i.e. do auto-enabling only for guest writes.
 
 * Remove an unnecessary kick of all vCPUs when synchronizing the dirty log
   without PML enabled.
 
 * Advertise "support" for non-serializing FS/GS base MSR writes as appropriate.
 
 * Harden the fast page fault path to guard against encountering an invalid
   root when walking SPTEs.
 
 * Omit "struct kvm_vcpu_xen" entirely when CONFIG_KVM_XEN=n.
 
 * Use the fast path directly from the timer callback when delivering Xen
   timer events, instead of waiting for the next iteration of the run loop.
   This was not done so far because previously proposed code had races,
   but now care is taken to stop the hrtimer at critical points such as
   restarting the timer or saving the timer information for userspace.
 
 * Follow the lead of upstream Xen and ignore the VCPU_SSHOTTMR_future flag.
 
 * Optimize injection of PMU interrupts that are simultaneous with NMIs.
 
 * Usual handful of fixes for typos and other warts.
 
 x86 - MTRR/PAT fixes and optimizations:
 
 * Clean up code that deals with honoring guest MTRRs when the VM has
   non-coherent DMA and host MTRRs are ignored, i.e. EPT is enabled.
 
 * Zap EPT entries when non-coherent DMA assignment stops/start to prevent
   using stale entries with the wrong memtype.
 
 * Don't ignore guest PAT for CR0.CD=1 && KVM_X86_QUIRK_CD_NW_CLEARED=y.
   This was done as a workaround for virtual machine BIOSes that did not
   bother to clear CR0.CD (because ancient KVM/QEMU did not bother to
   set it, in turn), and there's zero reason to extend the quirk to
   also ignore guest PAT.
 
 x86 - SEV fixes:
 
 * Report KVM_EXIT_SHUTDOWN instead of EINVAL if KVM intercepts SHUTDOWN while
   running an SEV-ES guest.
 
 * Clean up the recognition of emulation failures on SEV guests, when KVM would
   like to "skip" the instruction but it had already been partially emulated.
   This makes it possible to drop a hack that second guessed the (insufficient)
   information provided by the emulator, and just do the right thing.
 
 Documentation:
 
 * Various updates and fixes, mostly for x86
 
 * MTRR and PAT fixes and optimizations:
 -----BEGIN PGP SIGNATURE-----
 
 iQFIBAABCAAyFiEE8TM4V0tmI4mGbHaCv/vSX3jHroMFAmVBZc0UHHBib256aW5p
 QHJlZGhhdC5jb20ACgkQv/vSX3jHroP1LQf+NgsmZ1lkGQlKdSdijoQ856w+k0or
 l2SV1wUwiEdFPSGK+RTUlHV5Y1ni1dn/CqCVIJZKEI3ZtZ1m9/4HKIRXvbMwFHIH
 hx+E4Lnf8YUjsGjKTLd531UKcpphztZavQ6pXLEwazkSkDEra+JIKtooI8uU+9/p
 bd/eF1V+13a8CHQf1iNztFJVxqBJbVlnPx4cZDRQQvewskIDGnVDtwbrwCUKGtzD
 eNSzhY7si6O2kdQNkuA8xPhg29dYX9XLaCK2K1l8xOUm8WipLdtF86GAKJ5BVuOL
 6ek/2QCYjZ7a+coAZNfgSEUi8JmFHEqCo7cnKmWzPJp+2zyXsdudqAhT1g==
 =UIxm
 -----END PGP SIGNATURE-----

Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm

Pull kvm updates from Paolo Bonzini:
 "ARM:

   - Generalized infrastructure for 'writable' ID registers, effectively
     allowing userspace to opt-out of certain vCPU features for its
     guest

   - Optimization for vSGI injection, opportunistically compressing
     MPIDR to vCPU mapping into a table

   - Improvements to KVM's PMU emulation, allowing userspace to select
     the number of PMCs available to a VM

   - Guest support for memory operation instructions (FEAT_MOPS)

   - Cleanups to handling feature flags in KVM_ARM_VCPU_INIT, squashing
     bugs and getting rid of useless code

   - Changes to the way the SMCCC filter is constructed, avoiding wasted
     memory allocations when not in use

   - Load the stage-2 MMU context at vcpu_load() for VHE systems,
     reducing the overhead of errata mitigations

   - Miscellaneous kernel and selftest fixes

  LoongArch:

   - New architecture for kvm.

     The hardware uses the same model as x86, s390 and RISC-V, where
     guest/host mode is orthogonal to supervisor/user mode. The
     virtualization extensions are very similar to MIPS, therefore the
     code also has some similarities but it's been cleaned up to avoid
     some of the historical bogosities that are found in arch/mips. The
     kernel emulates MMU, timer and CSR accesses, while interrupt
     controllers are only emulated in userspace, at least for now.

  RISC-V:

   - Support for the Smstateen and Zicond extensions

   - Support for virtualizing senvcfg

   - Support for virtualized SBI debug console (DBCN)

  S390:

   - Nested page table management can be monitored through tracepoints
     and statistics

  x86:

   - Fix incorrect handling of VMX posted interrupt descriptor in
     KVM_SET_LAPIC, which could result in a dropped timer IRQ

   - Avoid WARN on systems with Intel IPI virtualization

   - Add CONFIG_KVM_MAX_NR_VCPUS, to allow supporting up to 4096 vCPUs
     without forcing more common use cases to eat the extra memory
     overhead.

   - Add virtualization support for AMD SRSO mitigation (IBPB_BRTYPE and
     SBPB, aka Selective Branch Predictor Barrier).

   - Fix a bug where restoring a vCPU snapshot that was taken within 1
     second of creating the original vCPU would cause KVM to try to
     synchronize the vCPU's TSC and thus clobber the correct TSC being
     set by userspace.

   - Compute guest wall clock using a single TSC read to avoid
     generating an inaccurate time, e.g. if the vCPU is preempted
     between multiple TSC reads.

   - "Virtualize" HWCR.TscFreqSel to make Linux guests happy, which
     complain about a "Firmware Bug" if the bit isn't set for select
     F/M/S combos. Likewise "virtualize" (ignore) MSR_AMD64_TW_CFG to
     appease Windows Server 2022.

   - Don't apply side effects to Hyper-V's synthetic timer on writes
     from userspace to fix an issue where the auto-enable behavior can
     trigger spurious interrupts, i.e. do auto-enabling only for guest
     writes.

   - Remove an unnecessary kick of all vCPUs when synchronizing the
     dirty log without PML enabled.

   - Advertise "support" for non-serializing FS/GS base MSR writes as
     appropriate.

   - Harden the fast page fault path to guard against encountering an
     invalid root when walking SPTEs.

   - Omit "struct kvm_vcpu_xen" entirely when CONFIG_KVM_XEN=n.

   - Use the fast path directly from the timer callback when delivering
     Xen timer events, instead of waiting for the next iteration of the
     run loop. This was not done so far because previously proposed code
     had races, but now care is taken to stop the hrtimer at critical
     points such as restarting the timer or saving the timer information
     for userspace.

   - Follow the lead of upstream Xen and ignore the VCPU_SSHOTTMR_future
     flag.

   - Optimize injection of PMU interrupts that are simultaneous with
     NMIs.

   - Usual handful of fixes for typos and other warts.

  x86 - MTRR/PAT fixes and optimizations:

   - Clean up code that deals with honoring guest MTRRs when the VM has
     non-coherent DMA and host MTRRs are ignored, i.e. EPT is enabled.

   - Zap EPT entries when non-coherent DMA assignment stops/start to
     prevent using stale entries with the wrong memtype.

   - Don't ignore guest PAT for CR0.CD=1 && KVM_X86_QUIRK_CD_NW_CLEARED=y

     This was done as a workaround for virtual machine BIOSes that did
     not bother to clear CR0.CD (because ancient KVM/QEMU did not bother
     to set it, in turn), and there's zero reason to extend the quirk to
     also ignore guest PAT.

  x86 - SEV fixes:

   - Report KVM_EXIT_SHUTDOWN instead of EINVAL if KVM intercepts
     SHUTDOWN while running an SEV-ES guest.

   - Clean up the recognition of emulation failures on SEV guests, when
     KVM would like to "skip" the instruction but it had already been
     partially emulated. This makes it possible to drop a hack that
     second guessed the (insufficient) information provided by the
     emulator, and just do the right thing.

  Documentation:

   - Various updates and fixes, mostly for x86

   - MTRR and PAT fixes and optimizations"

* tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (164 commits)
  KVM: selftests: Avoid using forced target for generating arm64 headers
  tools headers arm64: Fix references to top srcdir in Makefile
  KVM: arm64: Add tracepoint for MMIO accesses where ISV==0
  KVM: arm64: selftest: Perform ISB before reading PAR_EL1
  KVM: arm64: selftest: Add the missing .guest_prepare()
  KVM: arm64: Always invalidate TLB for stage-2 permission faults
  KVM: x86: Service NMI requests after PMI requests in VM-Enter path
  KVM: arm64: Handle AArch32 SPSR_{irq,abt,und,fiq} as RAZ/WI
  KVM: arm64: Do not let a L1 hypervisor access the *32_EL2 sysregs
  KVM: arm64: Refine _EL2 system register list that require trap reinjection
  arm64: Add missing _EL2 encodings
  arm64: Add missing _EL12 encodings
  KVM: selftests: aarch64: vPMU test for validating user accesses
  KVM: selftests: aarch64: vPMU register test for unimplemented counters
  KVM: selftests: aarch64: vPMU register test for implemented counters
  KVM: selftests: aarch64: Introduce vpmu_counter_access test
  tools: Import arm_pmuv3.h
  KVM: arm64: PMU: Allow userspace to limit PMCR_EL0.N for the guest
  KVM: arm64: Sanitize PM{C,I}NTEN{SET,CLR}, PMOVS{SET,CLR} before first run
  KVM: arm64: Add {get,set}_user for PM{C,I}NTEN{SET,CLR}, PMOVS{SET,CLR}
  ...
This commit is contained in:
Linus Torvalds 2023-11-02 15:45:15 -10:00
parent 5be9911406 45b890f768
commit 6803bd7956
127 changed files with 8890 additions and 1508 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
Documentation/devicetree/bindings/riscv/extensions.yaml
View File

@ -128,6 +128,12 @@ properties:
changes to interrupts as frozen at commit ccbddab ("Merge pull
request #42 from riscv/jhauser-2023-RC4") of riscv-aia.
- const: smstateen
description: |
The standard Smstateen extension for controlling access to CSRs
added by other RISC-V extensions in H/S/VS/U/VU modes and as
ratified at commit a28bfae (Ratified (#7)) of riscv-state-enable.
- const: ssaia
description: |
The standard Ssaia supervisor-level extension for the advanced
@ -212,6 +218,12 @@ properties:
ratified in the 20191213 version of the unprivileged ISA
specification.
- const: zicond
description:
The standard Zicond extension for conditional arithmetic and
conditional-select/move operations as ratified in commit 95cf1f9
("Add changes requested by Ved during signoff") of riscv-zicond.
- const: zicsr
description: |
The standard Zicsr extension for control and status register

158
Documentation/virt/kvm/api.rst
View File

@ -416,6 +416,13 @@ Reads the general purpose registers from the vcpu.
__u64 pc;
};
/* LoongArch */
struct kvm_regs {
/* out (KVM_GET_REGS) / in (KVM_SET_REGS) */
unsigned long gpr[32];
unsigned long pc;
};
4.12 KVM_SET_REGS
-----------------
@ -506,7 +513,7 @@ translation mode.
------------------
:Capability: basic
:Architectures: x86, ppc, mips, riscv
:Architectures: x86, ppc, mips, riscv, loongarch
:Type: vcpu ioctl
:Parameters: struct kvm_interrupt (in)
:Returns: 0 on success, negative on failure.
@ -540,7 +547,7 @@ ioctl is useful if the in-kernel PIC is not used.
PPC:
^^^^
Queues an external interrupt to be injected. This ioctl is overleaded
Queues an external interrupt to be injected. This ioctl is overloaded
with 3 different irq values:
a) KVM_INTERRUPT_SET
@ -592,6 +599,14 @@ b) KVM_INTERRUPT_UNSET
This is an asynchronous vcpu ioctl and can be invoked from any thread.
LOONGARCH:
^^^^^^^^^^
Queues an external interrupt to be injected into the virtual CPU. A negative
interrupt number dequeues the interrupt.
This is an asynchronous vcpu ioctl and can be invoked from any thread.
4.17 KVM_DEBUG_GUEST
--------------------
@ -737,7 +752,7 @@ signal mask.
----------------
:Capability: basic
:Architectures: x86
:Architectures: x86, loongarch
:Type: vcpu ioctl
:Parameters: struct kvm_fpu (out)
:Returns: 0 on success, -1 on error
@ -746,7 +761,7 @@ Reads the floating point state from the vcpu.
::
/* for KVM_GET_FPU and KVM_SET_FPU */
/* x86: for KVM_GET_FPU and KVM_SET_FPU */
struct kvm_fpu {
__u8 fpr[8][16];
__u16 fcw;
@ -761,12 +776,21 @@ Reads the floating point state from the vcpu.
__u32 pad2;
};
/* LoongArch: for KVM_GET_FPU and KVM_SET_FPU */
struct kvm_fpu {
__u32 fcsr;
__u64 fcc;
struct kvm_fpureg {
__u64 val64[4];
}fpr[32];
};
4.23 KVM_SET_FPU
----------------
:Capability: basic
:Architectures: x86
:Architectures: x86, loongarch
:Type: vcpu ioctl
:Parameters: struct kvm_fpu (in)
:Returns: 0 on success, -1 on error
@ -775,7 +799,7 @@ Writes the floating point state to the vcpu.
::
/* for KVM_GET_FPU and KVM_SET_FPU */
/* x86: for KVM_GET_FPU and KVM_SET_FPU */
struct kvm_fpu {
__u8 fpr[8][16];
__u16 fcw;
@ -790,6 +814,15 @@ Writes the floating point state to the vcpu.
__u32 pad2;
};
/* LoongArch: for KVM_GET_FPU and KVM_SET_FPU */
struct kvm_fpu {
__u32 fcsr;
__u64 fcc;
struct kvm_fpureg {
__u64 val64[4];
}fpr[32];
};
4.24 KVM_CREATE_IRQCHIP
-----------------------
@ -965,7 +998,7 @@ be set in the flags field of this ioctl:
The KVM_XEN_HVM_CONFIG_INTERCEPT_HCALL flag requests KVM to generate
the contents of the hypercall page automatically; hypercalls will be
intercepted and passed to userspace through KVM_EXIT_XEN. In this
ase, all of the blob size and address fields must be zero.
case, all of the blob size and address fields must be zero.
The KVM_XEN_HVM_CONFIG_EVTCHN_SEND flag indicates to KVM that userspace
will always use the KVM_XEN_HVM_EVTCHN_SEND ioctl to deliver event
@ -1070,7 +1103,7 @@ Other flags returned by ``KVM_GET_CLOCK`` are accepted but ignored.
:Extended by: KVM_CAP_INTR_SHADOW
:Architectures: x86, arm64
:Type: vcpu ioctl
:Parameters: struct kvm_vcpu_event (out)
:Parameters: struct kvm_vcpu_events (out)
:Returns: 0 on success, -1 on error
X86:
@ -1193,7 +1226,7 @@ directly to the virtual CPU).
:Extended by: KVM_CAP_INTR_SHADOW
:Architectures: x86, arm64
:Type: vcpu ioctl
:Parameters: struct kvm_vcpu_event (in)
:Parameters: struct kvm_vcpu_events (in)
:Returns: 0 on success, -1 on error
X86:
@ -1387,7 +1420,7 @@ documentation when it pops into existence).
-------------------
:Capability: KVM_CAP_ENABLE_CAP
:Architectures: mips, ppc, s390, x86
:Architectures: mips, ppc, s390, x86, loongarch
:Type: vcpu ioctl
:Parameters: struct kvm_enable_cap (in)
:Returns: 0 on success; -1 on error
@ -1442,7 +1475,7 @@ for vm-wide capabilities.
---------------------
:Capability: KVM_CAP_MP_STATE
:Architectures: x86, s390, arm64, riscv
:Architectures: x86, s390, arm64, riscv, loongarch
:Type: vcpu ioctl
:Parameters: struct kvm_mp_state (out)
:Returns: 0 on success; -1 on error
@ -1460,7 +1493,7 @@ Possible values are:
========================== ===============================================
KVM_MP_STATE_RUNNABLE the vcpu is currently running
[x86,arm64,riscv]
[x86,arm64,riscv,loongarch]
KVM_MP_STATE_UNINITIALIZED the vcpu is an application processor (AP)
which has not yet received an INIT signal [x86]
KVM_MP_STATE_INIT_RECEIVED the vcpu has received an INIT signal, and is
@ -1516,11 +1549,14 @@ For riscv:
The only states that are valid are KVM_MP_STATE_STOPPED and
KVM_MP_STATE_RUNNABLE which reflect if the vcpu is paused or not.
On LoongArch, only the KVM_MP_STATE_RUNNABLE state is used to reflect
whether the vcpu is runnable.
4.39 KVM_SET_MP_STATE
---------------------
:Capability: KVM_CAP_MP_STATE
:Architectures: x86, s390, arm64, riscv
:Architectures: x86, s390, arm64, riscv, loongarch
:Type: vcpu ioctl
:Parameters: struct kvm_mp_state (in)
:Returns: 0 on success; -1 on error
@ -1538,6 +1574,9 @@ For arm64/riscv:
The only states that are valid are KVM_MP_STATE_STOPPED and
KVM_MP_STATE_RUNNABLE which reflect if the vcpu should be paused or not.
On LoongArch, only the KVM_MP_STATE_RUNNABLE state is used to reflect
whether the vcpu is runnable.
4.40 KVM_SET_IDENTITY_MAP_ADDR
------------------------------
@ -2841,6 +2880,19 @@ Following are the RISC-V D-extension registers:
0x8020 0000 0600 0020 fcsr Floating point control and status register
======================= ========= =============================================
LoongArch registers are mapped using the lower 32 bits. The upper 16 bits of
that is the register group type.
LoongArch csr registers are used to control guest cpu or get status of guest
cpu, and they have the following id bit patterns::
0x9030 0000 0001 00 <reg:5> <sel:3> (64-bit)
LoongArch KVM control registers are used to implement some new defined functions
such as set vcpu counter or reset vcpu, and they have the following id bit patterns::
0x9030 0000 0002 <reg:16>
4.69 KVM_GET_ONE_REG
--------------------
@ -3063,7 +3115,7 @@ as follow::
};
An entry with a "page_shift" of 0 is unused. Because the array is
organized in increasing order, a lookup can stop when encoutering
organized in increasing order, a lookup can stop when encountering
such an entry.
The "slb_enc" field provides the encoding to use in the SLB for the
@ -3370,6 +3422,8 @@ return indicates the attribute is implemented. It does not necessarily
indicate that the attribute can be read or written in the device's
current state. "addr" is ignored.
.. _KVM_ARM_VCPU_INIT:
4.82 KVM_ARM_VCPU_INIT
----------------------
@ -3455,7 +3509,7 @@ Possible features:
- KVM_RUN and KVM_GET_REG_LIST are not available;
- KVM_GET_ONE_REG and KVM_SET_ONE_REG cannot be used to access
the scalable archietctural SVE registers
the scalable architectural SVE registers
KVM_REG_ARM64_SVE_ZREG(), KVM_REG_ARM64_SVE_PREG() or
KVM_REG_ARM64_SVE_FFR;
@ -4401,7 +4455,7 @@ This will have undefined effects on the guest if it has not already
placed itself in a quiescent state where no vcpu will make MMU enabled
memory accesses.
On succsful completion, the pending HPT will become the guest's active
On successful completion, the pending HPT will become the guest's active
HPT and the previous HPT will be discarded.
On failure, the guest will still be operating on its previous HPT.
@ -5016,7 +5070,7 @@ before the vcpu is fully usable.
Between KVM_ARM_VCPU_INIT and KVM_ARM_VCPU_FINALIZE, the feature may be
configured by use of ioctls such as KVM_SET_ONE_REG. The exact configuration
that should be performaned and how to do it are feature-dependent.
that should be performed and how to do it are feature-dependent.
Other calls that depend on a particular feature being finalized, such as
KVM_RUN, KVM_GET_REG_LIST, KVM_GET_ONE_REG and KVM_SET_ONE_REG, will fail with
@ -5124,6 +5178,24 @@ Valid values for 'action'::
#define KVM_PMU_EVENT_ALLOW 0
#define KVM_PMU_EVENT_DENY 1
Via this API, KVM userspace can also control the behavior of the VM's fixed
counters (if any) by configuring the "action" and "fixed_counter_bitmap" fields.
Specifically, KVM follows the following pseudo-code when determining whether to
allow the guest FixCtr[i] to count its pre-defined fixed event::
FixCtr[i]_is_allowed = (action == ALLOW) && (bitmap & BIT(i)) ||
(action == DENY) && !(bitmap & BIT(i));
FixCtr[i]_is_denied = !FixCtr[i]_is_allowed;
KVM always consumes fixed_counter_bitmap, it's userspace's responsibility to
ensure fixed_counter_bitmap is set correctly, e.g. if userspace wants to define
a filter that only affects general purpose counters.
Note, the "events" field also applies to fixed counters' hardcoded event_select
and unit_mask values. "fixed_counter_bitmap" has higher priority than "events"
if there is a contradiction between the two.
4.121 KVM_PPC_SVM_OFF
---------------------
@ -5475,7 +5547,7 @@ KVM_XEN_ATTR_TYPE_EVTCHN
from the guest. A given sending port number may be directed back to
a specified vCPU (by APIC ID) / port / priority on the guest, or to
trigger events on an eventfd. The vCPU and priority can be changed
by setting KVM_XEN_EVTCHN_UPDATE in a subsequent call, but but other
by setting KVM_XEN_EVTCHN_UPDATE in a subsequent call, but other
fields cannot change for a given sending port. A port mapping is
removed by using KVM_XEN_EVTCHN_DEASSIGN in the flags field. Passing
KVM_XEN_EVTCHN_RESET in the flags field removes all interception of
@ -6070,6 +6142,56 @@ writes to the CNTVCT_EL0 and CNTPCT_EL0 registers using the SET_ONE_REG
interface. No error will be returned, but the resulting offset will not be
applied.
.. _KVM_ARM_GET_REG_WRITABLE_MASKS:
4.139 KVM_ARM_GET_REG_WRITABLE_MASKS
-------------------------------------------
:Capability: KVM_CAP_ARM_SUPPORTED_REG_MASK_RANGES
:Architectures: arm64
:Type: vm ioctl
:Parameters: struct reg_mask_range (in/out)
:Returns: 0 on success, < 0 on error
::
#define KVM_ARM_FEATURE_ID_RANGE 0
#define KVM_ARM_FEATURE_ID_RANGE_SIZE (3 * 8 * 8)
struct reg_mask_range {
__u64 addr; /* Pointer to mask array */
__u32 range; /* Requested range */
__u32 reserved[13];
};
This ioctl copies the writable masks for a selected range of registers to
userspace.
The ``addr`` field is a pointer to the destination array where KVM copies
the writable masks.
The ``range`` field indicates the requested range of registers.
``KVM_CHECK_EXTENSION`` for the ``KVM_CAP_ARM_SUPPORTED_REG_MASK_RANGES``
capability returns the supported ranges, expressed as a set of flags. Each
flag's bit index represents a possible value for the ``range`` field.
All other values are reserved for future use and KVM may return an error.
The ``reserved[13]`` array is reserved for future use and should be 0, or
KVM may return an error.
KVM_ARM_FEATURE_ID_RANGE (0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The Feature ID range is defined as the AArch64 System register space with
op0==3, op1=={0, 1, 3}, CRn==0, CRm=={0-7}, op2=={0-7}.
The mask returned array pointed to by ``addr`` is indexed by the macro
``ARM64_FEATURE_ID_RANGE_IDX(op0, op1, crn, crm, op2)``, allowing userspace
to know what fields can be changed for the system register described by
``op0, op1, crn, crm, op2``. KVM rejects ID register values that describe a
superset of the features supported by the system.
5. The kvm_run structure
========================

1
Documentation/virt/kvm/arm/index.rst
View File

@ -11,3 +11,4 @@ ARM
hypercalls
pvtime
ptp_kvm
vcpu-features

48
Documentation/virt/kvm/arm/vcpu-features.rst Normal file
View File

@ -0,0 +1,48 @@
.. SPDX-License-Identifier: GPL-2.0
===============================
vCPU feature selection on arm64
===============================
KVM/arm64 provides two mechanisms that allow userspace to configure
the CPU features presented to the guest.
KVM_ARM_VCPU_INIT
=================
The ``KVM_ARM_VCPU_INIT`` ioctl accepts a bitmap of feature flags
(``struct kvm_vcpu_init::features``). Features enabled by this interface are
*opt-in* and may change/extend UAPI. See :ref:`KVM_ARM_VCPU_INIT` for complete
documentation of the features controlled by the ioctl.
Otherwise, all CPU features supported by KVM are described by the architected
ID registers.
The ID Registers
================
The Arm architecture specifies a range of *ID Registers* that describe the set
of architectural features supported by the CPU implementation. KVM initializes
the guest's ID registers to the maximum set of CPU features supported by the
system. The ID register values may be VM-scoped in KVM, meaning that the
values could be shared for all vCPUs in a VM.
KVM allows userspace to *opt-out* of certain CPU features described by the ID
registers by writing values to them via the ``KVM_SET_ONE_REG`` ioctl. The ID
registers are mutable until the VM has started, i.e. userspace has called
``KVM_RUN`` on at least one vCPU in the VM. Userspace can discover what fields
are mutable in the ID registers using the ``KVM_ARM_GET_REG_WRITABLE_MASKS``.
See the :ref:`ioctl documentation <KVM_ARM_GET_REG_WRITABLE_MASKS>` for more
details.
Userspace is allowed to *limit* or *mask* CPU features according to the rules
outlined by the architecture in DDI0487J.a D19.1.3 'Principles of the ID
scheme for fields in ID register'. KVM does not allow ID register values that
exceed the capabilities of the system.
.. warning::
It is **strongly recommended** that userspace modify the ID register values
before accessing the rest of the vCPU's CPU register state. KVM may use the
ID register values to control feature emulation. Interleaving ID register
modification with other system register accesses may lead to unpredictable
behavior.

7
Documentation/virt/kvm/devices/arm-vgic-v3.rst
View File

@ -59,6 +59,13 @@ Groups:
It is invalid to mix calls with KVM_VGIC_V3_ADDR_TYPE_REDIST and
KVM_VGIC_V3_ADDR_TYPE_REDIST_REGION attributes.
Note that to obtain reproducible results (the same VCPU being associated
with the same redistributor across a save/restore operation), VCPU creation
order, redistributor region creation order as well as the respective
interleaves of VCPU and region creation MUST be preserved. Any change in
either ordering may result in a different vcpu_id/redistributor association,
resulting in a VM that will fail to run at restore time.
Errors:
======= =============================================================

43
Documentation/virt/kvm/x86/mmu.rst
View File

@ -202,10 +202,22 @@ Shadow pages contain the following information:
Is 1 if the MMU instance cannot use A/D bits. EPT did not have A/D
bits before Haswell; shadow EPT page tables also cannot use A/D bits
if the L1 hypervisor does not enable them.
role.guest_mode:
Indicates the shadow page is created for a nested guest.
role.passthrough:
The page is not backed by a guest page table, but its first entry
points to one. This is set if NPT uses 5-level page tables (host
CR4.LA57=1) and is shadowing L1's 4-level NPT (L1 CR4.LA57=0).
mmu_valid_gen:
The MMU generation of this page, used to fast zap of all MMU pages within a
VM without blocking vCPUs too long. Specifically, KVM updates the per-VM
valid MMU generation which causes the mismatch of mmu_valid_gen for each mmu
page. This makes all existing MMU pages obsolete. Obsolete pages can't be
used. Therefore, vCPUs must load a new, valid root before re-entering the
guest. The MMU generation is only ever '0' or '1'. Note, the TDP MMU doesn't
use this field as non-root TDP MMU pages are reachable only from their
owning root. Thus it suffices for TDP MMU to use role.invalid in root pages
to invalidate all MMU pages.
gfn:
Either the guest page table containing the translations shadowed by this
page, or the base page frame for linear translations. See role.direct.
@ -219,21 +231,30 @@ Shadow pages contain the following information:
at __pa(sp2->spt). sp2 will point back at sp1 through parent_pte.
The spt array forms a DAG structure with the shadow page as a node, and
guest pages as leaves.
gfns:
An array of 512 guest frame numbers, one for each present pte. Used to
perform a reverse map from a pte to a gfn. When role.direct is set, any
element of this array can be calculated from the gfn field when used, in
this case, the array of gfns is not allocated. See role.direct and gfn.
root_count:
A counter keeping track of how many hardware registers (guest cr3 or
pdptrs) are now pointing at the page. While this counter is nonzero, the
page cannot be destroyed. See role.invalid.
shadowed_translation:
An array of 512 shadow translation entries, one for each present pte. Used
to perform a reverse map from a pte to a gfn as well as its access
permission. When role.direct is set, the shadow_translation array is not
allocated. This is because the gfn contained in any element of this array
can be calculated from the gfn field when used. In addition, when
role.direct is set, KVM does not track access permission for each of the
gfn. See role.direct and gfn.
root_count / tdp_mmu_root_count:
root_count is a reference counter for root shadow pages in Shadow MMU.
vCPUs elevate the refcount when getting a shadow page that will be used as
a root page, i.e. page that will be loaded into hardware directly (CR3,
PDPTRs, nCR3 EPTP). Root pages cannot be destroyed while their refcount is
non-zero. See role.invalid. tdp_mmu_root_count is similar but exclusively
used in TDP MMU as an atomic refcount.
parent_ptes:
The reverse mapping for the pte/ptes pointing at this page's spt. If
parent_ptes bit 0 is zero, only one spte points at this page and
parent_ptes points at this single spte, otherwise, there exists multiple
sptes pointing at this page and (parent_ptes & ~0x1) points at a data
structure with a list of parent sptes.
ptep:
The kernel virtual address of the SPTE that points at this shadow page.
Used exclusively by the TDP MMU, this field is a union with parent_ptes.
unsync:
If true, then the translations in this page may not match the guest's
translation. This is equivalent to the state of the tlb when a pte is
@ -261,6 +282,10 @@ Shadow pages contain the following information:
since the last time the page table was actually used; if emulation
is triggered too frequently on this page, KVM will unmap the page
to avoid emulation in the future.
tdp_mmu_page:
Is 1 if the shadow page is a TDP MMU page. This variable is used to
bifurcate the control flows for KVM when walking any data structure that
may contain pages from both TDP MMU and shadow MMU.
Reverse map
===========

13
MAINTAINERS
View File

@ -11604,6 +11604,18 @@ F: include/kvm/arm_*
F: tools/testing/selftests/kvm/*/aarch64/
F: tools/testing/selftests/kvm/aarch64/
KERNEL VIRTUAL MACHINE FOR LOONGARCH (KVM/LoongArch)
M: Tianrui Zhao <zhaotianrui@loongson.cn>
M: Bibo Mao <maobibo@loongson.cn>
M: Huacai Chen <chenhuacai@kernel.org>
L: kvm@vger.kernel.org
L: loongarch@lists.linux.dev
S: Maintained
T: git git://git.kernel.org/pub/scm/virt/kvm/kvm.git
F: arch/loongarch/include/asm/kvm*
F: arch/loongarch/include/uapi/asm/kvm*
F: arch/loongarch/kvm/
KERNEL VIRTUAL MACHINE FOR MIPS (KVM/mips)
M: Huacai Chen <chenhuacai@kernel.org>
L: linux-mips@vger.kernel.org
@ -11640,6 +11652,7 @@ F: arch/riscv/include/asm/kvm*
F: arch/riscv/include/uapi/asm/kvm*
F: arch/riscv/kvm/
F: tools/testing/selftests/kvm/*/riscv/
F: tools/testing/selftests/kvm/riscv/
KERNEL VIRTUAL MACHINE for s390 (KVM/s390)
M: Christian Borntraeger <borntraeger@linux.ibm.com>

4
arch/arm64/include/asm/kvm_arm.h
View File

@ -102,7 +102,9 @@
#define HCR_HOST_NVHE_PROTECTED_FLAGS (HCR_HOST_NVHE_FLAGS | HCR_TSC)
#define HCR_HOST_VHE_FLAGS (HCR_RW | HCR_TGE | HCR_E2H)
#define HCRX_GUEST_FLAGS (HCRX_EL2_SMPME | HCRX_EL2_TCR2En)
#define HCRX_GUEST_FLAGS \
(HCRX_EL2_SMPME | HCRX_EL2_TCR2En | \
(cpus_have_final_cap(ARM64_HAS_MOPS) ? (HCRX_EL2_MSCEn | HCRX_EL2_MCE2) : 0))
#define HCRX_HOST_FLAGS (HCRX_EL2_MSCEn | HCRX_EL2_TCR2En)
/* TCR_EL2 Registers bits */

15
arch/arm64/include/asm/kvm_emulate.h
View File

@ -54,6 +54,11 @@ void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu);
int kvm_inject_nested_sync(struct kvm_vcpu *vcpu, u64 esr_el2);
int kvm_inject_nested_irq(struct kvm_vcpu *vcpu);
static inline bool vcpu_has_feature(const struct kvm_vcpu *vcpu, int feature)
{
return test_bit(feature, vcpu->kvm->arch.vcpu_features);
}
#if defined(__KVM_VHE_HYPERVISOR__) || defined(__KVM_NVHE_HYPERVISOR__)
static __always_inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu)
{
@ -62,7 +67,7 @@ static __always_inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu)
#else
static __always_inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu)
{
return test_bit(KVM_ARM_VCPU_EL1_32BIT, vcpu->arch.features);
return vcpu_has_feature(vcpu, KVM_ARM_VCPU_EL1_32BIT);
}
#endif
@ -465,7 +470,7 @@ static inline bool kvm_is_write_fault(struct kvm_vcpu *vcpu)
static inline unsigned long kvm_vcpu_get_mpidr_aff(struct kvm_vcpu *vcpu)
{
return vcpu_read_sys_reg(vcpu, MPIDR_EL1) & MPIDR_HWID_BITMASK;
return __vcpu_sys_reg(vcpu, MPIDR_EL1) & MPIDR_HWID_BITMASK;
}
static inline void kvm_vcpu_set_be(struct kvm_vcpu *vcpu)
@ -565,12 +570,6 @@ static __always_inline void kvm_incr_pc(struct kvm_vcpu *vcpu)
vcpu_set_flag((v), e); \
} while (0)
static inline bool vcpu_has_feature(struct kvm_vcpu *vcpu, int feature)
{
return test_bit(feature, vcpu->arch.features);
}
static __always_inline void kvm_write_cptr_el2(u64 val)
{
if (has_vhe() || has_hvhe())

61
arch/arm64/include/asm/kvm_host.h
View File

@ -78,7 +78,7 @@ extern unsigned int __ro_after_init kvm_sve_max_vl;
int __init kvm_arm_init_sve(void);
u32 __attribute_const__ kvm_target_cpu(void);
int kvm_reset_vcpu(struct kvm_vcpu *vcpu);
void kvm_reset_vcpu(struct kvm_vcpu *vcpu);
void kvm_arm_vcpu_destroy(struct kvm_vcpu *vcpu);
struct kvm_hyp_memcache {
@ -158,6 +158,16 @@ struct kvm_s2_mmu {
phys_addr_t pgd_phys;
struct kvm_pgtable *pgt;
/*
* VTCR value used on the host. For a non-NV guest (or a NV
* guest that runs in a context where its own S2 doesn't
* apply), its T0SZ value reflects that of the IPA size.
*
* For a shadow S2 MMU, T0SZ reflects the PARange exposed to
* the guest.
*/
u64 vtcr;
/* The last vcpu id that ran on each physical CPU */
int __percpu *last_vcpu_ran;
@ -202,12 +212,34 @@ struct kvm_protected_vm {
struct kvm_hyp_memcache teardown_mc;
};
struct kvm_mpidr_data {
u64 mpidr_mask;
DECLARE_FLEX_ARRAY(u16, cmpidr_to_idx);
};
static inline u16 kvm_mpidr_index(struct kvm_mpidr_data *data, u64 mpidr)
{
unsigned long mask = data->mpidr_mask;
u64 aff = mpidr & MPIDR_HWID_BITMASK;
int nbits, bit, bit_idx = 0;
u16 index = 0;
/*
* If this looks like RISC-V's BEXT or x86's PEXT
* instructions, it isn't by accident.
*/
nbits = fls(mask);
for_each_set_bit(bit, &mask, nbits) {
index |= (aff & BIT(bit)) >> (bit - bit_idx);
bit_idx++;
}
return index;
}
struct kvm_arch {
struct kvm_s2_mmu mmu;
/* VTCR_EL2 value for this VM */
u64 vtcr;
/* Interrupt controller */
struct vgic_dist vgic;
@ -239,15 +271,16 @@ struct kvm_arch {
#define KVM_ARCH_FLAG_VM_COUNTER_OFFSET 5
/* Timer PPIs made immutable */
#define KVM_ARCH_FLAG_TIMER_PPIS_IMMUTABLE 6
/* SMCCC filter initialized for the VM */
#define KVM_ARCH_FLAG_SMCCC_FILTER_CONFIGURED 7
/* Initial ID reg values loaded */
#define KVM_ARCH_FLAG_ID_REGS_INITIALIZED 8
#define KVM_ARCH_FLAG_ID_REGS_INITIALIZED 7
unsigned long flags;
/* VM-wide vCPU feature set */
DECLARE_BITMAP(vcpu_features, KVM_VCPU_MAX_FEATURES);
/* MPIDR to vcpu index mapping, optional */
struct kvm_mpidr_data *mpidr_data;
/*
* VM-wide PMU filter, implemented as a bitmap and big enough for
* up to 2^10 events (ARMv8.0) or 2^16 events (ARMv8.1+).
@ -257,6 +290,9 @@ struct kvm_arch {
cpumask_var_t supported_cpus;
/* PMCR_EL0.N value for the guest */
u8 pmcr_n;
/* Hypercall features firmware registers' descriptor */
struct kvm_smccc_features smccc_feat;
struct maple_tree smccc_filter;
@ -574,9 +610,6 @@ struct kvm_vcpu_arch {
/* Cache some mmu pages needed inside spinlock regions */
struct kvm_mmu_memory_cache mmu_page_cache;
/* feature flags */
DECLARE_BITMAP(features, KVM_VCPU_MAX_FEATURES);
/* Virtual SError ESR to restore when HCR_EL2.VSE is set */
u64 vsesr_el2;
@ -1025,7 +1058,7 @@ int kvm_arm_pvtime_has_attr(struct kvm_vcpu *vcpu,
extern unsigned int __ro_after_init kvm_arm_vmid_bits;
int __init kvm_arm_vmid_alloc_init(void);
void __init kvm_arm_vmid_alloc_free(void);
void kvm_arm_vmid_update(struct kvm_vmid *kvm_vmid);
bool kvm_arm_vmid_update(struct kvm_vmid *kvm_vmid);
void kvm_arm_vmid_clear_active(void);
static inline void kvm_arm_pvtime_vcpu_init(struct kvm_vcpu_arch *vcpu_arch)
@ -1078,6 +1111,8 @@ int kvm_vm_ioctl_mte_copy_tags(struct kvm *kvm,
struct kvm_arm_copy_mte_tags *copy_tags);
int kvm_vm_ioctl_set_counter_offset(struct kvm *kvm,
struct kvm_arm_counter_offset *offset);
int kvm_vm_ioctl_get_reg_writable_masks(struct kvm *kvm,
struct reg_mask_range *range);
/* Guest/host FPSIMD coordination helpers */
int kvm_arch_vcpu_run_map_fp(struct kvm_vcpu *vcpu);
@ -1109,8 +1144,8 @@ static inline bool kvm_set_pmuserenr(u64 val)
}
#endif
void kvm_vcpu_load_sysregs_vhe(struct kvm_vcpu *vcpu);
void kvm_vcpu_put_sysregs_vhe(struct kvm_vcpu *vcpu);
void kvm_vcpu_load_vhe(struct kvm_vcpu *vcpu);
void kvm_vcpu_put_vhe(struct kvm_vcpu *vcpu);
int __init kvm_set_ipa_limit(void);

7
arch/arm64/include/asm/kvm_hyp.h
View File

@ -93,6 +93,8 @@ void __timer_disable_traps(struct kvm_vcpu *vcpu);
void __sysreg_save_state_nvhe(struct kvm_cpu_context *ctxt);
void __sysreg_restore_state_nvhe(struct kvm_cpu_context *ctxt);
#else
void __vcpu_load_switch_sysregs(struct kvm_vcpu *vcpu);
void __vcpu_put_switch_sysregs(struct kvm_vcpu *vcpu);
void sysreg_save_host_state_vhe(struct kvm_cpu_context *ctxt);
void sysreg_restore_host_state_vhe(struct kvm_cpu_context *ctxt);
void sysreg_save_guest_state_vhe(struct kvm_cpu_context *ctxt);
@ -111,11 +113,6 @@ void __fpsimd_save_state(struct user_fpsimd_state *fp_regs);
void __fpsimd_restore_state(struct user_fpsimd_state *fp_regs);
void __sve_restore_state(void *sve_pffr, u32 *fpsr);
#ifndef __KVM_NVHE_HYPERVISOR__
void activate_traps_vhe_load(struct kvm_vcpu *vcpu);
void deactivate_traps_vhe_put(struct kvm_vcpu *vcpu);
#endif
u64 __guest_enter(struct kvm_vcpu *vcpu);
bool kvm_host_psci_handler(struct kvm_cpu_context *host_ctxt, u32 func_id);

45
arch/arm64/include/asm/kvm_mmu.h
View File

@ -150,9 +150,9 @@ static __always_inline unsigned long __kern_hyp_va(unsigned long v)
*/
#define KVM_PHYS_SHIFT (40)
#define kvm_phys_shift(kvm) VTCR_EL2_IPA(kvm->arch.vtcr)
#define kvm_phys_size(kvm) (_AC(1, ULL) << kvm_phys_shift(kvm))
#define kvm_phys_mask(kvm) (kvm_phys_size(kvm) - _AC(1, ULL))
#define kvm_phys_shift(mmu) VTCR_EL2_IPA((mmu)->vtcr)
#define kvm_phys_size(mmu) (_AC(1, ULL) << kvm_phys_shift(mmu))
#define kvm_phys_mask(mmu) (kvm_phys_size(mmu) - _AC(1, ULL))
#include <asm/kvm_pgtable.h>
#include <asm/stage2_pgtable.h>
@ -224,16 +224,41 @@ static inline void __clean_dcache_guest_page(void *va, size_t size)
kvm_flush_dcache_to_poc(va, size);
}
static inline size_t __invalidate_icache_max_range(void)
{
u8 iminline;
u64 ctr;
asm volatile(ALTERNATIVE_CB("movz %0, #0\n"
"movk %0, #0, lsl #16\n"
"movk %0, #0, lsl #32\n"
"movk %0, #0, lsl #48\n",
ARM64_ALWAYS_SYSTEM,
kvm_compute_final_ctr_el0)
: "=r" (ctr));
iminline = SYS_FIELD_GET(CTR_EL0, IminLine, ctr) + 2;
return MAX_DVM_OPS << iminline;
}
static inline void __invalidate_icache_guest_page(void *va, size_t size)
{
if (icache_is_aliasing()) {
/* any kind of VIPT cache */
/*
* VPIPT I-cache maintenance must be done from EL2. See comment in the
* nVHE flavor of __kvm_tlb_flush_vmid_ipa().
*/
if (icache_is_vpipt() && read_sysreg(CurrentEL) != CurrentEL_EL2)
return;
/*
* Blow the whole I-cache if it is aliasing (i.e. VIPT) or the
* invalidation range exceeds our arbitrary limit on invadations by
* cache line.
*/
if (icache_is_aliasing() || size > __invalidate_icache_max_range())
icache_inval_all_pou();
} else if (read_sysreg(CurrentEL) != CurrentEL_EL1 ||
!icache_is_vpipt()) {
/* PIPT or VPIPT at EL2 (see comment in __kvm_tlb_flush_vmid_ipa) */
else
icache_inval_pou((unsigned long)va, (unsigned long)va + size);
}
}
void kvm_set_way_flush(struct kvm_vcpu *vcpu);
@ -299,7 +324,7 @@ static __always_inline u64 kvm_get_vttbr(struct kvm_s2_mmu *mmu)
static __always_inline void __load_stage2(struct kvm_s2_mmu *mmu,
struct kvm_arch *arch)
{
write_sysreg(arch->vtcr, vtcr_el2);
write_sysreg(mmu->vtcr, vtcr_el2);
write_sysreg(kvm_get_vttbr(mmu), vttbr_el2);
/*

3
arch/arm64/include/asm/kvm_nested.h
View File

@ -2,13 +2,14 @@
#ifndef __ARM64_KVM_NESTED_H
#define __ARM64_KVM_NESTED_H
#include <asm/kvm_emulate.h>
#include <linux/kvm_host.h>
static inline bool vcpu_has_nv(const struct kvm_vcpu *vcpu)
{
return (!__is_defined(__KVM_NVHE_HYPERVISOR__) &&
cpus_have_final_cap(ARM64_HAS_NESTED_VIRT) &&
test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features));
vcpu_has_feature(vcpu, KVM_ARM_VCPU_HAS_EL2));
}
extern bool __check_nv_sr_forward(struct kvm_vcpu *vcpu);

4
arch/arm64/include/asm/stage2_pgtable.h
View File

@ -21,13 +21,13 @@
* (IPA_SHIFT - 4).
*/
#define stage2_pgtable_levels(ipa) ARM64_HW_PGTABLE_LEVELS((ipa) - 4)
#define kvm_stage2_levels(kvm) VTCR_EL2_LVLS(kvm->arch.vtcr)
#define kvm_stage2_levels(mmu) VTCR_EL2_LVLS((mmu)->vtcr)
/*
* kvm_mmmu_cache_min_pages() is the number of pages required to install
* a stage-2 translation. We pre-allocate the entry level page table at
* the VM creation.
*/
#define kvm_mmu_cache_min_pages(kvm) (kvm_stage2_levels(kvm) - 1)
#define kvm_mmu_cache_min_pages(mmu) (kvm_stage2_levels(mmu) - 1)
#endif /* __ARM64_S2_PGTABLE_H_ */

45
arch/arm64/include/asm/sysreg.h
View File

@ -270,6 +270,8 @@
/* ETM */
#define SYS_TRCOSLAR sys_reg(2, 1, 1, 0, 4)
#define SYS_BRBCR_EL2 sys_reg(2, 4, 9, 0, 0)
#define SYS_MIDR_EL1 sys_reg(3, 0, 0, 0, 0)
#define SYS_MPIDR_EL1 sys_reg(3, 0, 0, 0, 5)
#define SYS_REVIDR_EL1 sys_reg(3, 0, 0, 0, 6)
@ -484,6 +486,7 @@
#define SYS_SCTLR_EL2 sys_reg(3, 4, 1, 0, 0)
#define SYS_ACTLR_EL2 sys_reg(3, 4, 1, 0, 1)
#define SYS_SCTLR2_EL2 sys_reg(3, 4, 1, 0, 3)
#define SYS_HCR_EL2 sys_reg(3, 4, 1, 1, 0)
#define SYS_MDCR_EL2 sys_reg(3, 4, 1, 1, 1)
#define SYS_CPTR_EL2 sys_reg(3, 4, 1, 1, 2)
@ -497,10 +500,15 @@
#define SYS_VTCR_EL2 sys_reg(3, 4, 2, 1, 2)
#define SYS_TRFCR_EL2 sys_reg(3, 4, 1, 2, 1)
#define SYS_VNCR_EL2 sys_reg(3, 4, 2, 2, 0)
#define SYS_HAFGRTR_EL2 sys_reg(3, 4, 3, 1, 6)
#define SYS_SPSR_EL2 sys_reg(3, 4, 4, 0, 0)
#define SYS_ELR_EL2 sys_reg(3, 4, 4, 0, 1)
#define SYS_SP_EL1 sys_reg(3, 4, 4, 1, 0)
#define SYS_SPSR_irq sys_reg(3, 4, 4, 3, 0)
#define SYS_SPSR_abt sys_reg(3, 4, 4, 3, 1)
#define SYS_SPSR_und sys_reg(3, 4, 4, 3, 2)
#define SYS_SPSR_fiq sys_reg(3, 4, 4, 3, 3)
#define SYS_IFSR32_EL2 sys_reg(3, 4, 5, 0, 1)
#define SYS_AFSR0_EL2 sys_reg(3, 4, 5, 1, 0)
#define SYS_AFSR1_EL2 sys_reg(3, 4, 5, 1, 1)
@ -514,6 +522,18 @@
#define SYS_MAIR_EL2 sys_reg(3, 4, 10, 2, 0)
#define SYS_AMAIR_EL2 sys_reg(3, 4, 10, 3, 0)
#define SYS_MPAMHCR_EL2 sys_reg(3, 4, 10, 4, 0)
#define SYS_MPAMVPMV_EL2 sys_reg(3, 4, 10, 4, 1)
#define SYS_MPAM2_EL2 sys_reg(3, 4, 10, 5, 0)
#define __SYS__MPAMVPMx_EL2(x) sys_reg(3, 4, 10, 6, x)
#define SYS_MPAMVPM0_EL2 __SYS__MPAMVPMx_EL2(0)
#define SYS_MPAMVPM1_EL2 __SYS__MPAMVPMx_EL2(1)
#define SYS_MPAMVPM2_EL2 __SYS__MPAMVPMx_EL2(2)
#define SYS_MPAMVPM3_EL2 __SYS__MPAMVPMx_EL2(3)
#define SYS_MPAMVPM4_EL2 __SYS__MPAMVPMx_EL2(4)
#define SYS_MPAMVPM5_EL2 __SYS__MPAMVPMx_EL2(5)
#define SYS_MPAMVPM6_EL2 __SYS__MPAMVPMx_EL2(6)
#define SYS_MPAMVPM7_EL2 __SYS__MPAMVPMx_EL2(7)
#define SYS_VBAR_EL2 sys_reg(3, 4, 12, 0, 0)
#define SYS_RVBAR_EL2 sys_reg(3, 4, 12, 0, 1)
@ -562,24 +582,49 @@
#define SYS_CONTEXTIDR_EL2 sys_reg(3, 4, 13, 0, 1)
#define SYS_TPIDR_EL2 sys_reg(3, 4, 13, 0, 2)
#define SYS_SCXTNUM_EL2 sys_reg(3, 4, 13, 0, 7)
#define __AMEV_op2(m) (m & 0x7)
#define __AMEV_CRm(n, m) (n | ((m & 0x8) >> 3))
#define __SYS__AMEVCNTVOFF0n_EL2(m) sys_reg(3, 4, 13, __AMEV_CRm(0x8, m), __AMEV_op2(m))
#define SYS_AMEVCNTVOFF0n_EL2(m) __SYS__AMEVCNTVOFF0n_EL2(m)
#define __SYS__AMEVCNTVOFF1n_EL2(m) sys_reg(3, 4, 13, __AMEV_CRm(0xA, m), __AMEV_op2(m))
#define SYS_AMEVCNTVOFF1n_EL2(m) __SYS__AMEVCNTVOFF1n_EL2(m)
#define SYS_CNTVOFF_EL2 sys_reg(3, 4, 14, 0, 3)
#define SYS_CNTHCTL_EL2 sys_reg(3, 4, 14, 1, 0)
#define SYS_CNTHP_TVAL_EL2 sys_reg(3, 4, 14, 2, 0)
#define SYS_CNTHP_CTL_EL2 sys_reg(3, 4, 14, 2, 1)
#define SYS_CNTHP_CVAL_EL2 sys_reg(3, 4, 14, 2, 2)
#define SYS_CNTHV_TVAL_EL2 sys_reg(3, 4, 14, 3, 0)
#define SYS_CNTHV_CTL_EL2 sys_reg(3, 4, 14, 3, 1)
#define SYS_CNTHV_CVAL_EL2 sys_reg(3, 4, 14, 3, 2)
/* VHE encodings for architectural EL0/1 system registers */
#define SYS_BRBCR_EL12 sys_reg(2, 5, 9, 0, 0)
#define SYS_SCTLR_EL12 sys_reg(3, 5, 1, 0, 0)
#define SYS_CPACR_EL12 sys_reg(3, 5, 1, 0, 2)
#define SYS_SCTLR2_EL12 sys_reg(3, 5, 1, 0, 3)
#define SYS_ZCR_EL12 sys_reg(3, 5, 1, 2, 0)
#define SYS_TRFCR_EL12 sys_reg(3, 5, 1, 2, 1)
#define SYS_SMCR_EL12 sys_reg(3, 5, 1, 2, 6)
#define SYS_TTBR0_EL12 sys_reg(3, 5, 2, 0, 0)
#define SYS_TTBR1_EL12 sys_reg(3, 5, 2, 0, 1)
#define SYS_TCR_EL12 sys_reg(3, 5, 2, 0, 2)
#define SYS_TCR2_EL12 sys_reg(3, 5, 2, 0, 3)
#define SYS_SPSR_EL12 sys_reg(3, 5, 4, 0, 0)
#define SYS_ELR_EL12 sys_reg(3, 5, 4, 0, 1)
#define SYS_AFSR0_EL12 sys_reg(3, 5, 5, 1, 0)
#define SYS_AFSR1_EL12 sys_reg(3, 5, 5, 1, 1)
#define SYS_ESR_EL12 sys_reg(3, 5, 5, 2, 0)
#define SYS_TFSR_EL12 sys_reg(3, 5, 5, 6, 0)
#define SYS_FAR_EL12 sys_reg(3, 5, 6, 0, 0)
#define SYS_PMSCR_EL12 sys_reg(3, 5, 9, 9, 0)
#define SYS_MAIR_EL12 sys_reg(3, 5, 10, 2, 0)
#define SYS_AMAIR_EL12 sys_reg(3, 5, 10, 3, 0)
#define SYS_VBAR_EL12 sys_reg(3, 5, 12, 0, 0)
#define SYS_CONTEXTIDR_EL12 sys_reg(3, 5, 13, 0, 1)
#define SYS_SCXTNUM_EL12 sys_reg(3, 5, 13, 0, 7)
#define SYS_CNTKCTL_EL12 sys_reg(3, 5, 14, 1, 0)
#define SYS_CNTP_TVAL_EL02 sys_reg(3, 5, 14, 2, 0)
#define SYS_CNTP_CTL_EL02 sys_reg(3, 5, 14, 2, 1)

8
arch/arm64/include/asm/tlbflush.h
View File

@ -332,7 +332,7 @@ static inline void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
* This is meant to avoid soft lock-ups on large TLB flushing ranges and not
* necessarily a performance improvement.
*/
#define MAX_TLBI_OPS PTRS_PER_PTE
#define MAX_DVM_OPS PTRS_PER_PTE
/*
* __flush_tlb_range_op - Perform TLBI operation upon a range
@ -412,12 +412,12 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma,
/*
* When not uses TLB range ops, we can handle up to
* (MAX_TLBI_OPS - 1) pages;
* (MAX_DVM_OPS - 1) pages;
* When uses TLB range ops, we can handle up to
* (MAX_TLBI_RANGE_PAGES - 1) pages.
*/
if ((!system_supports_tlb_range() &&
(end - start) >= (MAX_TLBI_OPS * stride)) ||
(end - start) >= (MAX_DVM_OPS * stride)) ||
pages >= MAX_TLBI_RANGE_PAGES) {
flush_tlb_mm(vma->vm_mm);
return;
@ -450,7 +450,7 @@ static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end
{
unsigned long addr;
if ((end - start) > (MAX_TLBI_OPS * PAGE_SIZE)) {
if ((end - start) > (MAX_DVM_OPS * PAGE_SIZE)) {
flush_tlb_all();
return;
}

54
arch/arm64/include/asm/traps.h
View File

@ -9,10 +9,9 @@
#include <linux/list.h>
#include <asm/esr.h>
#include <asm/ptrace.h>
#include <asm/sections.h>
struct pt_regs;
#ifdef CONFIG_ARMV8_DEPRECATED
bool try_emulate_armv8_deprecated(struct pt_regs *regs, u32 insn);
#else
@ -101,4 +100,55 @@ static inline unsigned long arm64_ras_serror_get_severity(unsigned long esr)
bool arm64_is_fatal_ras_serror(struct pt_regs *regs, unsigned long esr);
void __noreturn arm64_serror_panic(struct pt_regs *regs, unsigned long esr);
static inline void arm64_mops_reset_regs(struct user_pt_regs *regs, unsigned long esr)
{
bool wrong_option = esr & ESR_ELx_MOPS_ISS_WRONG_OPTION;
bool option_a = esr & ESR_ELx_MOPS_ISS_OPTION_A;
int dstreg = ESR_ELx_MOPS_ISS_DESTREG(esr);
int srcreg = ESR_ELx_MOPS_ISS_SRCREG(esr);
int sizereg = ESR_ELx_MOPS_ISS_SIZEREG(esr);
unsigned long dst, src, size;
dst = regs->regs[dstreg];
src = regs->regs[srcreg];
size = regs->regs[sizereg];
/*
* Put the registers back in the original format suitable for a
* prologue instruction, using the generic return routine from the
* Arm ARM (DDI 0487I.a) rules CNTMJ and MWFQH.
*/
if (esr & ESR_ELx_MOPS_ISS_MEM_INST) {
/* SET* instruction */
if (option_a ^ wrong_option) {
/* Format is from Option A; forward set */
regs->regs[dstreg] = dst + size;
regs->regs[sizereg] = -size;
}
} else {
/* CPY* instruction */
if (!(option_a ^ wrong_option)) {
/* Format is from Option B */
if (regs->pstate & PSR_N_BIT) {
/* Backward copy */
regs->regs[dstreg] = dst - size;
regs->regs[srcreg] = src - size;
}
} else {
/* Format is from Option A */
if (size & BIT(63)) {
/* Forward copy */
regs->regs[dstreg] = dst + size;
regs->regs[srcreg] = src + size;
regs->regs[sizereg] = -size;
}
}
}
if (esr & ESR_ELx_MOPS_ISS_FROM_EPILOGUE)
regs->pc -= 8;
else
regs->pc -= 4;
}
#endif

32
arch/arm64/include/uapi/asm/kvm.h
View File

@ -505,6 +505,38 @@ struct kvm_smccc_filter {
#define KVM_HYPERCALL_EXIT_SMC (1U << 0)
#define KVM_HYPERCALL_EXIT_16BIT (1U << 1)
/*
* Get feature ID registers userspace writable mask.
*
* From DDI0487J.a, D19.2.66 ("ID_AA64MMFR2_EL1, AArch64 Memory Model
* Feature Register 2"):
*
* "The Feature ID space is defined as the System register space in
* AArch64 with op0==3, op1=={0, 1, 3}, CRn==0, CRm=={0-7},
* op2=={0-7}."
*
* This covers all currently known R/O registers that indicate
* anything useful feature wise, including the ID registers.
*
* If we ever need to introduce a new range, it will be described as
* such in the range field.
*/
#define KVM_ARM_FEATURE_ID_RANGE_IDX(op0, op1, crn, crm, op2) \
({ \
__u64 __op1 = (op1) & 3; \
__op1 -= (__op1 == 3); \
(__op1 << 6 | ((crm) & 7) << 3 | (op2)); \
})
#define KVM_ARM_FEATURE_ID_RANGE 0
#define KVM_ARM_FEATURE_ID_RANGE_SIZE (3 * 8 * 8)
struct reg_mask_range {
__u64 addr; /* Pointer to mask array */
__u32 range; /* Requested range */
__u32 reserved[13];
};
#endif
#endif /* __ARM_KVM_H__ */

48
arch/arm64/kernel/traps.c
View File

@ -516,53 +516,7 @@ void do_el1_fpac(struct pt_regs *regs, unsigned long esr)
void do_el0_mops(struct pt_regs *regs, unsigned long esr)
{
bool wrong_option = esr & ESR_ELx_MOPS_ISS_WRONG_OPTION;
bool option_a = esr & ESR_ELx_MOPS_ISS_OPTION_A;
int dstreg = ESR_ELx_MOPS_ISS_DESTREG(esr);
int srcreg = ESR_ELx_MOPS_ISS_SRCREG(esr);
int sizereg = ESR_ELx_MOPS_ISS_SIZEREG(esr);
unsigned long dst, src, size;
dst = pt_regs_read_reg(regs, dstreg);
src = pt_regs_read_reg(regs, srcreg);
size = pt_regs_read_reg(regs, sizereg);
/*
* Put the registers back in the original format suitable for a
* prologue instruction, using the generic return routine from the
* Arm ARM (DDI 0487I.a) rules CNTMJ and MWFQH.
*/
if (esr & ESR_ELx_MOPS_ISS_MEM_INST) {
/* SET* instruction */
if (option_a ^ wrong_option) {
/* Format is from Option A; forward set */
pt_regs_write_reg(regs, dstreg, dst + size);
pt_regs_write_reg(regs, sizereg, -size);
}
} else {
/* CPY* instruction */
if (!(option_a ^ wrong_option)) {
/* Format is from Option B */
if (regs->pstate & PSR_N_BIT) {
/* Backward copy */
pt_regs_write_reg(regs, dstreg, dst - size);
pt_regs_write_reg(regs, srcreg, src - size);
}
} else {
/* Format is from Option A */
if (size & BIT(63)) {
/* Forward copy */
pt_regs_write_reg(regs, dstreg, dst + size);
pt_regs_write_reg(regs, srcreg, src + size);
pt_regs_write_reg(regs, sizereg, -size);
}
}
}
if (esr & ESR_ELx_MOPS_ISS_FROM_EPILOGUE)
regs->pc -= 8;
else
regs->pc -= 4;
arm64_mops_reset_regs(&regs->user_regs, esr);
/*
* If single stepping then finish the step before executing the

6
arch/arm64/kvm/arch_timer.c
View File

@ -453,7 +453,7 @@ static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level,
timer_ctx->irq.level);
if (!userspace_irqchip(vcpu->kvm)) {
ret = kvm_vgic_inject_irq(vcpu->kvm, vcpu->vcpu_id,
ret = kvm_vgic_inject_irq(vcpu->kvm, vcpu,
timer_irq(timer_ctx),
timer_ctx->irq.level,
timer_ctx);
@ -936,7 +936,7 @@ void kvm_timer_sync_user(struct kvm_vcpu *vcpu)
unmask_vtimer_irq_user(vcpu);
}
int kvm_timer_vcpu_reset(struct kvm_vcpu *vcpu)
void kvm_timer_vcpu_reset(struct kvm_vcpu *vcpu)
{
struct arch_timer_cpu *timer = vcpu_timer(vcpu);
struct timer_map map;
@ -980,8 +980,6 @@ int kvm_timer_vcpu_reset(struct kvm_vcpu *vcpu)
soft_timer_cancel(&map.emul_vtimer->hrtimer);
if (map.emul_ptimer)
soft_timer_cancel(&map.emul_ptimer->hrtimer);
return 0;
}
static void timer_context_init(struct kvm_vcpu *vcpu, int timerid)

200
arch/arm64/kvm/arm.c
View File

@ -205,6 +205,7 @@ void kvm_arch_destroy_vm(struct kvm *kvm)
if (is_protected_kvm_enabled())
pkvm_destroy_hyp_vm(kvm);
kfree(kvm->arch.mpidr_data);
kvm_destroy_vcpus(kvm);
kvm_unshare_hyp(kvm, kvm + 1);
@ -317,6 +318,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
case KVM_CAP_ARM_SUPPORTED_BLOCK_SIZES:
r = kvm_supported_block_sizes();
break;
case KVM_CAP_ARM_SUPPORTED_REG_MASK_RANGES:
r = BIT(0);
break;
default:
r = 0;
}
@ -367,7 +371,6 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
/* Force users to call KVM_ARM_VCPU_INIT */
vcpu_clear_flag(vcpu, VCPU_INITIALIZED);
bitmap_zero(vcpu->arch.features, KVM_VCPU_MAX_FEATURES);
vcpu->arch.mmu_page_cache.gfp_zero = __GFP_ZERO;
@ -438,9 +441,9 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
* We might get preempted before the vCPU actually runs, but
* over-invalidation doesn't affect correctness.
*/
if (*last_ran != vcpu->vcpu_id) {
if (*last_ran != vcpu->vcpu_idx) {
kvm_call_hyp(__kvm_flush_cpu_context, mmu);
*last_ran = vcpu->vcpu_id;
*last_ran = vcpu->vcpu_idx;
}
vcpu->cpu = cpu;
@ -448,7 +451,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
kvm_vgic_load(vcpu);
kvm_timer_vcpu_load(vcpu);
if (has_vhe())
kvm_vcpu_load_sysregs_vhe(vcpu);
kvm_vcpu_load_vhe(vcpu);
kvm_arch_vcpu_load_fp(vcpu);
kvm_vcpu_pmu_restore_guest(vcpu);
if (kvm_arm_is_pvtime_enabled(&vcpu->arch))
@ -472,7 +475,7 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
kvm_arch_vcpu_put_debug_state_flags(vcpu);
kvm_arch_vcpu_put_fp(vcpu);
if (has_vhe())
kvm_vcpu_put_sysregs_vhe(vcpu);
kvm_vcpu_put_vhe(vcpu);
kvm_timer_vcpu_put(vcpu);
kvm_vgic_put(vcpu);
kvm_vcpu_pmu_restore_host(vcpu);
@ -578,6 +581,57 @@ static int kvm_vcpu_initialized(struct kvm_vcpu *vcpu)
return vcpu_get_flag(vcpu, VCPU_INITIALIZED);
}
static void kvm_init_mpidr_data(struct kvm *kvm)
{
struct kvm_mpidr_data *data = NULL;
unsigned long c, mask, nr_entries;
u64 aff_set = 0, aff_clr = ~0UL;
struct kvm_vcpu *vcpu;
mutex_lock(&kvm->arch.config_lock);
if (kvm->arch.mpidr_data || atomic_read(&kvm->online_vcpus) == 1)
goto out;
kvm_for_each_vcpu(c, vcpu, kvm) {
u64 aff = kvm_vcpu_get_mpidr_aff(vcpu);
aff_set |= aff;
aff_clr &= aff;
}
/*
* A significant bit can be either 0 or 1, and will only appear in
* aff_set. Use aff_clr to weed out the useless stuff.
*/
mask = aff_set ^ aff_clr;
nr_entries = BIT_ULL(hweight_long(mask));
/*
* Don't let userspace fool us. If we need more than a single page
* to describe the compressed MPIDR array, just fall back to the
* iterative method. Single vcpu VMs do not need this either.
*/
if (struct_size(data, cmpidr_to_idx, nr_entries) <= PAGE_SIZE)
data = kzalloc(struct_size(data, cmpidr_to_idx, nr_entries),
GFP_KERNEL_ACCOUNT);
if (!data)
goto out;
data->mpidr_mask = mask;
kvm_for_each_vcpu(c, vcpu, kvm) {
u64 aff = kvm_vcpu_get_mpidr_aff(vcpu);
u16 index = kvm_mpidr_index(data, aff);
data->cmpidr_to_idx[index] = c;
}
kvm->arch.mpidr_data = data;
out:
mutex_unlock(&kvm->arch.config_lock);
}
/*
* Handle both the initialisation that is being done when the vcpu is
* run for the first time, as well as the updates that must be
@ -601,6 +655,8 @@ int kvm_arch_vcpu_run_pid_change(struct kvm_vcpu *vcpu)
if (likely(vcpu_has_run_once(vcpu)))
return 0;
kvm_init_mpidr_data(kvm);
kvm_arm_vcpu_init_debug(vcpu);
if (likely(irqchip_in_kernel(kvm))) {
@ -801,8 +857,7 @@ static int check_vcpu_requests(struct kvm_vcpu *vcpu)
}
if (kvm_check_request(KVM_REQ_RELOAD_PMU, vcpu))
kvm_pmu_handle_pmcr(vcpu,
__vcpu_sys_reg(vcpu, PMCR_EL0));
kvm_vcpu_reload_pmu(vcpu);
if (kvm_check_request(KVM_REQ_RESYNC_PMU_EL0, vcpu))
kvm_vcpu_pmu_restore_guest(vcpu);
@ -950,7 +1005,10 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
* making a thread's VMID inactive. So we need to call
* kvm_arm_vmid_update() in non-premptible context.
*/
kvm_arm_vmid_update(&vcpu->arch.hw_mmu->vmid);
if (kvm_arm_vmid_update(&vcpu->arch.hw_mmu->vmid) &&
has_vhe())
__load_stage2(vcpu->arch.hw_mmu,
vcpu->arch.hw_mmu->arch);
kvm_pmu_flush_hwstate(vcpu);
@ -1134,27 +1192,23 @@ int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_level,
bool line_status)
{
u32 irq = irq_level->irq;
unsigned int irq_type, vcpu_idx, irq_num;
int nrcpus = atomic_read(&kvm->online_vcpus);
unsigned int irq_type, vcpu_id, irq_num;
struct kvm_vcpu *vcpu = NULL;
bool level = irq_level->level;
irq_type = (irq >> KVM_ARM_IRQ_TYPE_SHIFT) & KVM_ARM_IRQ_TYPE_MASK;
vcpu_idx = (irq >> KVM_ARM_IRQ_VCPU_SHIFT) & KVM_ARM_IRQ_VCPU_MASK;
vcpu_idx += ((irq >> KVM_ARM_IRQ_VCPU2_SHIFT) & KVM_ARM_IRQ_VCPU2_MASK) * (KVM_ARM_IRQ_VCPU_MASK + 1);
vcpu_id = (irq >> KVM_ARM_IRQ_VCPU_SHIFT) & KVM_ARM_IRQ_VCPU_MASK;
vcpu_id += ((irq >> KVM_ARM_IRQ_VCPU2_SHIFT) & KVM_ARM_IRQ_VCPU2_MASK) * (KVM_ARM_IRQ_VCPU_MASK + 1);
irq_num = (irq >> KVM_ARM_IRQ_NUM_SHIFT) & KVM_ARM_IRQ_NUM_MASK;
trace_kvm_irq_line(irq_type, vcpu_idx, irq_num, irq_level->level);
trace_kvm_irq_line(irq_type, vcpu_id, irq_num, irq_level->level);
switch (irq_type) {
case KVM_ARM_IRQ_TYPE_CPU:
if (irqchip_in_kernel(kvm))
return -ENXIO;
if (vcpu_idx >= nrcpus)
return -EINVAL;
vcpu = kvm_get_vcpu(kvm, vcpu_idx);
vcpu = kvm_get_vcpu_by_id(kvm, vcpu_id);
if (!vcpu)
return -EINVAL;
@ -1166,17 +1220,14 @@ int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_level,
if (!irqchip_in_kernel(kvm))
return -ENXIO;
if (vcpu_idx >= nrcpus)
return -EINVAL;
vcpu = kvm_get_vcpu(kvm, vcpu_idx);
vcpu = kvm_get_vcpu_by_id(kvm, vcpu_id);
if (!vcpu)
return -EINVAL;
if (irq_num < VGIC_NR_SGIS || irq_num >= VGIC_NR_PRIVATE_IRQS)
return -EINVAL;
return kvm_vgic_inject_irq(kvm, vcpu->vcpu_id, irq_num, level, NULL);
return kvm_vgic_inject_irq(kvm, vcpu, irq_num, level, NULL);
case KVM_ARM_IRQ_TYPE_SPI:
if (!irqchip_in_kernel(kvm))
return -ENXIO;
@ -1184,12 +1235,36 @@ int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_level,
if (irq_num < VGIC_NR_PRIVATE_IRQS)
return -EINVAL;
return kvm_vgic_inject_irq(kvm, 0, irq_num, level, NULL);
return kvm_vgic_inject_irq(kvm, NULL, irq_num, level, NULL);
}
return -EINVAL;
}
static unsigned long system_supported_vcpu_features(void)
{
unsigned long features = KVM_VCPU_VALID_FEATURES;
if (!cpus_have_final_cap(ARM64_HAS_32BIT_EL1))
clear_bit(KVM_ARM_VCPU_EL1_32BIT, &features);
if (!kvm_arm_support_pmu_v3())
clear_bit(KVM_ARM_VCPU_PMU_V3, &features);
if (!system_supports_sve())
clear_bit(KVM_ARM_VCPU_SVE, &features);
if (!system_has_full_ptr_auth()) {
clear_bit(KVM_ARM_VCPU_PTRAUTH_ADDRESS, &features);
clear_bit(KVM_ARM_VCPU_PTRAUTH_GENERIC, &features);
}
if (!cpus_have_final_cap(ARM64_HAS_NESTED_VIRT))
clear_bit(KVM_ARM_VCPU_HAS_EL2, &features);
return features;
}
static int kvm_vcpu_init_check_features(struct kvm_vcpu *vcpu,
const struct kvm_vcpu_init *init)
{
@ -1204,12 +1279,25 @@ static int kvm_vcpu_init_check_features(struct kvm_vcpu *vcpu,
return -ENOENT;
}
if (features & ~system_supported_vcpu_features())
return -EINVAL;
/*
* For now make sure that both address/generic pointer authentication
* features are requested by the userspace together.
*/
if (test_bit(KVM_ARM_VCPU_PTRAUTH_ADDRESS, &features) !=
test_bit(KVM_ARM_VCPU_PTRAUTH_GENERIC, &features))
return -EINVAL;
/* Disallow NV+SVE for the time being */
if (test_bit(KVM_ARM_VCPU_HAS_EL2, &features) &&
test_bit(KVM_ARM_VCPU_SVE, &features))
return -EINVAL;
if (!test_bit(KVM_ARM_VCPU_EL1_32BIT, &features))
return 0;
if (!cpus_have_final_cap(ARM64_HAS_32BIT_EL1))
return -EINVAL;
/* MTE is incompatible with AArch32 */
if (kvm_has_mte(vcpu->kvm))
return -EINVAL;
@ -1226,7 +1314,23 @@ static bool kvm_vcpu_init_changed(struct kvm_vcpu *vcpu,
{
unsigned long features = init->features[0];
return !bitmap_equal(vcpu->arch.features, &features, KVM_VCPU_MAX_FEATURES);
return !bitmap_equal(vcpu->kvm->arch.vcpu_features, &features,
KVM_VCPU_MAX_FEATURES);
}
static int kvm_setup_vcpu(struct kvm_vcpu *vcpu)
{
struct kvm *kvm = vcpu->kvm;
int ret = 0;
/*
* When the vCPU has a PMU, but no PMU is set for the guest
* yet, set the default one.
*/
if (kvm_vcpu_has_pmu(vcpu) && !kvm->arch.arm_pmu)
ret = kvm_arm_set_default_pmu(kvm);
return ret;
}
static int __kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
@ -1239,21 +1343,21 @@ static int __kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
mutex_lock(&kvm->arch.config_lock);
if (test_bit(KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGURED, &kvm->arch.flags) &&
!bitmap_equal(kvm->arch.vcpu_features, &features, KVM_VCPU_MAX_FEATURES))
kvm_vcpu_init_changed(vcpu, init))
goto out_unlock;
bitmap_copy(vcpu->arch.features, &features, KVM_VCPU_MAX_FEATURES);
/* Now we know what it is, we can reset it. */
ret = kvm_reset_vcpu(vcpu);
if (ret) {
bitmap_zero(vcpu->arch.features, KVM_VCPU_MAX_FEATURES);
goto out_unlock;
}
bitmap_copy(kvm->arch.vcpu_features, &features, KVM_VCPU_MAX_FEATURES);
ret = kvm_setup_vcpu(vcpu);
if (ret)
goto out_unlock;
/* Now we know what it is, we can reset it. */
kvm_reset_vcpu(vcpu);
set_bit(KVM_ARCH_FLAG_VCPU_FEATURES_CONFIGURED, &kvm->arch.flags);
vcpu_set_flag(vcpu, VCPU_INITIALIZED);
ret = 0;
out_unlock:
mutex_unlock(&kvm->arch.config_lock);
return ret;
@ -1278,7 +1382,8 @@ static int kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
if (kvm_vcpu_init_changed(vcpu, init))
return -EINVAL;
return kvm_reset_vcpu(vcpu);
kvm_reset_vcpu(vcpu);
return 0;
}
static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu,
@ -1629,6 +1734,13 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg)
return kvm_vm_set_attr(kvm, &attr);
}
case KVM_ARM_GET_REG_WRITABLE_MASKS: {
struct reg_mask_range range;
if (copy_from_user(&range, argp, sizeof(range)))
return -EFAULT;
return kvm_vm_ioctl_get_reg_writable_masks(kvm, &range);
}
default:
return -EINVAL;
}
@ -2341,6 +2453,18 @@ struct kvm_vcpu *kvm_mpidr_to_vcpu(struct kvm *kvm, unsigned long mpidr)
unsigned long i;
mpidr &= MPIDR_HWID_BITMASK;
if (kvm->arch.mpidr_data) {
u16 idx = kvm_mpidr_index(kvm->arch.mpidr_data, mpidr);
vcpu = kvm_get_vcpu(kvm,
kvm->arch.mpidr_data->cmpidr_to_idx[idx]);
if (mpidr != kvm_vcpu_get_mpidr_aff(vcpu))
vcpu = NULL;
return vcpu;
}
kvm_for_each_vcpu(i, vcpu, kvm) {
if (mpidr == kvm_vcpu_get_mpidr_aff(vcpu))
return vcpu;

77
arch/arm64/kvm/emulate-nested.c
View File

@ -648,15 +648,80 @@ static const struct encoding_to_trap_config encoding_to_cgt[] __initconst = {
SR_TRAP(SYS_APGAKEYLO_EL1, CGT_HCR_APK),
SR_TRAP(SYS_APGAKEYHI_EL1, CGT_HCR_APK),
/* All _EL2 registers */
SR_RANGE_TRAP(sys_reg(3, 4, 0, 0, 0),
sys_reg(3, 4, 3, 15, 7), CGT_HCR_NV),
SR_TRAP(SYS_BRBCR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_VPIDR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_VMPIDR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_SCTLR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_ACTLR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_SCTLR2_EL2, CGT_HCR_NV),
SR_RANGE_TRAP(SYS_HCR_EL2,
SYS_HCRX_EL2, CGT_HCR_NV),
SR_TRAP(SYS_SMPRIMAP_EL2, CGT_HCR_NV),
SR_TRAP(SYS_SMCR_EL2, CGT_HCR_NV),
SR_RANGE_TRAP(SYS_TTBR0_EL2,
SYS_TCR2_EL2, CGT_HCR_NV),
SR_TRAP(SYS_VTTBR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_VTCR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_VNCR_EL2, CGT_HCR_NV),
SR_RANGE_TRAP(SYS_HDFGRTR_EL2,
SYS_HAFGRTR_EL2, CGT_HCR_NV),
/* Skip the SP_EL1 encoding... */
SR_TRAP(SYS_SPSR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_ELR_EL2, CGT_HCR_NV),
SR_RANGE_TRAP(sys_reg(3, 4, 4, 1, 1),
sys_reg(3, 4, 10, 15, 7), CGT_HCR_NV),
SR_RANGE_TRAP(sys_reg(3, 4, 12, 0, 0),
sys_reg(3, 4, 14, 15, 7), CGT_HCR_NV),
/* Skip SPSR_irq, SPSR_abt, SPSR_und, SPSR_fiq */
SR_TRAP(SYS_AFSR0_EL2, CGT_HCR_NV),
SR_TRAP(SYS_AFSR1_EL2, CGT_HCR_NV),
SR_TRAP(SYS_ESR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_VSESR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_TFSR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_FAR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_HPFAR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_PMSCR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_MAIR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_AMAIR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_MPAMHCR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_MPAMVPMV_EL2, CGT_HCR_NV),
SR_TRAP(SYS_MPAM2_EL2, CGT_HCR_NV),
SR_RANGE_TRAP(SYS_MPAMVPM0_EL2,
SYS_MPAMVPM7_EL2, CGT_HCR_NV),
/*
* Note that the spec. describes a group of MEC registers
* whose access should not trap, therefore skip the following:
* MECID_A0_EL2, MECID_A1_EL2, MECID_P0_EL2,
* MECID_P1_EL2, MECIDR_EL2, VMECID_A_EL2,
* VMECID_P_EL2.
*/
SR_RANGE_TRAP(SYS_VBAR_EL2,
SYS_RMR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_VDISR_EL2, CGT_HCR_NV),
/* ICH_AP0R<m>_EL2 */
SR_RANGE_TRAP(SYS_ICH_AP0R0_EL2,
SYS_ICH_AP0R3_EL2, CGT_HCR_NV),
/* ICH_AP1R<m>_EL2 */
SR_RANGE_TRAP(SYS_ICH_AP1R0_EL2,
SYS_ICH_AP1R3_EL2, CGT_HCR_NV),
SR_TRAP(SYS_ICC_SRE_EL2, CGT_HCR_NV),
SR_RANGE_TRAP(SYS_ICH_HCR_EL2,
SYS_ICH_EISR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_ICH_ELRSR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_ICH_VMCR_EL2, CGT_HCR_NV),
/* ICH_LR<m>_EL2 */
SR_RANGE_TRAP(SYS_ICH_LR0_EL2,
SYS_ICH_LR15_EL2, CGT_HCR_NV),
SR_TRAP(SYS_CONTEXTIDR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_TPIDR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_SCXTNUM_EL2, CGT_HCR_NV),
/* AMEVCNTVOFF0<n>_EL2, AMEVCNTVOFF1<n>_EL2 */
SR_RANGE_TRAP(SYS_AMEVCNTVOFF0n_EL2(0),
SYS_AMEVCNTVOFF1n_EL2(15), CGT_HCR_NV),
/* CNT*_EL2 */
SR_TRAP(SYS_CNTVOFF_EL2, CGT_HCR_NV),
SR_TRAP(SYS_CNTPOFF_EL2, CGT_HCR_NV),
SR_TRAP(SYS_CNTHCTL_EL2, CGT_HCR_NV),
SR_RANGE_TRAP(SYS_CNTHP_TVAL_EL2,
SYS_CNTHP_CVAL_EL2, CGT_HCR_NV),
SR_RANGE_TRAP(SYS_CNTHV_TVAL_EL2,
SYS_CNTHV_CVAL_EL2, CGT_HCR_NV),
/* All _EL02, _EL12 registers */
SR_RANGE_TRAP(sys_reg(3, 5, 0, 0, 0),
sys_reg(3, 5, 10, 15, 7), CGT_HCR_NV),

17
arch/arm64/kvm/hyp/include/hyp/switch.h
View File

@ -30,6 +30,7 @@
#include <asm/fpsimd.h>
#include <asm/debug-monitors.h>
#include <asm/processor.h>
#include <asm/traps.h>
struct kvm_exception_table_entry {
int insn, fixup;
@ -265,6 +266,22 @@ static inline bool __populate_fault_info(struct kvm_vcpu *vcpu)
return __get_fault_info(vcpu->arch.fault.esr_el2, &vcpu->arch.fault);
}
static bool kvm_hyp_handle_mops(struct kvm_vcpu *vcpu, u64 *exit_code)
{
*vcpu_pc(vcpu) = read_sysreg_el2(SYS_ELR);
arm64_mops_reset_regs(vcpu_gp_regs(vcpu), vcpu->arch.fault.esr_el2);
write_sysreg_el2(*vcpu_pc(vcpu), SYS_ELR);
/*
* Finish potential single step before executing the prologue
* instruction.
*/
*vcpu_cpsr(vcpu) &= ~DBG_SPSR_SS;
write_sysreg_el2(*vcpu_cpsr(vcpu), SYS_SPSR);
return true;
}
static inline void __hyp_sve_restore_guest(struct kvm_vcpu *vcpu)
{
sve_cond_update_zcr_vq(vcpu_sve_max_vq(vcpu) - 1, SYS_ZCR_EL2);

3
arch/arm64/kvm/hyp/include/nvhe/fixed_config.h
View File

@ -197,7 +197,8 @@
#define PVM_ID_AA64ISAR2_ALLOW (\
ARM64_FEATURE_MASK(ID_AA64ISAR2_EL1_GPA3) | \
ARM64_FEATURE_MASK(ID_AA64ISAR2_EL1_APA3) \
ARM64_FEATURE_MASK(ID_AA64ISAR2_EL1_APA3) | \
ARM64_FEATURE_MASK(ID_AA64ISAR2_EL1_MOPS) \
)
u64 pvm_read_id_reg(const struct kvm_vcpu *vcpu, u32 id);

8
arch/arm64/kvm/hyp/nvhe/mem_protect.c
View File

@ -129,8 +129,8 @@ static void prepare_host_vtcr(void)
parange = kvm_get_parange(id_aa64mmfr0_el1_sys_val);
phys_shift = id_aa64mmfr0_parange_to_phys_shift(parange);
host_mmu.arch.vtcr = kvm_get_vtcr(id_aa64mmfr0_el1_sys_val,
id_aa64mmfr1_el1_sys_val, phys_shift);
host_mmu.arch.mmu.vtcr = kvm_get_vtcr(id_aa64mmfr0_el1_sys_val,
id_aa64mmfr1_el1_sys_val, phys_shift);
}
static bool host_stage2_force_pte_cb(u64 addr, u64 end, enum kvm_pgtable_prot prot);
@ -235,7 +235,7 @@ int kvm_guest_prepare_stage2(struct pkvm_hyp_vm *vm, void *pgd)
unsigned long nr_pages;
int ret;
nr_pages = kvm_pgtable_stage2_pgd_size(vm->kvm.arch.vtcr) >> PAGE_SHIFT;
nr_pages = kvm_pgtable_stage2_pgd_size(mmu->vtcr) >> PAGE_SHIFT;
ret = hyp_pool_init(&vm->pool, hyp_virt_to_pfn(pgd), nr_pages, 0);
if (ret)
return ret;
@ -295,7 +295,7 @@ int __pkvm_prot_finalize(void)
return -EPERM;
params->vttbr = kvm_get_vttbr(mmu);
params->vtcr = host_mmu.arch.vtcr;
params->vtcr = mmu->vtcr;
params->hcr_el2 |= HCR_VM;
/*

4
arch/arm64/kvm/hyp/nvhe/pkvm.c
View File

@ -303,7 +303,7 @@ static void init_pkvm_hyp_vm(struct kvm *host_kvm, struct pkvm_hyp_vm *hyp_vm,
{
hyp_vm->host_kvm = host_kvm;
hyp_vm->kvm.created_vcpus = nr_vcpus;
hyp_vm->kvm.arch.vtcr = host_mmu.arch.vtcr;
hyp_vm->kvm.arch.mmu.vtcr = host_mmu.arch.mmu.vtcr;
}
static int init_pkvm_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu,
@ -483,7 +483,7 @@ int __pkvm_init_vm(struct kvm *host_kvm, unsigned long vm_hva,
}
vm_size = pkvm_get_hyp_vm_size(nr_vcpus);
pgd_size = kvm_pgtable_stage2_pgd_size(host_mmu.arch.vtcr);
pgd_size = kvm_pgtable_stage2_pgd_size(host_mmu.arch.mmu.vtcr);
ret = -ENOMEM;

2
arch/arm64/kvm/hyp/nvhe/switch.c
View File

@ -192,6 +192,7 @@ static const exit_handler_fn hyp_exit_handlers[] = {
[ESR_ELx_EC_DABT_LOW] = kvm_hyp_handle_dabt_low,
[ESR_ELx_EC_WATCHPT_LOW] = kvm_hyp_handle_watchpt_low,
[ESR_ELx_EC_PAC] = kvm_hyp_handle_ptrauth,
[ESR_ELx_EC_MOPS] = kvm_hyp_handle_mops,
};
static const exit_handler_fn pvm_exit_handlers[] = {
@ -203,6 +204,7 @@ static const exit_handler_fn pvm_exit_handlers[] = {
[ESR_ELx_EC_DABT_LOW] = kvm_hyp_handle_dabt_low,
[ESR_ELx_EC_WATCHPT_LOW] = kvm_hyp_handle_watchpt_low,
[ESR_ELx_EC_PAC] = kvm_hyp_handle_ptrauth,
[ESR_ELx_EC_MOPS] = kvm_hyp_handle_mops,
};
static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *vcpu)

4
arch/arm64/kvm/hyp/pgtable.c
View File

@ -1314,7 +1314,7 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr,
ret = stage2_update_leaf_attrs(pgt, addr, 1, set, clr, NULL, &level,
KVM_PGTABLE_WALK_HANDLE_FAULT |
KVM_PGTABLE_WALK_SHARED);
if (!ret)
if (!ret || ret == -EAGAIN)
kvm_call_hyp(__kvm_tlb_flush_vmid_ipa_nsh, pgt->mmu, addr, level);
return ret;
}
@ -1511,7 +1511,7 @@ int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_s2_mmu *mmu,
kvm_pgtable_force_pte_cb_t force_pte_cb)
{
size_t pgd_sz;
u64 vtcr = mmu->arch->vtcr;
u64 vtcr = mmu->vtcr;
u32 ia_bits = VTCR_EL2_IPA(vtcr);
u32 sl0 = FIELD_GET(VTCR_EL2_SL0_MASK, vtcr);
u32 start_level = VTCR_EL2_TGRAN_SL0_BASE - sl0;

34
arch/arm64/kvm/hyp/vhe/switch.c
View File

@ -137,12 +137,12 @@ static void __deactivate_traps(struct kvm_vcpu *vcpu)
NOKPROBE_SYMBOL(__deactivate_traps);
/*
* Disable IRQs in {activate,deactivate}_traps_vhe_{load,put}() to
* Disable IRQs in __vcpu_{load,put}_{activate,deactivate}_traps() to
* prevent a race condition between context switching of PMUSERENR_EL0
* in __{activate,deactivate}_traps_common() and IPIs that attempts to
* update PMUSERENR_EL0. See also kvm_set_pmuserenr().
*/
void activate_traps_vhe_load(struct kvm_vcpu *vcpu)
static void __vcpu_load_activate_traps(struct kvm_vcpu *vcpu)
{
unsigned long flags;
@ -151,7 +151,7 @@ void activate_traps_vhe_load(struct kvm_vcpu *vcpu)
local_irq_restore(flags);
}
void deactivate_traps_vhe_put(struct kvm_vcpu *vcpu)
static void __vcpu_put_deactivate_traps(struct kvm_vcpu *vcpu)
{
unsigned long flags;
@ -160,6 +160,19 @@ void deactivate_traps_vhe_put(struct kvm_vcpu *vcpu)
local_irq_restore(flags);
}
void kvm_vcpu_load_vhe(struct kvm_vcpu *vcpu)
{
__vcpu_load_switch_sysregs(vcpu);
__vcpu_load_activate_traps(vcpu);
__load_stage2(vcpu->arch.hw_mmu, vcpu->arch.hw_mmu->arch);
}
void kvm_vcpu_put_vhe(struct kvm_vcpu *vcpu)
{
__vcpu_put_deactivate_traps(vcpu);
__vcpu_put_switch_sysregs(vcpu);
}
static const exit_handler_fn hyp_exit_handlers[] = {
[0 ... ESR_ELx_EC_MAX] = NULL,
[ESR_ELx_EC_CP15_32] = kvm_hyp_handle_cp15_32,
@ -170,6 +183,7 @@ static const exit_handler_fn hyp_exit_handlers[] = {
[ESR_ELx_EC_DABT_LOW] = kvm_hyp_handle_dabt_low,
[ESR_ELx_EC_WATCHPT_LOW] = kvm_hyp_handle_watchpt_low,
[ESR_ELx_EC_PAC] = kvm_hyp_handle_ptrauth,
[ESR_ELx_EC_MOPS] = kvm_hyp_handle_mops,
};
static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *vcpu)
@ -214,17 +228,11 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
sysreg_save_host_state_vhe(host_ctxt);
/*
* ARM erratum 1165522 requires us to configure both stage 1 and
* stage 2 translation for the guest context before we clear
* HCR_EL2.TGE.
*
* We have already configured the guest's stage 1 translation in
* kvm_vcpu_load_sysregs_vhe above. We must now call
* __load_stage2 before __activate_traps, because
* __load_stage2 configures stage 2 translation, and
* __activate_traps clear HCR_EL2.TGE (among other things).
* Note that ARM erratum 1165522 requires us to configure both stage 1
* and stage 2 translation for the guest context before we clear
* HCR_EL2.TGE. The stage 1 and stage 2 guest context has already been
* loaded on the CPU in kvm_vcpu_load_vhe().
*/
__load_stage2(vcpu->arch.hw_mmu, vcpu->arch.hw_mmu->arch);
__activate_traps(vcpu);
__kvm_adjust_pc(vcpu);

11
arch/arm64/kvm/hyp/vhe/sysreg-sr.c
View File

@ -52,7 +52,7 @@ void sysreg_restore_guest_state_vhe(struct kvm_cpu_context *ctxt)
NOKPROBE_SYMBOL(sysreg_restore_guest_state_vhe);
/**
* kvm_vcpu_load_sysregs_vhe - Load guest system registers to the physical CPU
* __vcpu_load_switch_sysregs - Load guest system registers to the physical CPU
*
* @vcpu: The VCPU pointer
*
@ -62,7 +62,7 @@ NOKPROBE_SYMBOL(sysreg_restore_guest_state_vhe);
* and loading system register state early avoids having to load them on
* every entry to the VM.
*/
void kvm_vcpu_load_sysregs_vhe(struct kvm_vcpu *vcpu)
void __vcpu_load_switch_sysregs(struct kvm_vcpu *vcpu)
{
struct kvm_cpu_context *guest_ctxt = &vcpu->arch.ctxt;
struct kvm_cpu_context *host_ctxt;
@ -92,12 +92,10 @@ void kvm_vcpu_load_sysregs_vhe(struct kvm_vcpu *vcpu)
__sysreg_restore_el1_state(guest_ctxt);
vcpu_set_flag(vcpu, SYSREGS_ON_CPU);
activate_traps_vhe_load(vcpu);
}
/**
* kvm_vcpu_put_sysregs_vhe - Restore host system registers to the physical CPU
* __vcpu_put_switch_syregs - Restore host system registers to the physical CPU
*
* @vcpu: The VCPU pointer
*
@ -107,13 +105,12 @@ void kvm_vcpu_load_sysregs_vhe(struct kvm_vcpu *vcpu)
* and deferring saving system register state until we're no longer running the
* VCPU avoids having to save them on every exit from the VM.
*/
void kvm_vcpu_put_sysregs_vhe(struct kvm_vcpu *vcpu)
void __vcpu_put_switch_sysregs(struct kvm_vcpu *vcpu)
{
struct kvm_cpu_context *guest_ctxt = &vcpu->arch.ctxt;
struct kvm_cpu_context *host_ctxt;
host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
deactivate_traps_vhe_put(vcpu);
__sysreg_save_el1_state(guest_ctxt);
__sysreg_save_user_state(guest_ctxt);

18
arch/arm64/kvm/hyp/vhe/tlb.c
View File

@ -11,18 +11,25 @@
#include <asm/tlbflush.h>
struct tlb_inv_context {
unsigned long flags;
u64 tcr;
u64 sctlr;
struct kvm_s2_mmu *mmu;
unsigned long flags;
u64 tcr;
u64 sctlr;
};
static void __tlb_switch_to_guest(struct kvm_s2_mmu *mmu,
struct tlb_inv_context *cxt)
{
struct kvm_vcpu *vcpu = kvm_get_running_vcpu();
u64 val;
local_irq_save(cxt->flags);
if (vcpu && mmu != vcpu->arch.hw_mmu)
cxt->mmu = vcpu->arch.hw_mmu;
else
cxt->mmu = NULL;
if (cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT)) {
/*
* For CPUs that are affected by ARM errata 1165522 or 1530923,
@ -66,10 +73,13 @@ static void __tlb_switch_to_host(struct tlb_inv_context *cxt)
* We're done with the TLB operation, let's restore the host's
* view of HCR_EL2.
*/
write_sysreg(0, vttbr_el2);
write_sysreg(HCR_HOST_VHE_FLAGS, hcr_el2);
isb();
/* ... and the stage-2 MMU context that we switched away from */
if (cxt->mmu)
__load_stage2(cxt->mmu, cxt->mmu->arch);
if (cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT)) {
/* Restore the registers to what they were */
write_sysreg_el1(cxt->tcr, SYS_TCR);

36
arch/arm64/kvm/hypercalls.c
View File

@ -133,12 +133,10 @@ static bool kvm_smccc_test_fw_bmap(struct kvm_vcpu *vcpu, u32 func_id)
ARM_SMCCC_SMC_64, \
0, ARM_SMCCC_FUNC_MASK)
static void init_smccc_filter(struct kvm *kvm)
static int kvm_smccc_filter_insert_reserved(struct kvm *kvm)
{
int r;
mt_init(&kvm->arch.smccc_filter);
/*
* Prevent userspace from handling any SMCCC calls in the architecture
* range, avoiding the risk of misrepresenting Spectre mitigation status
@ -148,14 +146,25 @@ static void init_smccc_filter(struct kvm *kvm)
SMC32_ARCH_RANGE_BEGIN, SMC32_ARCH_RANGE_END,
xa_mk_value(KVM_SMCCC_FILTER_HANDLE),
GFP_KERNEL_ACCOUNT);
WARN_ON_ONCE(r);
if (r)
goto out_destroy;
r = mtree_insert_range(&kvm->arch.smccc_filter,
SMC64_ARCH_RANGE_BEGIN, SMC64_ARCH_RANGE_END,
xa_mk_value(KVM_SMCCC_FILTER_HANDLE),
GFP_KERNEL_ACCOUNT);
WARN_ON_ONCE(r);
if (r)
goto out_destroy;
return 0;
out_destroy:
mtree_destroy(&kvm->arch.smccc_filter);
return r;
}
static bool kvm_smccc_filter_configured(struct kvm *kvm)
{
return !mtree_empty(&kvm->arch.smccc_filter);
}
static int kvm_smccc_set_filter(struct kvm *kvm, struct kvm_smccc_filter __user *uaddr)
@ -184,13 +193,14 @@ static int kvm_smccc_set_filter(struct kvm *kvm, struct kvm_smccc_filter __user
goto out_unlock;
}
if (!kvm_smccc_filter_configured(kvm)) {
r = kvm_smccc_filter_insert_reserved(kvm);
if (WARN_ON_ONCE(r))
goto out_unlock;
}
r = mtree_insert_range(&kvm->arch.smccc_filter, start, end,
xa_mk_value(filter.action), GFP_KERNEL_ACCOUNT);
if (r)
goto out_unlock;
set_bit(KVM_ARCH_FLAG_SMCCC_FILTER_CONFIGURED, &kvm->arch.flags);
out_unlock:
mutex_unlock(&kvm->arch.config_lock);
return r;
@ -201,7 +211,7 @@ static u8 kvm_smccc_filter_get_action(struct kvm *kvm, u32 func_id)
unsigned long idx = func_id;
void *val;
if (!test_bit(KVM_ARCH_FLAG_SMCCC_FILTER_CONFIGURED, &kvm->arch.flags))
if (!kvm_smccc_filter_configured(kvm))
return KVM_SMCCC_FILTER_HANDLE;
/*
@ -387,7 +397,7 @@ void kvm_arm_init_hypercalls(struct kvm *kvm)
smccc_feat->std_hyp_bmap = KVM_ARM_SMCCC_STD_HYP_FEATURES;
smccc_feat->vendor_hyp_bmap = KVM_ARM_SMCCC_VENDOR_HYP_FEATURES;
init_smccc_filter(kvm);
mt_init(&kvm->arch.smccc_filter);
}
void kvm_arm_teardown_hypercalls(struct kvm *kvm)
@ -554,7 +564,7 @@ int kvm_arm_set_fw_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
{
bool wants_02;
wants_02 = test_bit(KVM_ARM_VCPU_PSCI_0_2, vcpu->arch.features);
wants_02 = vcpu_has_feature(vcpu, KVM_ARM_VCPU_PSCI_0_2);
switch (val) {
case KVM_ARM_PSCI_0_1:

4
arch/arm64/kvm/mmio.c
View File

@ -135,6 +135,9 @@ int io_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa)
* volunteered to do so, and bail out otherwise.
*/
if (!kvm_vcpu_dabt_isvalid(vcpu)) {
trace_kvm_mmio_nisv(*vcpu_pc(vcpu), kvm_vcpu_get_esr(vcpu),
kvm_vcpu_get_hfar(vcpu), fault_ipa);
if (test_bit(KVM_ARCH_FLAG_RETURN_NISV_IO_ABORT_TO_USER,
&vcpu->kvm->arch.flags)) {
run->exit_reason = KVM_EXIT_ARM_NISV;
@ -143,7 +146,6 @@ int io_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa)
return 0;
}
kvm_pr_unimpl("Data abort outside memslots with no valid syndrome info\n");
return -ENOSYS;
}

33
arch/arm64/kvm/mmu.c
View File

@ -892,7 +892,7 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t
mmfr0 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1);
mmfr1 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR1_EL1);
kvm->arch.vtcr = kvm_get_vtcr(mmfr0, mmfr1, phys_shift);
mmu->vtcr = kvm_get_vtcr(mmfr0, mmfr1, phys_shift);
if (mmu->pgt != NULL) {
kvm_err("kvm_arch already initialized?\n");
@ -1067,7 +1067,8 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa,
phys_addr_t addr;
int ret = 0;
struct kvm_mmu_memory_cache cache = { .gfp_zero = __GFP_ZERO };
struct kvm_pgtable *pgt = kvm->arch.mmu.pgt;
struct kvm_s2_mmu *mmu = &kvm->arch.mmu;
struct kvm_pgtable *pgt = mmu->pgt;
enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_DEVICE |
KVM_PGTABLE_PROT_R |
(writable ? KVM_PGTABLE_PROT_W : 0);
@ -1080,7 +1081,7 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa,
for (addr = guest_ipa; addr < guest_ipa + size; addr += PAGE_SIZE) {
ret = kvm_mmu_topup_memory_cache(&cache,
kvm_mmu_cache_min_pages(kvm));
kvm_mmu_cache_min_pages(mmu));
if (ret)
break;
@ -1298,28 +1299,8 @@ transparent_hugepage_adjust(struct kvm *kvm, struct kvm_memory_slot *memslot,
if (sz < PMD_SIZE)
return PAGE_SIZE;
/*
* The address we faulted on is backed by a transparent huge
* page. However, because we map the compound huge page and
* not the individual tail page, we need to transfer the
* refcount to the head page. We have to be careful that the
* THP doesn't start to split while we are adjusting the
* refcounts.
*
* We are sure this doesn't happen, because mmu_invalidate_retry
* was successful and we are holding the mmu_lock, so if this
* THP is trying to split, it will be blocked in the mmu
* notifier before touching any of the pages, specifically
* before being able to call __split_huge_page_refcount().
*
* We can therefore safely transfer the refcount from PG_tail
* to PG_head and switch the pfn from a tail page to the head
* page accordingly.
*/
*ipap &= PMD_MASK;
kvm_release_pfn_clean(pfn);
pfn &= ~(PTRS_PER_PMD - 1);
get_page(pfn_to_page(pfn));
*pfnp = pfn;
return PMD_SIZE;
@ -1431,7 +1412,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
if (fault_status != ESR_ELx_FSC_PERM ||
(logging_active && write_fault)) {
ret = kvm_mmu_topup_memory_cache(memcache,
kvm_mmu_cache_min_pages(kvm));
kvm_mmu_cache_min_pages(vcpu->arch.hw_mmu));
if (ret)
return ret;
}
@ -1747,7 +1728,7 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu)
}
/* Userspace should not be able to register out-of-bounds IPAs */
VM_BUG_ON(fault_ipa >= kvm_phys_size(vcpu->kvm));
VM_BUG_ON(fault_ipa >= kvm_phys_size(vcpu->arch.hw_mmu));
if (fault_status == ESR_ELx_FSC_ACCESS) {
handle_access_fault(vcpu, fault_ipa);
@ -2021,7 +2002,7 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,
* Prevent userspace from creating a memory region outside of the IPA
* space addressable by the KVM guest IPA space.
*/
if ((new->base_gfn + new->npages) > (kvm_phys_size(kvm) >> PAGE_SHIFT))
if ((new->base_gfn + new->npages) > (kvm_phys_size(&kvm->arch.mmu) >> PAGE_SHIFT))
return -EFAULT;
hva = new->userspace_addr;

2
arch/arm64/kvm/pkvm.c
View File

@ -123,7 +123,7 @@ static int __pkvm_create_hyp_vm(struct kvm *host_kvm)
if (host_kvm->created_vcpus < 1)
return -EINVAL;
pgd_sz = kvm_pgtable_stage2_pgd_size(host_kvm->arch.vtcr);
pgd_sz = kvm_pgtable_stage2_pgd_size(host_kvm->arch.mmu.vtcr);
/*
* The PGD pages will be reclaimed using a hyp_memcache which implies

145
arch/arm64/kvm/pmu-emul.c
View File

@ -60,6 +60,23 @@ static u32 kvm_pmu_event_mask(struct kvm *kvm)
return __kvm_pmu_event_mask(pmuver);
}
u64 kvm_pmu_evtyper_mask(struct kvm *kvm)
{
u64 mask = ARMV8_PMU_EXCLUDE_EL1 | ARMV8_PMU_EXCLUDE_EL0 |
kvm_pmu_event_mask(kvm);
u64 pfr0 = IDREG(kvm, SYS_ID_AA64PFR0_EL1);
if (SYS_FIELD_GET(ID_AA64PFR0_EL1, EL2, pfr0))
mask |= ARMV8_PMU_INCLUDE_EL2;
if (SYS_FIELD_GET(ID_AA64PFR0_EL1, EL3, pfr0))
mask |= ARMV8_PMU_EXCLUDE_NS_EL0 |
ARMV8_PMU_EXCLUDE_NS_EL1 |
ARMV8_PMU_EXCLUDE_EL3;
return mask;
}
/**
* kvm_pmc_is_64bit - determine if counter is 64bit
* @pmc: counter context
@ -72,7 +89,7 @@ static bool kvm_pmc_is_64bit(struct kvm_pmc *pmc)
static bool kvm_pmc_has_64bit_overflow(struct kvm_pmc *pmc)
{
u64 val = __vcpu_sys_reg(kvm_pmc_to_vcpu(pmc), PMCR_EL0);
u64 val = kvm_vcpu_read_pmcr(kvm_pmc_to_vcpu(pmc));
return (pmc->idx < ARMV8_PMU_CYCLE_IDX && (val & ARMV8_PMU_PMCR_LP)) ||
(pmc->idx == ARMV8_PMU_CYCLE_IDX && (val & ARMV8_PMU_PMCR_LC));
@ -250,7 +267,7 @@ void kvm_pmu_vcpu_destroy(struct kvm_vcpu *vcpu)
u64 kvm_pmu_valid_counter_mask(struct kvm_vcpu *vcpu)
{
u64 val = __vcpu_sys_reg(vcpu, PMCR_EL0) >> ARMV8_PMU_PMCR_N_SHIFT;
u64 val = kvm_vcpu_read_pmcr(vcpu) >> ARMV8_PMU_PMCR_N_SHIFT;
val &= ARMV8_PMU_PMCR_N_MASK;
if (val == 0)
@ -272,7 +289,7 @@ void kvm_pmu_enable_counter_mask(struct kvm_vcpu *vcpu, u64 val)
if (!kvm_vcpu_has_pmu(vcpu))
return;
if (!(__vcpu_sys_reg(vcpu, PMCR_EL0) & ARMV8_PMU_PMCR_E) || !val)
if (!(kvm_vcpu_read_pmcr(vcpu) & ARMV8_PMU_PMCR_E) || !val)
return;
for (i = 0; i < ARMV8_PMU_MAX_COUNTERS; i++) {
@ -324,7 +341,7 @@ static u64 kvm_pmu_overflow_status(struct kvm_vcpu *vcpu)
{
u64 reg = 0;
if ((__vcpu_sys_reg(vcpu, PMCR_EL0) & ARMV8_PMU_PMCR_E)) {
if ((kvm_vcpu_read_pmcr(vcpu) & ARMV8_PMU_PMCR_E)) {
reg = __vcpu_sys_reg(vcpu, PMOVSSET_EL0);
reg &= __vcpu_sys_reg(vcpu, PMCNTENSET_EL0);
reg &= __vcpu_sys_reg(vcpu, PMINTENSET_EL1);
@ -348,7 +365,7 @@ static void kvm_pmu_update_state(struct kvm_vcpu *vcpu)
pmu->irq_level = overflow;
if (likely(irqchip_in_kernel(vcpu->kvm))) {
int ret = kvm_vgic_inject_irq(vcpu->kvm, vcpu->vcpu_id,
int ret = kvm_vgic_inject_irq(vcpu->kvm, vcpu,
pmu->irq_num, overflow, pmu);
WARN_ON(ret);
}
@ -426,7 +443,7 @@ static void kvm_pmu_counter_increment(struct kvm_vcpu *vcpu,
{
int i;
if (!(__vcpu_sys_reg(vcpu, PMCR_EL0) & ARMV8_PMU_PMCR_E))
if (!(kvm_vcpu_read_pmcr(vcpu) & ARMV8_PMU_PMCR_E))
return;
/* Weed out disabled counters */
@ -569,7 +586,7 @@ void kvm_pmu_handle_pmcr(struct kvm_vcpu *vcpu, u64 val)
static bool kvm_pmu_counter_is_enabled(struct kvm_pmc *pmc)
{
struct kvm_vcpu *vcpu = kvm_pmc_to_vcpu(pmc);
return (__vcpu_sys_reg(vcpu, PMCR_EL0) & ARMV8_PMU_PMCR_E) &&
return (kvm_vcpu_read_pmcr(vcpu) & ARMV8_PMU_PMCR_E) &&
(__vcpu_sys_reg(vcpu, PMCNTENSET_EL0) & BIT(pmc->idx));
}
@ -584,6 +601,7 @@ static void kvm_pmu_create_perf_event(struct kvm_pmc *pmc)
struct perf_event *event;
struct perf_event_attr attr;
u64 eventsel, reg, data;
bool p, u, nsk, nsu;
reg = counter_index_to_evtreg(pmc->idx);
data = __vcpu_sys_reg(vcpu, reg);
@ -610,13 +628,18 @@ static void kvm_pmu_create_perf_event(struct kvm_pmc *pmc)
!test_bit(eventsel, vcpu->kvm->arch.pmu_filter))
return;
p = data & ARMV8_PMU_EXCLUDE_EL1;
u = data & ARMV8_PMU_EXCLUDE_EL0;
nsk = data & ARMV8_PMU_EXCLUDE_NS_EL1;
nsu = data & ARMV8_PMU_EXCLUDE_NS_EL0;
memset(&attr, 0, sizeof(struct perf_event_attr));
attr.type = arm_pmu->pmu.type;
attr.size = sizeof(attr);
attr.pinned = 1;
attr.disabled = !kvm_pmu_counter_is_enabled(pmc);
attr.exclude_user = data & ARMV8_PMU_EXCLUDE_EL0 ? 1 : 0;
attr.exclude_kernel = data & ARMV8_PMU_EXCLUDE_EL1 ? 1 : 0;
attr.exclude_user = (u != nsu);
attr.exclude_kernel = (p != nsk);
attr.exclude_hv = 1; /* Don't count EL2 events */
attr.exclude_host = 1; /* Don't count host events */
attr.config = eventsel;
@ -657,18 +680,13 @@ void kvm_pmu_set_counter_event_type(struct kvm_vcpu *vcpu, u64 data,
u64 select_idx)
{
struct kvm_pmc *pmc = kvm_vcpu_idx_to_pmc(vcpu, select_idx);
u64 reg, mask;
u64 reg;
if (!kvm_vcpu_has_pmu(vcpu))
return;
mask = ARMV8_PMU_EVTYPE_MASK;
mask &= ~ARMV8_PMU_EVTYPE_EVENT;
mask |= kvm_pmu_event_mask(vcpu->kvm);
reg = counter_index_to_evtreg(pmc->idx);
__vcpu_sys_reg(vcpu, reg) = data & mask;
__vcpu_sys_reg(vcpu, reg) = data & kvm_pmu_evtyper_mask(vcpu->kvm);
kvm_pmu_create_perf_event(pmc);
}
@ -717,10 +735,9 @@ static struct arm_pmu *kvm_pmu_probe_armpmu(void)
* It is still necessary to get a valid cpu, though, to probe for the
* default PMU instance as userspace is not required to specify a PMU
* type. In order to uphold the preexisting behavior KVM selects the
* PMU instance for the core where the first call to the
* KVM_ARM_VCPU_PMU_V3_CTRL attribute group occurs. A dependent use case
* would be a user with disdain of all things big.LITTLE that affines
* the VMM to a particular cluster of cores.
* PMU instance for the core during vcpu init. A dependent use
* case would be a user with disdain of all things big.LITTLE that
* affines the VMM to a particular cluster of cores.
*
* In any case, userspace should just do the sane thing and use the UAPI
* to select a PMU type directly. But, be wary of the baggage being
@ -786,6 +803,17 @@ u64 kvm_pmu_get_pmceid(struct kvm_vcpu *vcpu, bool pmceid1)
return val & mask;
}
void kvm_vcpu_reload_pmu(struct kvm_vcpu *vcpu)
{
u64 mask = kvm_pmu_valid_counter_mask(vcpu);
kvm_pmu_handle_pmcr(vcpu, kvm_vcpu_read_pmcr(vcpu));
__vcpu_sys_reg(vcpu, PMOVSSET_EL0) &= mask;
__vcpu_sys_reg(vcpu, PMINTENSET_EL1) &= mask;
__vcpu_sys_reg(vcpu, PMCNTENSET_EL0) &= mask;
}
int kvm_arm_pmu_v3_enable(struct kvm_vcpu *vcpu)
{
if (!kvm_vcpu_has_pmu(vcpu))
@ -874,6 +902,52 @@ static bool pmu_irq_is_valid(struct kvm *kvm, int irq)
return true;
}
/**
* kvm_arm_pmu_get_max_counters - Return the max number of PMU counters.
* @kvm: The kvm pointer
*/
u8 kvm_arm_pmu_get_max_counters(struct kvm *kvm)
{
struct arm_pmu *arm_pmu = kvm->arch.arm_pmu;
/*
* The arm_pmu->num_events considers the cycle counter as well.
* Ignore that and return only the general-purpose counters.
*/
return arm_pmu->num_events - 1;
}
static void kvm_arm_set_pmu(struct kvm *kvm, struct arm_pmu *arm_pmu)
{
lockdep_assert_held(&kvm->arch.config_lock);
kvm->arch.arm_pmu = arm_pmu;
kvm->arch.pmcr_n = kvm_arm_pmu_get_max_counters(kvm);
}
/**
* kvm_arm_set_default_pmu - No PMU set, get the default one.
* @kvm: The kvm pointer
*
* The observant among you will notice that the supported_cpus
* mask does not get updated for the default PMU even though it
* is quite possible the selected instance supports only a
* subset of cores in the system. This is intentional, and
* upholds the preexisting behavior on heterogeneous systems
* where vCPUs can be scheduled on any core but the guest
* counters could stop working.
*/
int kvm_arm_set_default_pmu(struct kvm *kvm)
{
struct arm_pmu *arm_pmu = kvm_pmu_probe_armpmu();
if (!arm_pmu)
return -ENODEV;
kvm_arm_set_pmu(kvm, arm_pmu);
return 0;
}
static int kvm_arm_pmu_v3_set_pmu(struct kvm_vcpu *vcpu, int pmu_id)
{
struct kvm *kvm = vcpu->kvm;
@ -893,7 +967,7 @@ static int kvm_arm_pmu_v3_set_pmu(struct kvm_vcpu *vcpu, int pmu_id)
break;
}
kvm->arch.arm_pmu = arm_pmu;
kvm_arm_set_pmu(kvm, arm_pmu);
cpumask_copy(kvm->arch.supported_cpus, &arm_pmu->supported_cpus);
ret = 0;
break;
@ -916,23 +990,6 @@ int kvm_arm_pmu_v3_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr)
if (vcpu->arch.pmu.created)
return -EBUSY;
if (!kvm->arch.arm_pmu) {
/*
* No PMU set, get the default one.
*
* The observant among you will notice that the supported_cpus
* mask does not get updated for the default PMU even though it
* is quite possible the selected instance supports only a
* subset of cores in the system. This is intentional, and
* upholds the preexisting behavior on heterogeneous systems
* where vCPUs can be scheduled on any core but the guest
* counters could stop working.
*/
kvm->arch.arm_pmu = kvm_pmu_probe_armpmu();
if (!kvm->arch.arm_pmu)
return -ENODEV;
}
switch (attr->attr) {
case KVM_ARM_VCPU_PMU_V3_IRQ: {
int __user *uaddr = (int __user *)(long)attr->addr;
@ -1072,3 +1129,15 @@ u8 kvm_arm_pmu_get_pmuver_limit(void)
ID_AA64DFR0_EL1_PMUVer_V3P5);
return FIELD_GET(ARM64_FEATURE_MASK(ID_AA64DFR0_EL1_PMUVer), tmp);
}
/**
* kvm_vcpu_read_pmcr - Read PMCR_EL0 register for the vCPU
* @vcpu: The vcpu pointer
*/
u64 kvm_vcpu_read_pmcr(struct kvm_vcpu *vcpu)
{
u64 pmcr = __vcpu_sys_reg(vcpu, PMCR_EL0) &
~(ARMV8_PMU_PMCR_N_MASK << ARMV8_PMU_PMCR_N_SHIFT);
return pmcr | ((u64)vcpu->kvm->arch.pmcr_n << ARMV8_PMU_PMCR_N_SHIFT);
}

56
arch/arm64/kvm/reset.c
View File

@ -73,11 +73,8 @@ int __init kvm_arm_init_sve(void)
return 0;
}
static int kvm_vcpu_enable_sve(struct kvm_vcpu *vcpu)
static void kvm_vcpu_enable_sve(struct kvm_vcpu *vcpu)
{
if (!system_supports_sve())
return -EINVAL;
vcpu->arch.sve_max_vl = kvm_sve_max_vl;
/*
@ -86,8 +83,6 @@ static int kvm_vcpu_enable_sve(struct kvm_vcpu *vcpu)
* kvm_arm_vcpu_finalize(), which freezes the configuration.
*/
vcpu_set_flag(vcpu, GUEST_HAS_SVE);
return 0;
}
/*
@ -170,20 +165,9 @@ static void kvm_vcpu_reset_sve(struct kvm_vcpu *vcpu)
memset(vcpu->arch.sve_state, 0, vcpu_sve_state_size(vcpu));
}
static int kvm_vcpu_enable_ptrauth(struct kvm_vcpu *vcpu)
static void kvm_vcpu_enable_ptrauth(struct kvm_vcpu *vcpu)
{
/*
* For now make sure that both address/generic pointer authentication
* features are requested by the userspace together and the system
* supports these capabilities.
*/
if (!test_bit(KVM_ARM_VCPU_PTRAUTH_ADDRESS, vcpu->arch.features) ||
!test_bit(KVM_ARM_VCPU_PTRAUTH_GENERIC, vcpu->arch.features) ||
!system_has_full_ptr_auth())
return -EINVAL;
vcpu_set_flag(vcpu, GUEST_HAS_PTRAUTH);
return 0;
}
/**
@ -204,10 +188,9 @@ static int kvm_vcpu_enable_ptrauth(struct kvm_vcpu *vcpu)
* disable preemption around the vcpu reset as we would otherwise race with
* preempt notifiers which also call put/load.
*/
int kvm_reset_vcpu(struct kvm_vcpu *vcpu)
void kvm_reset_vcpu(struct kvm_vcpu *vcpu)
{
struct vcpu_reset_state reset_state;
int ret;
bool loaded;
u32 pstate;
@ -224,29 +207,16 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu)
if (loaded)
kvm_arch_vcpu_put(vcpu);
/* Disallow NV+SVE for the time being */
if (vcpu_has_nv(vcpu) && vcpu_has_feature(vcpu, KVM_ARM_VCPU_SVE)) {
ret = -EINVAL;
goto out;
}
if (!kvm_arm_vcpu_sve_finalized(vcpu)) {
if (test_bit(KVM_ARM_VCPU_SVE, vcpu->arch.features)) {
ret = kvm_vcpu_enable_sve(vcpu);
if (ret)
goto out;
}
if (vcpu_has_feature(vcpu, KVM_ARM_VCPU_SVE))
kvm_vcpu_enable_sve(vcpu);
} else {
kvm_vcpu_reset_sve(vcpu);
}
if (test_bit(KVM_ARM_VCPU_PTRAUTH_ADDRESS, vcpu->arch.features) ||
test_bit(KVM_ARM_VCPU_PTRAUTH_GENERIC, vcpu->arch.features)) {
if (kvm_vcpu_enable_ptrauth(vcpu)) {
ret = -EINVAL;
goto out;
}
}
if (vcpu_has_feature(vcpu, KVM_ARM_VCPU_PTRAUTH_ADDRESS) ||
vcpu_has_feature(vcpu, KVM_ARM_VCPU_PTRAUTH_GENERIC))
kvm_vcpu_enable_ptrauth(vcpu);
if (vcpu_el1_is_32bit(vcpu))
pstate = VCPU_RESET_PSTATE_SVC;
@ -255,11 +225,6 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu)
else
pstate = VCPU_RESET_PSTATE_EL1;
if (kvm_vcpu_has_pmu(vcpu) && !kvm_arm_support_pmu_v3()) {
ret = -EINVAL;
goto out;
}
/* Reset core registers */
memset(vcpu_gp_regs(vcpu), 0, sizeof(*vcpu_gp_regs(vcpu)));
memset(&vcpu->arch.ctxt.fp_regs, 0, sizeof(vcpu->arch.ctxt.fp_regs));
@ -294,12 +259,11 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu)
}
/* Reset timer */
ret = kvm_timer_vcpu_reset(vcpu);
out:
kvm_timer_vcpu_reset(vcpu);
if (loaded)
kvm_arch_vcpu_load(vcpu, smp_processor_id());
preempt_enable();
return ret;
}
u32 get_kvm_ipa_limit(void)

353
arch/arm64/kvm/sys_regs.c
View File

@ -379,7 +379,7 @@ static bool trap_loregion(struct kvm_vcpu *vcpu,
struct sys_reg_params *p,
const struct sys_reg_desc *r)
{
u64 val = read_sanitised_ftr_reg(SYS_ID_AA64MMFR1_EL1);
u64 val = IDREG(vcpu->kvm, SYS_ID_AA64MMFR1_EL1);
u32 sr = reg_to_encoding(r);
if (!(val & (0xfUL << ID_AA64MMFR1_EL1_LO_SHIFT))) {
@ -719,14 +719,9 @@ static unsigned int pmu_visibility(const struct kvm_vcpu *vcpu,
static u64 reset_pmu_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
{
u64 n, mask = BIT(ARMV8_PMU_CYCLE_IDX);
u64 mask = BIT(ARMV8_PMU_CYCLE_IDX);
u8 n = vcpu->kvm->arch.pmcr_n;
/* No PMU available, any PMU reg may UNDEF... */
if (!kvm_arm_support_pmu_v3())
return 0;
n = read_sysreg(pmcr_el0) >> ARMV8_PMU_PMCR_N_SHIFT;
n &= ARMV8_PMU_PMCR_N_MASK;
if (n)
mask |= GENMASK(n - 1, 0);
@ -746,8 +741,12 @@ static u64 reset_pmevcntr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
static u64 reset_pmevtyper(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
{
/* This thing will UNDEF, who cares about the reset value? */
if (!kvm_vcpu_has_pmu(vcpu))
return 0;
reset_unknown(vcpu, r);
__vcpu_sys_reg(vcpu, r->reg) &= ARMV8_PMU_EVTYPE_MASK;
__vcpu_sys_reg(vcpu, r->reg) &= kvm_pmu_evtyper_mask(vcpu->kvm);
return __vcpu_sys_reg(vcpu, r->reg);
}
@ -762,17 +761,15 @@ static u64 reset_pmselr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
static u64 reset_pmcr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
{
u64 pmcr;
u64 pmcr = 0;
/* No PMU available, PMCR_EL0 may UNDEF... */
if (!kvm_arm_support_pmu_v3())
return 0;
/* Only preserve PMCR_EL0.N, and reset the rest to 0 */
pmcr = read_sysreg(pmcr_el0) & (ARMV8_PMU_PMCR_N_MASK << ARMV8_PMU_PMCR_N_SHIFT);
if (!kvm_supports_32bit_el0())
pmcr |= ARMV8_PMU_PMCR_LC;
/*
* The value of PMCR.N field is included when the
* vCPU register is read via kvm_vcpu_read_pmcr().
*/
__vcpu_sys_reg(vcpu, r->reg) = pmcr;
return __vcpu_sys_reg(vcpu, r->reg);
@ -822,7 +819,7 @@ static bool access_pmcr(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
* Only update writeable bits of PMCR (continuing into
* kvm_pmu_handle_pmcr() as well)
*/
val = __vcpu_sys_reg(vcpu, PMCR_EL0);
val = kvm_vcpu_read_pmcr(vcpu);
val &= ~ARMV8_PMU_PMCR_MASK;
val |= p->regval & ARMV8_PMU_PMCR_MASK;
if (!kvm_supports_32bit_el0())
@ -830,7 +827,7 @@ static bool access_pmcr(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
kvm_pmu_handle_pmcr(vcpu, val);
} else {
/* PMCR.P & PMCR.C are RAZ */
val = __vcpu_sys_reg(vcpu, PMCR_EL0)
val = kvm_vcpu_read_pmcr(vcpu)
& ~(ARMV8_PMU_PMCR_P | ARMV8_PMU_PMCR_C);
p->regval = val;
}
@ -879,7 +876,7 @@ static bool pmu_counter_idx_valid(struct kvm_vcpu *vcpu, u64 idx)
{
u64 pmcr, val;
pmcr = __vcpu_sys_reg(vcpu, PMCR_EL0);
pmcr = kvm_vcpu_read_pmcr(vcpu);
val = (pmcr >> ARMV8_PMU_PMCR_N_SHIFT) & ARMV8_PMU_PMCR_N_MASK;
if (idx >= val && idx != ARMV8_PMU_CYCLE_IDX) {
kvm_inject_undefined(vcpu);
@ -988,12 +985,45 @@ static bool access_pmu_evtyper(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
kvm_pmu_set_counter_event_type(vcpu, p->regval, idx);
kvm_vcpu_pmu_restore_guest(vcpu);
} else {
p->regval = __vcpu_sys_reg(vcpu, reg) & ARMV8_PMU_EVTYPE_MASK;
p->regval = __vcpu_sys_reg(vcpu, reg);
}
return true;
}
static int set_pmreg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r, u64 val)
{
bool set;
val &= kvm_pmu_valid_counter_mask(vcpu);
switch (r->reg) {
case PMOVSSET_EL0:
/* CRm[1] being set indicates a SET register, and CLR otherwise */
set = r->CRm & 2;
break;
default:
/* Op2[0] being set indicates a SET register, and CLR otherwise */
set = r->Op2 & 1;
break;
}
if (set)
__vcpu_sys_reg(vcpu, r->reg) |= val;
else
__vcpu_sys_reg(vcpu, r->reg) &= ~val;
return 0;
}
static int get_pmreg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r, u64 *val)
{
u64 mask = kvm_pmu_valid_counter_mask(vcpu);
*val = __vcpu_sys_reg(vcpu, r->reg) & mask;
return 0;
}
static bool access_pmcnten(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
const struct sys_reg_desc *r)
{
@ -1103,6 +1133,51 @@ static bool access_pmuserenr(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
return true;
}
static int get_pmcr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r,
u64 *val)
{
*val = kvm_vcpu_read_pmcr(vcpu);
return 0;
}
static int set_pmcr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r,
u64 val)
{
u8 new_n = (val >> ARMV8_PMU_PMCR_N_SHIFT) & ARMV8_PMU_PMCR_N_MASK;
struct kvm *kvm = vcpu->kvm;
mutex_lock(&kvm->arch.config_lock);
/*
* The vCPU can't have more counters than the PMU hardware
* implements. Ignore this error to maintain compatibility
* with the existing KVM behavior.
*/
if (!kvm_vm_has_ran_once(kvm) &&
new_n <= kvm_arm_pmu_get_max_counters(kvm))
kvm->arch.pmcr_n = new_n;
mutex_unlock(&kvm->arch.config_lock);
/*
* Ignore writes to RES0 bits, read only bits that are cleared on
* vCPU reset, and writable bits that KVM doesn't support yet.
* (i.e. only PMCR.N and bits [7:0] are mutable from userspace)
* The LP bit is RES0 when FEAT_PMUv3p5 is not supported on the vCPU.
* But, we leave the bit as it is here, as the vCPU's PMUver might
* be changed later (NOTE: the bit will be cleared on first vCPU run
* if necessary).
*/
val &= ARMV8_PMU_PMCR_MASK;
/* The LC bit is RES1 when AArch32 is not supported */
if (!kvm_supports_32bit_el0())
val |= ARMV8_PMU_PMCR_LC;
__vcpu_sys_reg(vcpu, r->reg) = val;
return 0;
}
/* Silly macro to expand the DBG{BCR,BVR,WVR,WCR}n_EL1 registers in one go */
#define DBG_BCR_BVR_WCR_WVR_EL1(n) \
{ SYS_DESC(SYS_DBGBVRn_EL1(n)), \
@ -1216,8 +1291,14 @@ static s64 kvm_arm64_ftr_safe_value(u32 id, const struct arm64_ftr_bits *ftrp,
/* Some features have different safe value type in KVM than host features */
switch (id) {
case SYS_ID_AA64DFR0_EL1:
if (kvm_ftr.shift == ID_AA64DFR0_EL1_PMUVer_SHIFT)
switch (kvm_ftr.shift) {
case ID_AA64DFR0_EL1_PMUVer_SHIFT:
kvm_ftr.type = FTR_LOWER_SAFE;
break;
case ID_AA64DFR0_EL1_DebugVer_SHIFT:
kvm_ftr.type = FTR_LOWER_SAFE;
break;
}
break;
case SYS_ID_DFR0_EL1:
if (kvm_ftr.shift == ID_DFR0_EL1_PerfMon_SHIFT)
@ -1228,7 +1309,7 @@ static s64 kvm_arm64_ftr_safe_value(u32 id, const struct arm64_ftr_bits *ftrp,
return arm64_ftr_safe_value(&kvm_ftr, new, cur);
}
/**
/*
* arm64_check_features() - Check if a feature register value constitutes
* a subset of features indicated by the idreg's KVM sanitised limit.
*
@ -1338,7 +1419,6 @@ static u64 __kvm_read_sanitised_id_reg(const struct kvm_vcpu *vcpu,
ARM64_FEATURE_MASK(ID_AA64ISAR2_EL1_GPA3));
if (!cpus_have_final_cap(ARM64_HAS_WFXT))
val &= ~ARM64_FEATURE_MASK(ID_AA64ISAR2_EL1_WFxT);
val &= ~ARM64_FEATURE_MASK(ID_AA64ISAR2_EL1_MOPS);
break;
case SYS_ID_AA64MMFR2_EL1:
val &= ~ID_AA64MMFR2_EL1_CCIDX_MASK;
@ -1373,6 +1453,13 @@ static inline bool is_id_reg(u32 id)
sys_reg_CRm(id) < 8);
}
static inline bool is_aa32_id_reg(u32 id)
{
return (sys_reg_Op0(id) == 3 && sys_reg_Op1(id) == 0 &&
sys_reg_CRn(id) == 0 && sys_reg_CRm(id) >= 1 &&
sys_reg_CRm(id) <= 3);
}
static unsigned int id_visibility(const struct kvm_vcpu *vcpu,
const struct sys_reg_desc *r)
{
@ -1469,14 +1556,21 @@ static u64 read_sanitised_id_aa64pfr0_el1(struct kvm_vcpu *vcpu,
return val;
}
#define ID_REG_LIMIT_FIELD_ENUM(val, reg, field, limit) \
({ \
u64 __f_val = FIELD_GET(reg##_##field##_MASK, val); \
(val) &= ~reg##_##field##_MASK; \
(val) |= FIELD_PREP(reg##_##field##_MASK, \
min(__f_val, (u64)reg##_##field##_##limit)); \
(val); \
})
static u64 read_sanitised_id_aa64dfr0_el1(struct kvm_vcpu *vcpu,
const struct sys_reg_desc *rd)
{
u64 val = read_sanitised_ftr_reg(SYS_ID_AA64DFR0_EL1);
/* Limit debug to ARMv8.0 */
val &= ~ID_AA64DFR0_EL1_DebugVer_MASK;
val |= SYS_FIELD_PREP_ENUM(ID_AA64DFR0_EL1, DebugVer, IMP);
val = ID_REG_LIMIT_FIELD_ENUM(val, ID_AA64DFR0_EL1, DebugVer, V8P8);
/*
* Only initialize the PMU version if the vCPU was configured with one.
@ -1496,6 +1590,7 @@ static int set_id_aa64dfr0_el1(struct kvm_vcpu *vcpu,
const struct sys_reg_desc *rd,
u64 val)
{
u8 debugver = SYS_FIELD_GET(ID_AA64DFR0_EL1, DebugVer, val);
u8 pmuver = SYS_FIELD_GET(ID_AA64DFR0_EL1, PMUVer, val);
/*
@ -1515,6 +1610,13 @@ static int set_id_aa64dfr0_el1(struct kvm_vcpu *vcpu,
if (pmuver == ID_AA64DFR0_EL1_PMUVer_IMP_DEF)
val &= ~ID_AA64DFR0_EL1_PMUVer_MASK;
/*
* ID_AA64DFR0_EL1.DebugVer is one of those awkward fields with a
* nonzero minimum safe value.
*/
if (debugver < ID_AA64DFR0_EL1_DebugVer_IMP)
return -EINVAL;
return set_id_reg(vcpu, rd, val);
}
@ -1528,6 +1630,8 @@ static u64 read_sanitised_id_dfr0_el1(struct kvm_vcpu *vcpu,
if (kvm_vcpu_has_pmu(vcpu))
val |= SYS_FIELD_PREP(ID_DFR0_EL1, PerfMon, perfmon);
val = ID_REG_LIMIT_FIELD_ENUM(val, ID_DFR0_EL1, CopDbg, Debugv8p8);
return val;
}
@ -1536,6 +1640,7 @@ static int set_id_dfr0_el1(struct kvm_vcpu *vcpu,
u64 val)
{
u8 perfmon = SYS_FIELD_GET(ID_DFR0_EL1, PerfMon, val);
u8 copdbg = SYS_FIELD_GET(ID_DFR0_EL1, CopDbg, val);
if (perfmon == ID_DFR0_EL1_PerfMon_IMPDEF) {
val &= ~ID_DFR0_EL1_PerfMon_MASK;
@ -1551,6 +1656,9 @@ static int set_id_dfr0_el1(struct kvm_vcpu *vcpu,
if (perfmon != 0 && perfmon < ID_DFR0_EL1_PerfMon_PMUv3)
return -EINVAL;
if (copdbg < ID_DFR0_EL1_CopDbg_Armv8)
return -EINVAL;
return set_id_reg(vcpu, rd, val);
}
@ -1791,8 +1899,8 @@ static unsigned int el2_visibility(const struct kvm_vcpu *vcpu,
* HCR_EL2.E2H==1, and only in the sysreg table for convenience of
* handling traps. Given that, they are always hidden from userspace.
*/
static unsigned int elx2_visibility(const struct kvm_vcpu *vcpu,
const struct sys_reg_desc *rd)
static unsigned int hidden_user_visibility(const struct kvm_vcpu *vcpu,
const struct sys_reg_desc *rd)
{
return REG_HIDDEN_USER;
}
@ -1803,7 +1911,7 @@ static unsigned int elx2_visibility(const struct kvm_vcpu *vcpu,
.reset = rst, \
.reg = name##_EL1, \
.val = v, \
.visibility = elx2_visibility, \
.visibility = hidden_user_visibility, \
}
/*
@ -1817,11 +1925,14 @@ static unsigned int elx2_visibility(const struct kvm_vcpu *vcpu,
* from userspace.
*/
/* sys_reg_desc initialiser for known cpufeature ID registers */
#define ID_SANITISED(name) { \
#define ID_DESC(name) \
SYS_DESC(SYS_##name), \
.access = access_id_reg, \
.get_user = get_id_reg, \
.get_user = get_id_reg \
/* sys_reg_desc initialiser for known cpufeature ID registers */
#define ID_SANITISED(name) { \
ID_DESC(name), \
.set_user = set_id_reg, \
.visibility = id_visibility, \
.reset = kvm_read_sanitised_id_reg, \
@ -1830,15 +1941,22 @@ static unsigned int elx2_visibility(const struct kvm_vcpu *vcpu,
/* sys_reg_desc initialiser for known cpufeature ID registers */
#define AA32_ID_SANITISED(name) { \
SYS_DESC(SYS_##name), \
.access = access_id_reg, \
.get_user = get_id_reg, \
ID_DESC(name), \
.set_user = set_id_reg, \
.visibility = aa32_id_visibility, \
.reset = kvm_read_sanitised_id_reg, \
.val = 0, \
}
/* sys_reg_desc initialiser for writable ID registers */
#define ID_WRITABLE(name, mask) { \
ID_DESC(name), \
.set_user = set_id_reg, \
.visibility = id_visibility, \
.reset = kvm_read_sanitised_id_reg, \
.val = mask, \
}
/*
* sys_reg_desc initialiser for architecturally unallocated cpufeature ID
* register with encoding Op0=3, Op1=0, CRn=0, CRm=crm, Op2=op2
@ -1860,9 +1978,7 @@ static unsigned int elx2_visibility(const struct kvm_vcpu *vcpu,
* RAZ for the guest.
*/
#define ID_HIDDEN(name) { \
SYS_DESC(SYS_##name), \
.access = access_id_reg, \
.get_user = get_id_reg, \
ID_DESC(name), \
.set_user = set_id_reg, \
.visibility = raz_visibility, \
.reset = kvm_read_sanitised_id_reg, \
@ -1961,7 +2077,7 @@ static const struct sys_reg_desc sys_reg_descs[] = {
// DBGDTR[TR]X_EL0 share the same encoding
{ SYS_DESC(SYS_DBGDTRTX_EL0), trap_raz_wi },
{ SYS_DESC(SYS_DBGVCR32_EL2), NULL, reset_val, DBGVCR32_EL2, 0 },
{ SYS_DESC(SYS_DBGVCR32_EL2), trap_undef, reset_val, DBGVCR32_EL2, 0 },
{ SYS_DESC(SYS_MPIDR_EL1), NULL, reset_mpidr, MPIDR_EL1 },
@ -1980,7 +2096,8 @@ static const struct sys_reg_desc sys_reg_descs[] = {
.set_user = set_id_dfr0_el1,
.visibility = aa32_id_visibility,
.reset = read_sanitised_id_dfr0_el1,
.val = ID_DFR0_EL1_PerfMon_MASK, },
.val = ID_DFR0_EL1_PerfMon_MASK |
ID_DFR0_EL1_CopDbg_MASK, },
ID_HIDDEN(ID_AFR0_EL1),
AA32_ID_SANITISED(ID_MMFR0_EL1),
AA32_ID_SANITISED(ID_MMFR1_EL1),
@ -2014,11 +2131,17 @@ static const struct sys_reg_desc sys_reg_descs[] = {
.get_user = get_id_reg,
.set_user = set_id_reg,
.reset = read_sanitised_id_aa64pfr0_el1,
.val = ID_AA64PFR0_EL1_CSV2_MASK | ID_AA64PFR0_EL1_CSV3_MASK, },
.val = ~(ID_AA64PFR0_EL1_AMU |
ID_AA64PFR0_EL1_MPAM |
ID_AA64PFR0_EL1_SVE |
ID_AA64PFR0_EL1_RAS |
ID_AA64PFR0_EL1_GIC |
ID_AA64PFR0_EL1_AdvSIMD |
ID_AA64PFR0_EL1_FP), },
ID_SANITISED(ID_AA64PFR1_EL1),
ID_UNALLOCATED(4,2),
ID_UNALLOCATED(4,3),
ID_SANITISED(ID_AA64ZFR0_EL1),
ID_WRITABLE(ID_AA64ZFR0_EL1, ~ID_AA64ZFR0_EL1_RES0),
ID_HIDDEN(ID_AA64SMFR0_EL1),
ID_UNALLOCATED(4,6),
ID_UNALLOCATED(4,7),
@ -2029,7 +2152,8 @@ static const struct sys_reg_desc sys_reg_descs[] = {
.get_user = get_id_reg,
.set_user = set_id_aa64dfr0_el1,
.reset = read_sanitised_id_aa64dfr0_el1,
.val = ID_AA64DFR0_EL1_PMUVer_MASK, },
.val = ID_AA64DFR0_EL1_PMUVer_MASK |
ID_AA64DFR0_EL1_DebugVer_MASK, },
ID_SANITISED(ID_AA64DFR1_EL1),
ID_UNALLOCATED(5,2),
ID_UNALLOCATED(5,3),
@ -2039,9 +2163,14 @@ static const struct sys_reg_desc sys_reg_descs[] = {
ID_UNALLOCATED(5,7),
/* CRm=6 */
ID_SANITISED(ID_AA64ISAR0_EL1),
ID_SANITISED(ID_AA64ISAR1_EL1),
ID_SANITISED(ID_AA64ISAR2_EL1),
ID_WRITABLE(ID_AA64ISAR0_EL1, ~ID_AA64ISAR0_EL1_RES0),
ID_WRITABLE(ID_AA64ISAR1_EL1, ~(ID_AA64ISAR1_EL1_GPI |
ID_AA64ISAR1_EL1_GPA |
ID_AA64ISAR1_EL1_API |
ID_AA64ISAR1_EL1_APA)),
ID_WRITABLE(ID_AA64ISAR2_EL1, ~(ID_AA64ISAR2_EL1_RES0 |
ID_AA64ISAR2_EL1_APA3 |
ID_AA64ISAR2_EL1_GPA3)),
ID_UNALLOCATED(6,3),
ID_UNALLOCATED(6,4),
ID_UNALLOCATED(6,5),
@ -2049,9 +2178,23 @@ static const struct sys_reg_desc sys_reg_descs[] = {
ID_UNALLOCATED(6,7),
/* CRm=7 */
ID_SANITISED(ID_AA64MMFR0_EL1),
ID_SANITISED(ID_AA64MMFR1_EL1),
ID_SANITISED(ID_AA64MMFR2_EL1),
ID_WRITABLE(ID_AA64MMFR0_EL1, ~(ID_AA64MMFR0_EL1_RES0 |
ID_AA64MMFR0_EL1_TGRAN4_2 |
ID_AA64MMFR0_EL1_TGRAN64_2 |
ID_AA64MMFR0_EL1_TGRAN16_2)),
ID_WRITABLE(ID_AA64MMFR1_EL1, ~(ID_AA64MMFR1_EL1_RES0 |
ID_AA64MMFR1_EL1_HCX |
ID_AA64MMFR1_EL1_XNX |
ID_AA64MMFR1_EL1_TWED |
ID_AA64MMFR1_EL1_XNX |
ID_AA64MMFR1_EL1_VH |
ID_AA64MMFR1_EL1_VMIDBits)),
ID_WRITABLE(ID_AA64MMFR2_EL1, ~(ID_AA64MMFR2_EL1_RES0 |
ID_AA64MMFR2_EL1_EVT |
ID_AA64MMFR2_EL1_FWB |
ID_AA64MMFR2_EL1_IDS |
ID_AA64MMFR2_EL1_NV |
ID_AA64MMFR2_EL1_CCIDX)),
ID_SANITISED(ID_AA64MMFR3_EL1),
ID_UNALLOCATED(7,4),
ID_UNALLOCATED(7,5),
@ -2116,9 +2259,11 @@ static const struct sys_reg_desc sys_reg_descs[] = {
/* PMBIDR_EL1 is not trapped */
{ PMU_SYS_REG(PMINTENSET_EL1),
.access = access_pminten, .reg = PMINTENSET_EL1 },
.access = access_pminten, .reg = PMINTENSET_EL1,
.get_user = get_pmreg, .set_user = set_pmreg },
{ PMU_SYS_REG(PMINTENCLR_EL1),
.access = access_pminten, .reg = PMINTENSET_EL1 },
.access = access_pminten, .reg = PMINTENSET_EL1,
.get_user = get_pmreg, .set_user = set_pmreg },
{ SYS_DESC(SYS_PMMIR_EL1), trap_raz_wi },
{ SYS_DESC(SYS_MAIR_EL1), access_vm_reg, reset_unknown, MAIR_EL1 },
@ -2166,14 +2311,17 @@ static const struct sys_reg_desc sys_reg_descs[] = {
{ SYS_DESC(SYS_CTR_EL0), access_ctr },
{ SYS_DESC(SYS_SVCR), undef_access },
{ PMU_SYS_REG(PMCR_EL0), .access = access_pmcr,
.reset = reset_pmcr, .reg = PMCR_EL0 },
{ PMU_SYS_REG(PMCR_EL0), .access = access_pmcr, .reset = reset_pmcr,
.reg = PMCR_EL0, .get_user = get_pmcr, .set_user = set_pmcr },
{ PMU_SYS_REG(PMCNTENSET_EL0),
.access = access_pmcnten, .reg = PMCNTENSET_EL0 },
.access = access_pmcnten, .reg = PMCNTENSET_EL0,
.get_user = get_pmreg, .set_user = set_pmreg },
{ PMU_SYS_REG(PMCNTENCLR_EL0),
.access = access_pmcnten, .reg = PMCNTENSET_EL0 },
.access = access_pmcnten, .reg = PMCNTENSET_EL0,
.get_user = get_pmreg, .set_user = set_pmreg },
{ PMU_SYS_REG(PMOVSCLR_EL0),
.access = access_pmovs, .reg = PMOVSSET_EL0 },
.access = access_pmovs, .reg = PMOVSSET_EL0,
.get_user = get_pmreg, .set_user = set_pmreg },
/*
* PM_SWINC_EL0 is exposed to userspace as RAZ/WI, as it was
* previously (and pointlessly) advertised in the past...
@ -2201,7 +2349,8 @@ static const struct sys_reg_desc sys_reg_descs[] = {
{ PMU_SYS_REG(PMUSERENR_EL0), .access = access_pmuserenr,
.reset = reset_val, .reg = PMUSERENR_EL0, .val = 0 },
{ PMU_SYS_REG(PMOVSSET_EL0),
.access = access_pmovs, .reg = PMOVSSET_EL0 },
.access = access_pmovs, .reg = PMOVSSET_EL0,
.get_user = get_pmreg, .set_user = set_pmreg },
{ SYS_DESC(SYS_TPIDR_EL0), NULL, reset_unknown, TPIDR_EL0 },
{ SYS_DESC(SYS_TPIDRRO_EL0), NULL, reset_unknown, TPIDRRO_EL0 },
@ -2380,18 +2529,28 @@ static const struct sys_reg_desc sys_reg_descs[] = {
EL2_REG(VTTBR_EL2, access_rw, reset_val, 0),
EL2_REG(VTCR_EL2, access_rw, reset_val, 0),
{ SYS_DESC(SYS_DACR32_EL2), NULL, reset_unknown, DACR32_EL2 },
{ SYS_DESC(SYS_DACR32_EL2), trap_undef, reset_unknown, DACR32_EL2 },
EL2_REG(HDFGRTR_EL2, access_rw, reset_val, 0),
EL2_REG(HDFGWTR_EL2, access_rw, reset_val, 0),
EL2_REG(SPSR_EL2, access_rw, reset_val, 0),
EL2_REG(ELR_EL2, access_rw, reset_val, 0),
{ SYS_DESC(SYS_SP_EL1), access_sp_el1},
{ SYS_DESC(SYS_IFSR32_EL2), NULL, reset_unknown, IFSR32_EL2 },
/* AArch32 SPSR_* are RES0 if trapped from a NV guest */
{ SYS_DESC(SYS_SPSR_irq), .access = trap_raz_wi,
.visibility = hidden_user_visibility },
{ SYS_DESC(SYS_SPSR_abt), .access = trap_raz_wi,
.visibility = hidden_user_visibility },
{ SYS_DESC(SYS_SPSR_und), .access = trap_raz_wi,
.visibility = hidden_user_visibility },
{ SYS_DESC(SYS_SPSR_fiq), .access = trap_raz_wi,
.visibility = hidden_user_visibility },
{ SYS_DESC(SYS_IFSR32_EL2), trap_undef, reset_unknown, IFSR32_EL2 },
EL2_REG(AFSR0_EL2, access_rw, reset_val, 0),
EL2_REG(AFSR1_EL2, access_rw, reset_val, 0),
EL2_REG(ESR_EL2, access_rw, reset_val, 0),
{ SYS_DESC(SYS_FPEXC32_EL2), NULL, reset_val, FPEXC32_EL2, 0x700 },
{ SYS_DESC(SYS_FPEXC32_EL2), trap_undef, reset_val, FPEXC32_EL2, 0x700 },
EL2_REG(FAR_EL2, access_rw, reset_val, 0),
EL2_REG(HPFAR_EL2, access_rw, reset_val, 0),
@ -2438,14 +2597,15 @@ static bool trap_dbgdidr(struct kvm_vcpu *vcpu,
if (p->is_write) {
return ignore_write(vcpu, p);
} else {
u64 dfr = read_sanitised_ftr_reg(SYS_ID_AA64DFR0_EL1);
u64 pfr = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);
u32 el3 = !!cpuid_feature_extract_unsigned_field(pfr, ID_AA64PFR0_EL1_EL3_SHIFT);
u64 dfr = IDREG(vcpu->kvm, SYS_ID_AA64DFR0_EL1);
u64 pfr = IDREG(vcpu->kvm, SYS_ID_AA64PFR0_EL1);
u32 el3 = !!SYS_FIELD_GET(ID_AA64PFR0_EL1, EL3, pfr);
p->regval = ((((dfr >> ID_AA64DFR0_EL1_WRPs_SHIFT) & 0xf) << 28) |
(((dfr >> ID_AA64DFR0_EL1_BRPs_SHIFT) & 0xf) << 24) |
(((dfr >> ID_AA64DFR0_EL1_CTX_CMPs_SHIFT) & 0xf) << 20)
| (6 << 16) | (1 << 15) | (el3 << 14) | (el3 << 12));
p->regval = ((SYS_FIELD_GET(ID_AA64DFR0_EL1, WRPs, dfr) << 28) |
(SYS_FIELD_GET(ID_AA64DFR0_EL1, BRPs, dfr) << 24) |
(SYS_FIELD_GET(ID_AA64DFR0_EL1, CTX_CMPs, dfr) << 20) |
(SYS_FIELD_GET(ID_AA64DFR0_EL1, DebugVer, dfr) << 16) |
(1 << 15) | (el3 << 14) | (el3 << 12));
return true;
}
}
@ -3572,6 +3732,65 @@ int kvm_arm_copy_sys_reg_indices(struct kvm_vcpu *vcpu, u64 __user *uindices)
return write_demux_regids(uindices);
}
#define KVM_ARM_FEATURE_ID_RANGE_INDEX(r) \
KVM_ARM_FEATURE_ID_RANGE_IDX(sys_reg_Op0(r), \
sys_reg_Op1(r), \
sys_reg_CRn(r), \
sys_reg_CRm(r), \
sys_reg_Op2(r))
static bool is_feature_id_reg(u32 encoding)
{
return (sys_reg_Op0(encoding) == 3 &&
(sys_reg_Op1(encoding) < 2 || sys_reg_Op1(encoding) == 3) &&
sys_reg_CRn(encoding) == 0 &&
sys_reg_CRm(encoding) <= 7);
}
int kvm_vm_ioctl_get_reg_writable_masks(struct kvm *kvm, struct reg_mask_range *range)
{
const void *zero_page = page_to_virt(ZERO_PAGE(0));
u64 __user *masks = (u64 __user *)range->addr;
/* Only feature id range is supported, reserved[13] must be zero. */
if (range->range ||
memcmp(range->reserved, zero_page, sizeof(range->reserved)))
return -EINVAL;
/* Wipe the whole thing first */
if (clear_user(masks, KVM_ARM_FEATURE_ID_RANGE_SIZE * sizeof(__u64)))
return -EFAULT;
for (int i = 0; i < ARRAY_SIZE(sys_reg_descs); i++) {
const struct sys_reg_desc *reg = &sys_reg_descs[i];
u32 encoding = reg_to_encoding(reg);
u64 val;
if (!is_feature_id_reg(encoding) || !reg->set_user)
continue;
/*
* For ID registers, we return the writable mask. Other feature
* registers return a full 64bit mask. That's not necessary
* compliant with a given revision of the architecture, but the
* RES0/RES1 definitions allow us to do that.
*/
if (is_id_reg(encoding)) {
if (!reg->val ||
(is_aa32_id_reg(encoding) && !kvm_supports_32bit_el0()))
continue;
val = reg->val;
} else {
val = ~0UL;
}
if (put_user(val, (masks + KVM_ARM_FEATURE_ID_RANGE_INDEX(encoding))))
return -EFAULT;
}
return 0;
}
int __init kvm_sys_reg_table_init(void)
{
struct sys_reg_params params;

25
arch/arm64/kvm/trace_arm.h
View File

@ -136,6 +136,31 @@ TRACE_EVENT(kvm_mmio_emulate,
__entry->vcpu_pc, __entry->instr, __entry->cpsr)
);
TRACE_EVENT(kvm_mmio_nisv,
TP_PROTO(unsigned long vcpu_pc, unsigned long esr,
unsigned long far, unsigned long ipa),
TP_ARGS(vcpu_pc, esr, far, ipa),
TP_STRUCT__entry(
__field( unsigned long, vcpu_pc )
__field( unsigned long, esr )
__field( unsigned long, far )
__field( unsigned long, ipa )
),
TP_fast_assign(
__entry->vcpu_pc = vcpu_pc;
__entry->esr = esr;
__entry->far = far;
__entry->ipa = ipa;
),
TP_printk("ipa %#016lx, esr %#016lx, far %#016lx, pc %#016lx",
__entry->ipa, __entry->esr,
__entry->far, __entry->vcpu_pc)
);
TRACE_EVENT(kvm_set_way_flush,
TP_PROTO(unsigned long vcpu_pc, bool cache),
TP_ARGS(vcpu_pc, cache),

6
arch/arm64/kvm/vgic/vgic-debug.c
View File

@ -166,7 +166,7 @@ static void print_header(struct seq_file *s, struct vgic_irq *irq,
if (vcpu) {
hdr = "VCPU";
id = vcpu->vcpu_id;
id = vcpu->vcpu_idx;
}
seq_printf(s, "\n");
@ -212,7 +212,7 @@ static void print_irq_state(struct seq_file *s, struct vgic_irq *irq,
" %2d "
"\n",
type, irq->intid,
(irq->target_vcpu) ? irq->target_vcpu->vcpu_id : -1,
(irq->target_vcpu) ? irq->target_vcpu->vcpu_idx : -1,
pending,
irq->line_level,
irq->active,
@ -224,7 +224,7 @@ static void print_irq_state(struct seq_file *s, struct vgic_irq *irq,
irq->mpidr,
irq->source,
irq->priority,
(irq->vcpu) ? irq->vcpu->vcpu_id : -1);
(irq->vcpu) ? irq->vcpu->vcpu_idx : -1);
}
static int vgic_debug_show(struct seq_file *s, void *v)

2
arch/arm64/kvm/vgic/vgic-irqfd.c
View File

@ -23,7 +23,7 @@ static int vgic_irqfd_set_irq(struct kvm_kernel_irq_routing_entry *e,
if (!vgic_valid_spi(kvm, spi_id))
return -EINVAL;
return kvm_vgic_inject_irq(kvm, 0, spi_id, level, NULL);
return kvm_vgic_inject_irq(kvm, NULL, spi_id, level, NULL);
}
/**

49
arch/arm64/kvm/vgic/vgic-its.c
View File

@ -378,6 +378,12 @@ static int update_affinity(struct vgic_irq *irq, struct kvm_vcpu *vcpu)
return ret;
}
static struct kvm_vcpu *collection_to_vcpu(struct kvm *kvm,
struct its_collection *col)
{
return kvm_get_vcpu_by_id(kvm, col->target_addr);
}
/*
* Promotes the ITS view of affinity of an ITTE (which redistributor this LPI
* is targeting) to the VGIC's view, which deals with target VCPUs.
@ -391,7 +397,7 @@ static void update_affinity_ite(struct kvm *kvm, struct its_ite *ite)
if (!its_is_collection_mapped(ite->collection))
return;
vcpu = kvm_get_vcpu(kvm, ite->collection->target_addr);
vcpu = collection_to_vcpu(kvm, ite->collection);
update_affinity(ite->irq, vcpu);
}
@ -679,7 +685,7 @@ int vgic_its_resolve_lpi(struct kvm *kvm, struct vgic_its *its,
if (!ite || !its_is_collection_mapped(ite->collection))
return E_ITS_INT_UNMAPPED_INTERRUPT;
vcpu = kvm_get_vcpu(kvm, ite->collection->target_addr);
vcpu = collection_to_vcpu(kvm, ite->collection);
if (!vcpu)
return E_ITS_INT_UNMAPPED_INTERRUPT;
@ -887,7 +893,7 @@ static int vgic_its_cmd_handle_movi(struct kvm *kvm, struct vgic_its *its,
return E_ITS_MOVI_UNMAPPED_COLLECTION;
ite->collection = collection;
vcpu = kvm_get_vcpu(kvm, collection->target_addr);
vcpu = collection_to_vcpu(kvm, collection);
vgic_its_invalidate_cache(kvm);
@ -1121,7 +1127,7 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
}
if (its_is_collection_mapped(collection))
vcpu = kvm_get_vcpu(kvm, collection->target_addr);
vcpu = collection_to_vcpu(kvm, collection);
irq = vgic_add_lpi(kvm, lpi_nr, vcpu);
if (IS_ERR(irq)) {
@ -1242,21 +1248,22 @@ static int vgic_its_cmd_handle_mapc(struct kvm *kvm, struct vgic_its *its,
u64 *its_cmd)
{
u16 coll_id;
u32 target_addr;
struct its_collection *collection;
bool valid;
valid = its_cmd_get_validbit(its_cmd);
coll_id = its_cmd_get_collection(its_cmd);
target_addr = its_cmd_get_target_addr(its_cmd);
if (target_addr >= atomic_read(&kvm->online_vcpus))
return E_ITS_MAPC_PROCNUM_OOR;
if (!valid) {
vgic_its_free_collection(its, coll_id);
vgic_its_invalidate_cache(kvm);
} else {
struct kvm_vcpu *vcpu;
vcpu = kvm_get_vcpu_by_id(kvm, its_cmd_get_target_addr(its_cmd));
if (!vcpu)
return E_ITS_MAPC_PROCNUM_OOR;
collection = find_collection(its, coll_id);
if (!collection) {
@ -1270,9 +1277,9 @@ static int vgic_its_cmd_handle_mapc(struct kvm *kvm, struct vgic_its *its,
coll_id);
if (ret)
return ret;
collection->target_addr = target_addr;
collection->target_addr = vcpu->vcpu_id;
} else {
collection->target_addr = target_addr;
collection->target_addr = vcpu->vcpu_id;
update_affinity_collection(kvm, its, collection);
}
}
@ -1382,7 +1389,7 @@ static int vgic_its_cmd_handle_invall(struct kvm *kvm, struct vgic_its *its,
if (!its_is_collection_mapped(collection))
return E_ITS_INVALL_UNMAPPED_COLLECTION;
vcpu = kvm_get_vcpu(kvm, collection->target_addr);
vcpu = collection_to_vcpu(kvm, collection);
vgic_its_invall(vcpu);
return 0;
@ -1399,23 +1406,21 @@ static int vgic_its_cmd_handle_invall(struct kvm *kvm, struct vgic_its *its,
static int vgic_its_cmd_handle_movall(struct kvm *kvm, struct vgic_its *its,
u64 *its_cmd)
{
u32 target1_addr = its_cmd_get_target_addr(its_cmd);
u32 target2_addr = its_cmd_mask_field(its_cmd, 3, 16, 32);
struct kvm_vcpu *vcpu1, *vcpu2;
struct vgic_irq *irq;
u32 *intids;
int irq_count, i;
if (target1_addr >= atomic_read(&kvm->online_vcpus) ||
target2_addr >= atomic_read(&kvm->online_vcpus))
/* We advertise GITS_TYPER.PTA==0, making the address the vcpu ID */
vcpu1 = kvm_get_vcpu_by_id(kvm, its_cmd_get_target_addr(its_cmd));
vcpu2 = kvm_get_vcpu_by_id(kvm, its_cmd_mask_field(its_cmd, 3, 16, 32));
if (!vcpu1 || !vcpu2)
return E_ITS_MOVALL_PROCNUM_OOR;
if (target1_addr == target2_addr)
if (vcpu1 == vcpu2)
return 0;
vcpu1 = kvm_get_vcpu(kvm, target1_addr);
vcpu2 = kvm_get_vcpu(kvm, target2_addr);
irq_count = vgic_copy_lpi_list(kvm, vcpu1, &intids);
if (irq_count < 0)
return irq_count;
@ -2258,7 +2263,7 @@ static int vgic_its_restore_ite(struct vgic_its *its, u32 event_id,
return PTR_ERR(ite);
if (its_is_collection_mapped(collection))
vcpu = kvm_get_vcpu(kvm, collection->target_addr);
vcpu = kvm_get_vcpu_by_id(kvm, collection->target_addr);
irq = vgic_add_lpi(kvm, lpi_id, vcpu);
if (IS_ERR(irq)) {
@ -2573,7 +2578,7 @@ static int vgic_its_restore_cte(struct vgic_its *its, gpa_t gpa, int esz)
coll_id = val & KVM_ITS_CTE_ICID_MASK;
if (target_addr != COLLECTION_NOT_MAPPED &&
target_addr >= atomic_read(&kvm->online_vcpus))
!kvm_get_vcpu_by_id(kvm, target_addr))
return -EINVAL;
collection = find_collection(its, coll_id);

11
arch/arm64/kvm/vgic/vgic-kvm-device.c
View File

@ -27,7 +27,8 @@ int vgic_check_iorange(struct kvm *kvm, phys_addr_t ioaddr,
if (addr + size < addr)
return -EINVAL;
if (addr & ~kvm_phys_mask(kvm) || addr + size > kvm_phys_size(kvm))
if (addr & ~kvm_phys_mask(&kvm->arch.mmu) ||
(addr + size) > kvm_phys_size(&kvm->arch.mmu))
return -E2BIG;
return 0;
@ -339,13 +340,9 @@ int vgic_v2_parse_attr(struct kvm_device *dev, struct kvm_device_attr *attr,
{
int cpuid;
cpuid = (attr->attr & KVM_DEV_ARM_VGIC_CPUID_MASK) >>
KVM_DEV_ARM_VGIC_CPUID_SHIFT;
cpuid = FIELD_GET(KVM_DEV_ARM_VGIC_CPUID_MASK, attr->attr);
if (cpuid >= atomic_read(&dev->kvm->online_vcpus))
return -EINVAL;
reg_attr->vcpu = kvm_get_vcpu(dev->kvm, cpuid);
reg_attr->vcpu = kvm_get_vcpu_by_id(dev->kvm, cpuid);
reg_attr->addr = attr->attr & KVM_DEV_ARM_VGIC_OFFSET_MASK;
return 0;

156
arch/arm64/kvm/vgic/vgic-mmio-v3.c
View File

@ -1013,35 +1013,6 @@ int vgic_v3_has_attr_regs(struct kvm_device *dev, struct kvm_device_attr *attr)
return 0;
}
/*
* Compare a given affinity (level 1-3 and a level 0 mask, from the SGI
* generation register ICC_SGI1R_EL1) with a given VCPU.
* If the VCPU's MPIDR matches, return the level0 affinity, otherwise
* return -1.
*/
static int match_mpidr(u64 sgi_aff, u16 sgi_cpu_mask, struct kvm_vcpu *vcpu)
{
unsigned long affinity;
int level0;
/*
* Split the current VCPU's MPIDR into affinity level 0 and the
* rest as this is what we have to compare against.
*/
affinity = kvm_vcpu_get_mpidr_aff(vcpu);
level0 = MPIDR_AFFINITY_LEVEL(affinity, 0);
affinity &= ~MPIDR_LEVEL_MASK;
/* bail out if the upper three levels don't match */
if (sgi_aff != affinity)
return -1;
/* Is this VCPU's bit set in the mask ? */
if (!(sgi_cpu_mask & BIT(level0)))
return -1;
return level0;
}
/*
* The ICC_SGI* registers encode the affinity differently from the MPIDR,
@ -1052,6 +1023,38 @@ static int match_mpidr(u64 sgi_aff, u16 sgi_cpu_mask, struct kvm_vcpu *vcpu)
((((reg) & ICC_SGI1R_AFFINITY_## level ##_MASK) \
>> ICC_SGI1R_AFFINITY_## level ##_SHIFT) << MPIDR_LEVEL_SHIFT(level))
static void vgic_v3_queue_sgi(struct kvm_vcpu *vcpu, u32 sgi, bool allow_group1)
{
struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, sgi);
unsigned long flags;
raw_spin_lock_irqsave(&irq->irq_lock, flags);
/*
* An access targeting Group0 SGIs can only generate
* those, while an access targeting Group1 SGIs can
* generate interrupts of either group.
*/
if (!irq->group || allow_group1) {
if (!irq->hw) {
irq->pending_latch = true;
vgic_queue_irq_unlock(vcpu->kvm, irq, flags);
} else {
/* HW SGI? Ask the GIC to inject it */
int err;
err = irq_set_irqchip_state(irq->host_irq,
IRQCHIP_STATE_PENDING,
true);
WARN_RATELIMIT(err, "IRQ %d", irq->host_irq);
raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
}
} else {
raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
}
vgic_put_irq(vcpu->kvm, irq);
}
/**
* vgic_v3_dispatch_sgi - handle SGI requests from VCPUs
* @vcpu: The VCPU requesting a SGI
@ -1062,83 +1065,46 @@ static int match_mpidr(u64 sgi_aff, u16 sgi_cpu_mask, struct kvm_vcpu *vcpu)
* This will trap in sys_regs.c and call this function.
* This ICC_SGI1R_EL1 register contains the upper three affinity levels of the
* target processors as well as a bitmask of 16 Aff0 CPUs.
* If the interrupt routing mode bit is not set, we iterate over all VCPUs to
* check for matching ones. If this bit is set, we signal all, but not the
* calling VCPU.
*
* If the interrupt routing mode bit is not set, we iterate over the Aff0
* bits and signal the VCPUs matching the provided Aff{3,2,1}.
*
* If this bit is set, we signal all, but not the calling VCPU.
*/
void vgic_v3_dispatch_sgi(struct kvm_vcpu *vcpu, u64 reg, bool allow_group1)
{
struct kvm *kvm = vcpu->kvm;
struct kvm_vcpu *c_vcpu;
u16 target_cpus;
unsigned long target_cpus;
u64 mpidr;
int sgi;
int vcpu_id = vcpu->vcpu_id;
bool broadcast;
unsigned long c, flags;
u32 sgi, aff0;
unsigned long c;
sgi = (reg & ICC_SGI1R_SGI_ID_MASK) >> ICC_SGI1R_SGI_ID_SHIFT;
broadcast = reg & BIT_ULL(ICC_SGI1R_IRQ_ROUTING_MODE_BIT);
target_cpus = (reg & ICC_SGI1R_TARGET_LIST_MASK) >> ICC_SGI1R_TARGET_LIST_SHIFT;
sgi = FIELD_GET(ICC_SGI1R_SGI_ID_MASK, reg);
/* Broadcast */
if (unlikely(reg & BIT_ULL(ICC_SGI1R_IRQ_ROUTING_MODE_BIT))) {
kvm_for_each_vcpu(c, c_vcpu, kvm) {
/* Don't signal the calling VCPU */
if (c_vcpu == vcpu)
continue;
vgic_v3_queue_sgi(c_vcpu, sgi, allow_group1);
}
return;
}
/* We iterate over affinities to find the corresponding vcpus */
mpidr = SGI_AFFINITY_LEVEL(reg, 3);
mpidr |= SGI_AFFINITY_LEVEL(reg, 2);
mpidr |= SGI_AFFINITY_LEVEL(reg, 1);
target_cpus = FIELD_GET(ICC_SGI1R_TARGET_LIST_MASK, reg);
/*
* We iterate over all VCPUs to find the MPIDRs matching the request.
* If we have handled one CPU, we clear its bit to detect early
* if we are already finished. This avoids iterating through all
* VCPUs when most of the times we just signal a single VCPU.
*/
kvm_for_each_vcpu(c, c_vcpu, kvm) {
struct vgic_irq *irq;
/* Exit early if we have dealt with all requested CPUs */
if (!broadcast && target_cpus == 0)
break;
/* Don't signal the calling VCPU */
if (broadcast && c == vcpu_id)
continue;
if (!broadcast) {
int level0;
level0 = match_mpidr(mpidr, target_cpus, c_vcpu);
if (level0 == -1)
continue;
/* remove this matching VCPU from the mask */
target_cpus &= ~BIT(level0);
}
irq = vgic_get_irq(vcpu->kvm, c_vcpu, sgi);
raw_spin_lock_irqsave(&irq->irq_lock, flags);
/*
* An access targeting Group0 SGIs can only generate
* those, while an access targeting Group1 SGIs can
* generate interrupts of either group.
*/
if (!irq->group || allow_group1) {
if (!irq->hw) {
irq->pending_latch = true;
vgic_queue_irq_unlock(vcpu->kvm, irq, flags);
} else {
/* HW SGI? Ask the GIC to inject it */
int err;
err = irq_set_irqchip_state(irq->host_irq,
IRQCHIP_STATE_PENDING,
true);
WARN_RATELIMIT(err, "IRQ %d", irq->host_irq);
raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
}
} else {
raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
}
vgic_put_irq(vcpu->kvm, irq);
for_each_set_bit(aff0, &target_cpus, hweight_long(ICC_SGI1R_TARGET_LIST_MASK)) {
c_vcpu = kvm_mpidr_to_vcpu(kvm, mpidr | aff0);
if (c_vcpu)
vgic_v3_queue_sgi(c_vcpu, sgi, allow_group1);
}
}

12
arch/arm64/kvm/vgic/vgic.c
View File

@ -422,7 +422,7 @@ retry:
/**
* kvm_vgic_inject_irq - Inject an IRQ from a device to the vgic
* @kvm: The VM structure pointer
* @cpuid: The CPU for PPIs
* @vcpu: The CPU for PPIs or NULL for global interrupts
* @intid: The INTID to inject a new state to.
* @level: Edge-triggered: true: to trigger the interrupt
* false: to ignore the call
@ -436,24 +436,22 @@ retry:
* level-sensitive interrupts. You can think of the level parameter as 1
* being HIGH and 0 being LOW and all devices being active-HIGH.
*/
int kvm_vgic_inject_irq(struct kvm *kvm, int cpuid, unsigned int intid,
bool level, void *owner)
int kvm_vgic_inject_irq(struct kvm *kvm, struct kvm_vcpu *vcpu,
unsigned int intid, bool level, void *owner)
{
struct kvm_vcpu *vcpu;
struct vgic_irq *irq;
unsigned long flags;
int ret;
trace_vgic_update_irq_pending(cpuid, intid, level);
ret = vgic_lazy_init(kvm);
if (ret)
return ret;
vcpu = kvm_get_vcpu(kvm, cpuid);
if (!vcpu && intid < VGIC_NR_PRIVATE_IRQS)
return -EINVAL;
trace_vgic_update_irq_pending(vcpu ? vcpu->vcpu_idx : 0, intid, level);
irq = vgic_get_irq(kvm, vcpu, intid);
if (!irq)
return -EINVAL;

11
arch/arm64/kvm/vmid.c
View File

@ -135,10 +135,11 @@ void kvm_arm_vmid_clear_active(void)
atomic64_set(this_cpu_ptr(&active_vmids), VMID_ACTIVE_INVALID);
}
void kvm_arm_vmid_update(struct kvm_vmid *kvm_vmid)
bool kvm_arm_vmid_update(struct kvm_vmid *kvm_vmid)
{
unsigned long flags;
u64 vmid, old_active_vmid;
bool updated = false;
vmid = atomic64_read(&kvm_vmid->id);
@ -156,17 +157,21 @@ void kvm_arm_vmid_update(struct kvm_vmid *kvm_vmid)
if (old_active_vmid != 0 && vmid_gen_match(vmid) &&
0 != atomic64_cmpxchg_relaxed(this_cpu_ptr(&active_vmids),
old_active_vmid, vmid))
return;
return false;
raw_spin_lock_irqsave(&cpu_vmid_lock, flags);
/* Check that our VMID belongs to the current generation. */
vmid = atomic64_read(&kvm_vmid->id);
if (!vmid_gen_match(vmid))
if (!vmid_gen_match(vmid)) {
vmid = new_vmid(kvm_vmid);
updated = true;
}
atomic64_set(this_cpu_ptr(&active_vmids), vmid);
raw_spin_unlock_irqrestore(&cpu_vmid_lock, flags);
return updated;
}
/*

2
arch/loongarch/Kbuild
View File

@ -3,5 +3,7 @@ obj-y += mm/
obj-y += net/
obj-y += vdso/
obj-$(CONFIG_KVM) += kvm/
# for cleaning
subdir- += boot

6
arch/loongarch/Kconfig
View File

@ -129,6 +129,7 @@ config LOONGARCH
select HAVE_KPROBES
select HAVE_KPROBES_ON_FTRACE
select HAVE_KRETPROBES
select HAVE_KVM
select HAVE_MOD_ARCH_SPECIFIC
select HAVE_NMI
select HAVE_PCI
@ -263,6 +264,9 @@ config AS_HAS_LASX_EXTENSION
config AS_HAS_LBT_EXTENSION
def_bool $(as-instr,movscr2gr \$a0$(comma)\$scr0)
config AS_HAS_LVZ_EXTENSION
def_bool $(as-instr,hvcl 0)
menu "Kernel type and options"
source "kernel/Kconfig.hz"
@ -676,3 +680,5 @@ source "kernel/power/Kconfig"
source "drivers/acpi/Kconfig"
endmenu
source "arch/loongarch/kvm/Kconfig"

2
arch/loongarch/configs/loongson3_defconfig
View File

@ -66,6 +66,8 @@ CONFIG_EFI_ZBOOT=y
CONFIG_EFI_GENERIC_STUB_INITRD_CMDLINE_LOADER=y
CONFIG_EFI_CAPSULE_LOADER=m
CONFIG_EFI_TEST=m
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=m
CONFIG_JUMP_LABEL=y
CONFIG_MODULES=y
CONFIG_MODULE_FORCE_LOAD=y

16
arch/loongarch/include/asm/inst.h
View File

@ -65,6 +65,14 @@ enum reg2_op {
revbd_op = 0x0f,
revh2w_op = 0x10,
revhd_op = 0x11,
iocsrrdb_op = 0x19200,
iocsrrdh_op = 0x19201,
iocsrrdw_op = 0x19202,
iocsrrdd_op = 0x19203,
iocsrwrb_op = 0x19204,
iocsrwrh_op = 0x19205,
iocsrwrw_op = 0x19206,
iocsrwrd_op = 0x19207,
};
enum reg2i5_op {
@ -318,6 +326,13 @@ struct reg2bstrd_format {
unsigned int opcode : 10;
};
struct reg2csr_format {
unsigned int rd : 5;
unsigned int rj : 5;
unsigned int csr : 14;
unsigned int opcode : 8;
};
struct reg3_format {
unsigned int rd : 5;
unsigned int rj : 5;
@ -346,6 +361,7 @@ union loongarch_instruction {
struct reg2i14_format reg2i14_format;
struct reg2i16_format reg2i16_format;
struct reg2bstrd_format reg2bstrd_format;
struct reg2csr_format reg2csr_format;
struct reg3_format reg3_format;
struct reg3sa2_format reg3sa2_format;
};

211
arch/loongarch/include/asm/kvm_csr.h Normal file
View File

@ -0,0 +1,211 @@
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Copyright (C) 2020-2023 Loongson Technology Corporation Limited
*/
#ifndef __ASM_LOONGARCH_KVM_CSR_H__
#define __ASM_LOONGARCH_KVM_CSR_H__
#include <linux/uaccess.h>
#include <linux/kvm_host.h>
#include <asm/loongarch.h>
#include <asm/kvm_vcpu.h>
#define gcsr_read(csr) \
({ \
register unsigned long __v; \
__asm__ __volatile__( \
" gcsrrd %[val], %[reg]\n\t" \
: [val] "=r" (__v) \
: [reg] "i" (csr) \
: "memory"); \
__v; \
})
#define gcsr_write(v, csr) \
({ \
register unsigned long __v = v; \
__asm__ __volatile__ ( \
" gcsrwr %[val], %[reg]\n\t" \
: [val] "+r" (__v) \
: [reg] "i" (csr) \
: "memory"); \
})
#define gcsr_xchg(v, m, csr) \
({ \
register unsigned long __v = v; \
__asm__ __volatile__( \
" gcsrxchg %[val], %[mask], %[reg]\n\t" \
: [val] "+r" (__v) \
: [mask] "r" (m), [reg] "i" (csr) \
: "memory"); \
__v; \
})
/* Guest CSRS read and write */
#define read_gcsr_crmd() gcsr_read(LOONGARCH_CSR_CRMD)
#define write_gcsr_crmd(val) gcsr_write(val, LOONGARCH_CSR_CRMD)
#define read_gcsr_prmd() gcsr_read(LOONGARCH_CSR_PRMD)
#define write_gcsr_prmd(val) gcsr_write(val, LOONGARCH_CSR_PRMD)
#define read_gcsr_euen() gcsr_read(LOONGARCH_CSR_EUEN)
#define write_gcsr_euen(val) gcsr_write(val, LOONGARCH_CSR_EUEN)
#define read_gcsr_misc() gcsr_read(LOONGARCH_CSR_MISC)
#define write_gcsr_misc(val) gcsr_write(val, LOONGARCH_CSR_MISC)
#define read_gcsr_ecfg() gcsr_read(LOONGARCH_CSR_ECFG)
#define write_gcsr_ecfg(val) gcsr_write(val, LOONGARCH_CSR_ECFG)
#define read_gcsr_estat() gcsr_read(LOONGARCH_CSR_ESTAT)
#define write_gcsr_estat(val) gcsr_write(val, LOONGARCH_CSR_ESTAT)
#define read_gcsr_era() gcsr_read(LOONGARCH_CSR_ERA)
#define write_gcsr_era(val) gcsr_write(val, LOONGARCH_CSR_ERA)
#define read_gcsr_badv() gcsr_read(LOONGARCH_CSR_BADV)
#define write_gcsr_badv(val) gcsr_write(val, LOONGARCH_CSR_BADV)
#define read_gcsr_badi() gcsr_read(LOONGARCH_CSR_BADI)
#define write_gcsr_badi(val) gcsr_write(val, LOONGARCH_CSR_BADI)
#define read_gcsr_eentry() gcsr_read(LOONGARCH_CSR_EENTRY)
#define write_gcsr_eentry(val) gcsr_write(val, LOONGARCH_CSR_EENTRY)
#define read_gcsr_asid() gcsr_read(LOONGARCH_CSR_ASID)
#define write_gcsr_asid(val) gcsr_write(val, LOONGARCH_CSR_ASID)
#define read_gcsr_pgdl() gcsr_read(LOONGARCH_CSR_PGDL)
#define write_gcsr_pgdl(val) gcsr_write(val, LOONGARCH_CSR_PGDL)
#define read_gcsr_pgdh() gcsr_read(LOONGARCH_CSR_PGDH)
#define write_gcsr_pgdh(val) gcsr_write(val, LOONGARCH_CSR_PGDH)
#define write_gcsr_pgd(val) gcsr_write(val, LOONGARCH_CSR_PGD)
#define read_gcsr_pgd() gcsr_read(LOONGARCH_CSR_PGD)
#define read_gcsr_pwctl0() gcsr_read(LOONGARCH_CSR_PWCTL0)
#define write_gcsr_pwctl0(val) gcsr_write(val, LOONGARCH_CSR_PWCTL0)
#define read_gcsr_pwctl1() gcsr_read(LOONGARCH_CSR_PWCTL1)
#define write_gcsr_pwctl1(val) gcsr_write(val, LOONGARCH_CSR_PWCTL1)
#define read_gcsr_stlbpgsize() gcsr_read(LOONGARCH_CSR_STLBPGSIZE)
#define write_gcsr_stlbpgsize(val) gcsr_write(val, LOONGARCH_CSR_STLBPGSIZE)
#define read_gcsr_rvacfg() gcsr_read(LOONGARCH_CSR_RVACFG)
#define write_gcsr_rvacfg(val) gcsr_write(val, LOONGARCH_CSR_RVACFG)
#define read_gcsr_cpuid() gcsr_read(LOONGARCH_CSR_CPUID)
#define write_gcsr_cpuid(val) gcsr_write(val, LOONGARCH_CSR_CPUID)
#define read_gcsr_prcfg1() gcsr_read(LOONGARCH_CSR_PRCFG1)
#define write_gcsr_prcfg1(val) gcsr_write(val, LOONGARCH_CSR_PRCFG1)
#define read_gcsr_prcfg2() gcsr_read(LOONGARCH_CSR_PRCFG2)
#define write_gcsr_prcfg2(val) gcsr_write(val, LOONGARCH_CSR_PRCFG2)
#define read_gcsr_prcfg3() gcsr_read(LOONGARCH_CSR_PRCFG3)
#define write_gcsr_prcfg3(val) gcsr_write(val, LOONGARCH_CSR_PRCFG3)
#define read_gcsr_kscratch0() gcsr_read(LOONGARCH_CSR_KS0)
#define write_gcsr_kscratch0(val) gcsr_write(val, LOONGARCH_CSR_KS0)
#define read_gcsr_kscratch1() gcsr_read(LOONGARCH_CSR_KS1)
#define write_gcsr_kscratch1(val) gcsr_write(val, LOONGARCH_CSR_KS1)
#define read_gcsr_kscratch2() gcsr_read(LOONGARCH_CSR_KS2)
#define write_gcsr_kscratch2(val) gcsr_write(val, LOONGARCH_CSR_KS2)
#define read_gcsr_kscratch3() gcsr_read(LOONGARCH_CSR_KS3)
#define write_gcsr_kscratch3(val) gcsr_write(val, LOONGARCH_CSR_KS3)
#define read_gcsr_kscratch4() gcsr_read(LOONGARCH_CSR_KS4)
#define write_gcsr_kscratch4(val) gcsr_write(val, LOONGARCH_CSR_KS4)
#define read_gcsr_kscratch5() gcsr_read(LOONGARCH_CSR_KS5)
#define write_gcsr_kscratch5(val) gcsr_write(val, LOONGARCH_CSR_KS5)
#define read_gcsr_kscratch6() gcsr_read(LOONGARCH_CSR_KS6)
#define write_gcsr_kscratch6(val) gcsr_write(val, LOONGARCH_CSR_KS6)
#define read_gcsr_kscratch7() gcsr_read(LOONGARCH_CSR_KS7)
#define write_gcsr_kscratch7(val) gcsr_write(val, LOONGARCH_CSR_KS7)
#define read_gcsr_timerid() gcsr_read(LOONGARCH_CSR_TMID)
#define write_gcsr_timerid(val) gcsr_write(val, LOONGARCH_CSR_TMID)
#define read_gcsr_timercfg() gcsr_read(LOONGARCH_CSR_TCFG)
#define write_gcsr_timercfg(val) gcsr_write(val, LOONGARCH_CSR_TCFG)
#define read_gcsr_timertick() gcsr_read(LOONGARCH_CSR_TVAL)
#define write_gcsr_timertick(val) gcsr_write(val, LOONGARCH_CSR_TVAL)
#define read_gcsr_timeroffset() gcsr_read(LOONGARCH_CSR_CNTC)
#define write_gcsr_timeroffset(val) gcsr_write(val, LOONGARCH_CSR_CNTC)
#define read_gcsr_llbctl() gcsr_read(LOONGARCH_CSR_LLBCTL)
#define write_gcsr_llbctl(val) gcsr_write(val, LOONGARCH_CSR_LLBCTL)
#define read_gcsr_tlbidx() gcsr_read(LOONGARCH_CSR_TLBIDX)
#define write_gcsr_tlbidx(val) gcsr_write(val, LOONGARCH_CSR_TLBIDX)
#define read_gcsr_tlbrentry() gcsr_read(LOONGARCH_CSR_TLBRENTRY)
#define write_gcsr_tlbrentry(val) gcsr_write(val, LOONGARCH_CSR_TLBRENTRY)
#define read_gcsr_tlbrbadv() gcsr_read(LOONGARCH_CSR_TLBRBADV)
#define write_gcsr_tlbrbadv(val) gcsr_write(val, LOONGARCH_CSR_TLBRBADV)
#define read_gcsr_tlbrera() gcsr_read(LOONGARCH_CSR_TLBRERA)
#define write_gcsr_tlbrera(val) gcsr_write(val, LOONGARCH_CSR_TLBRERA)
#define read_gcsr_tlbrsave() gcsr_read(LOONGARCH_CSR_TLBRSAVE)
#define write_gcsr_tlbrsave(val) gcsr_write(val, LOONGARCH_CSR_TLBRSAVE)
#define read_gcsr_tlbrelo0() gcsr_read(LOONGARCH_CSR_TLBRELO0)
#define write_gcsr_tlbrelo0(val) gcsr_write(val, LOONGARCH_CSR_TLBRELO0)
#define read_gcsr_tlbrelo1() gcsr_read(LOONGARCH_CSR_TLBRELO1)
#define write_gcsr_tlbrelo1(val) gcsr_write(val, LOONGARCH_CSR_TLBRELO1)
#define read_gcsr_tlbrehi() gcsr_read(LOONGARCH_CSR_TLBREHI)
#define write_gcsr_tlbrehi(val) gcsr_write(val, LOONGARCH_CSR_TLBREHI)
#define read_gcsr_tlbrprmd() gcsr_read(LOONGARCH_CSR_TLBRPRMD)
#define write_gcsr_tlbrprmd(val) gcsr_write(val, LOONGARCH_CSR_TLBRPRMD)
#define read_gcsr_directwin0() gcsr_read(LOONGARCH_CSR_DMWIN0)
#define write_gcsr_directwin0(val) gcsr_write(val, LOONGARCH_CSR_DMWIN0)
#define read_gcsr_directwin1() gcsr_read(LOONGARCH_CSR_DMWIN1)
#define write_gcsr_directwin1(val) gcsr_write(val, LOONGARCH_CSR_DMWIN1)
#define read_gcsr_directwin2() gcsr_read(LOONGARCH_CSR_DMWIN2)
#define write_gcsr_directwin2(val) gcsr_write(val, LOONGARCH_CSR_DMWIN2)
#define read_gcsr_directwin3() gcsr_read(LOONGARCH_CSR_DMWIN3)
#define write_gcsr_directwin3(val) gcsr_write(val, LOONGARCH_CSR_DMWIN3)
/* Guest related CSRs */
#define read_csr_gtlbc() csr_read64(LOONGARCH_CSR_GTLBC)
#define write_csr_gtlbc(val) csr_write64(val, LOONGARCH_CSR_GTLBC)
#define read_csr_trgp() csr_read64(LOONGARCH_CSR_TRGP)
#define read_csr_gcfg() csr_read64(LOONGARCH_CSR_GCFG)
#define write_csr_gcfg(val) csr_write64(val, LOONGARCH_CSR_GCFG)
#define read_csr_gstat() csr_read64(LOONGARCH_CSR_GSTAT)
#define write_csr_gstat(val) csr_write64(val, LOONGARCH_CSR_GSTAT)
#define read_csr_gintc() csr_read64(LOONGARCH_CSR_GINTC)
#define write_csr_gintc(val) csr_write64(val, LOONGARCH_CSR_GINTC)
#define read_csr_gcntc() csr_read64(LOONGARCH_CSR_GCNTC)
#define write_csr_gcntc(val) csr_write64(val, LOONGARCH_CSR_GCNTC)
#define __BUILD_GCSR_OP(name) __BUILD_CSR_COMMON(gcsr_##name)
__BUILD_CSR_OP(gcfg)
__BUILD_CSR_OP(gstat)
__BUILD_CSR_OP(gtlbc)
__BUILD_CSR_OP(gintc)
__BUILD_GCSR_OP(llbctl)
__BUILD_GCSR_OP(tlbidx)
#define set_gcsr_estat(val) \
gcsr_xchg(val, val, LOONGARCH_CSR_ESTAT)
#define clear_gcsr_estat(val) \
gcsr_xchg(~(val), val, LOONGARCH_CSR_ESTAT)
#define kvm_read_hw_gcsr(id) gcsr_read(id)
#define kvm_write_hw_gcsr(id, val) gcsr_write(val, id)
#define kvm_save_hw_gcsr(csr, gid) (csr->csrs[gid] = gcsr_read(gid))
#define kvm_restore_hw_gcsr(csr, gid) (gcsr_write(csr->csrs[gid], gid))
int kvm_emu_iocsr(larch_inst inst, struct kvm_run *run, struct kvm_vcpu *vcpu);
static __always_inline unsigned long kvm_read_sw_gcsr(struct loongarch_csrs *csr, int gid)
{
return csr->csrs[gid];
}
static __always_inline void kvm_write_sw_gcsr(struct loongarch_csrs *csr, int gid, unsigned long val)
{
csr->csrs[gid] = val;
}
static __always_inline void kvm_set_sw_gcsr(struct loongarch_csrs *csr,
int gid, unsigned long val)
{
csr->csrs[gid] |= val;
}
static __always_inline void kvm_change_sw_gcsr(struct loongarch_csrs *csr,
int gid, unsigned long mask, unsigned long val)
{
unsigned long _mask = mask;
csr->csrs[gid] &= ~_mask;
csr->csrs[gid] |= val & _mask;
}
#endif /* __ASM_LOONGARCH_KVM_CSR_H__ */

237
arch/loongarch/include/asm/kvm_host.h Normal file
View File

@ -0,0 +1,237 @@
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Copyright (C) 2020-2023 Loongson Technology Corporation Limited
*/
#ifndef __ASM_LOONGARCH_KVM_HOST_H__
#define __ASM_LOONGARCH_KVM_HOST_H__
#include <linux/cpumask.h>
#include <linux/hrtimer.h>
#include <linux/interrupt.h>
#include <linux/kvm.h>
#include <linux/kvm_types.h>
#include <linux/mutex.h>
#include <linux/spinlock.h>
#include <linux/threads.h>
#include <linux/types.h>
#include <asm/inst.h>
#include <asm/kvm_mmu.h>
#include <asm/loongarch.h>
/* Loongarch KVM register ids */
#define KVM_GET_IOC_CSR_IDX(id) ((id & KVM_CSR_IDX_MASK) >> LOONGARCH_REG_SHIFT)
#define KVM_GET_IOC_CPUCFG_IDX(id) ((id & KVM_CPUCFG_IDX_MASK) >> LOONGARCH_REG_SHIFT)
#define KVM_MAX_VCPUS 256
#define KVM_MAX_CPUCFG_REGS 21
/* memory slots that does not exposed to userspace */
#define KVM_PRIVATE_MEM_SLOTS 0
#define KVM_HALT_POLL_NS_DEFAULT 500000
struct kvm_vm_stat {
struct kvm_vm_stat_generic generic;
u64 pages;
u64 hugepages;
};
struct kvm_vcpu_stat {
struct kvm_vcpu_stat_generic generic;
u64 int_exits;
u64 idle_exits;
u64 cpucfg_exits;
u64 signal_exits;
};
struct kvm_arch_memory_slot {
};
struct kvm_context {
unsigned long vpid_cache;
struct kvm_vcpu *last_vcpu;
};
struct kvm_world_switch {
int (*exc_entry)(void);
int (*enter_guest)(struct kvm_run *run, struct kvm_vcpu *vcpu);
unsigned long page_order;
};
#define MAX_PGTABLE_LEVELS 4
struct kvm_arch {
/* Guest physical mm */
kvm_pte_t *pgd;
unsigned long gpa_size;
unsigned long invalid_ptes[MAX_PGTABLE_LEVELS];
unsigned int pte_shifts[MAX_PGTABLE_LEVELS];
unsigned int root_level;
s64 time_offset;
struct kvm_context __percpu *vmcs;
};
#define CSR_MAX_NUMS 0x800
struct loongarch_csrs {
unsigned long csrs[CSR_MAX_NUMS];
};
/* Resume Flags */
#define RESUME_HOST 0
#define RESUME_GUEST 1
enum emulation_result {
EMULATE_DONE, /* no further processing */
EMULATE_DO_MMIO, /* kvm_run filled with MMIO request */
EMULATE_DO_IOCSR, /* handle IOCSR request */
EMULATE_FAIL, /* can't emulate this instruction */
EMULATE_EXCEPT, /* A guest exception has been generated */
};
#define KVM_LARCH_FPU (0x1 << 0)
#define KVM_LARCH_SWCSR_LATEST (0x1 << 1)
#define KVM_LARCH_HWCSR_USABLE (0x1 << 2)
struct kvm_vcpu_arch {
/*
* Switch pointer-to-function type to unsigned long
* for loading the value into register directly.
*/
unsigned long host_eentry;
unsigned long guest_eentry;
/* Pointers stored here for easy accessing from assembly code */
int (*handle_exit)(struct kvm_run *run, struct kvm_vcpu *vcpu);
/* Host registers preserved across guest mode execution */
unsigned long host_sp;
unsigned long host_tp;
unsigned long host_pgd;
/* Host CSRs are used when handling exits from guest */
unsigned long badi;
unsigned long badv;
unsigned long host_ecfg;
unsigned long host_estat;
unsigned long host_percpu;
/* GPRs */
unsigned long gprs[32];
unsigned long pc;
/* Which auxiliary state is loaded (KVM_LARCH_*) */
unsigned int aux_inuse;
/* FPU state */
struct loongarch_fpu fpu FPU_ALIGN;
/* CSR state */
struct loongarch_csrs *csr;
/* GPR used as IO source/target */
u32 io_gpr;
/* KVM register to control count timer */
u32 count_ctl;
struct hrtimer swtimer;
/* Bitmask of intr that are pending */
unsigned long irq_pending;
/* Bitmask of pending intr to be cleared */
unsigned long irq_clear;
/* Bitmask of exceptions that are pending */
unsigned long exception_pending;
unsigned int esubcode;
/* Cache for pages needed inside spinlock regions */
struct kvm_mmu_memory_cache mmu_page_cache;
/* vcpu's vpid */
u64 vpid;
/* Frequency of stable timer in Hz */
u64 timer_mhz;
ktime_t expire;
/* Last CPU the vCPU state was loaded on */
int last_sched_cpu;
/* mp state */
struct kvm_mp_state mp_state;
/* cpucfg */
u32 cpucfg[KVM_MAX_CPUCFG_REGS];
};
static inline unsigned long readl_sw_gcsr(struct loongarch_csrs *csr, int reg)
{
return csr->csrs[reg];
}
static inline void writel_sw_gcsr(struct loongarch_csrs *csr, int reg, unsigned long val)
{
csr->csrs[reg] = val;
}
/* Debug: dump vcpu state */
int kvm_arch_vcpu_dump_regs(struct kvm_vcpu *vcpu);
/* MMU handling */
void kvm_flush_tlb_all(void);
void kvm_flush_tlb_gpa(struct kvm_vcpu *vcpu, unsigned long gpa);
int kvm_handle_mm_fault(struct kvm_vcpu *vcpu, unsigned long badv, bool write);
#define KVM_ARCH_WANT_MMU_NOTIFIER
void kvm_set_spte_hva(struct kvm *kvm, unsigned long hva, pte_t pte);
int kvm_unmap_hva_range(struct kvm *kvm, unsigned long start, unsigned long end, bool blockable);
int kvm_age_hva(struct kvm *kvm, unsigned long start, unsigned long end);
int kvm_test_age_hva(struct kvm *kvm, unsigned long hva);
static inline void update_pc(struct kvm_vcpu_arch *arch)
{
arch->pc += 4;
}
/*
* kvm_is_ifetch_fault() - Find whether a TLBL exception is due to ifetch fault.
* @vcpu: Virtual CPU.
*
* Returns: Whether the TLBL exception was likely due to an instruction
* fetch fault rather than a data load fault.
*/
static inline bool kvm_is_ifetch_fault(struct kvm_vcpu_arch *arch)
{
return arch->pc == arch->badv;
}
/* Misc */
static inline void kvm_arch_hardware_unsetup(void) {}
static inline void kvm_arch_sync_events(struct kvm *kvm) {}
static inline void kvm_arch_memslots_updated(struct kvm *kvm, u64 gen) {}
static inline void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu) {}
static inline void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) {}
static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
static inline void kvm_arch_vcpu_block_finish(struct kvm_vcpu *vcpu) {}
static inline void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot) {}
void kvm_check_vpid(struct kvm_vcpu *vcpu);
enum hrtimer_restart kvm_swtimer_wakeup(struct hrtimer *timer);
void kvm_arch_flush_remote_tlbs_memslot(struct kvm *kvm, const struct kvm_memory_slot *memslot);
void kvm_init_vmcs(struct kvm *kvm);
void kvm_exc_entry(void);
int kvm_enter_guest(struct kvm_run *run, struct kvm_vcpu *vcpu);
extern unsigned long vpid_mask;
extern const unsigned long kvm_exception_size;
extern const unsigned long kvm_enter_guest_size;
extern struct kvm_world_switch *kvm_loongarch_ops;
#define SW_GCSR (1 << 0)
#define HW_GCSR (1 << 1)
#define INVALID_GCSR (1 << 2)
int get_gcsr_flag(int csr);
void set_hw_gcsr(int csr_id, unsigned long val);
#endif /* __ASM_LOONGARCH_KVM_HOST_H__ */

139
arch/loongarch/include/asm/kvm_mmu.h Normal file
View File

@ -0,0 +1,139 @@
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Copyright (C) 2020-2023 Loongson Technology Corporation Limited
*/
#ifndef __ASM_LOONGARCH_KVM_MMU_H__
#define __ASM_LOONGARCH_KVM_MMU_H__
#include <linux/kvm_host.h>
#include <asm/pgalloc.h>
#include <asm/tlb.h>
/*
* KVM_MMU_CACHE_MIN_PAGES is the number of GPA page table translation levels
* for which pages need to be cached.
*/
#define KVM_MMU_CACHE_MIN_PAGES (CONFIG_PGTABLE_LEVELS - 1)
#define _KVM_FLUSH_PGTABLE 0x1
#define _KVM_HAS_PGMASK 0x2
#define kvm_pfn_pte(pfn, prot) (((pfn) << PFN_PTE_SHIFT) | pgprot_val(prot))
#define kvm_pte_pfn(x) ((phys_addr_t)((x & _PFN_MASK) >> PFN_PTE_SHIFT))
typedef unsigned long kvm_pte_t;
typedef struct kvm_ptw_ctx kvm_ptw_ctx;
typedef int (*kvm_pte_ops)(kvm_pte_t *pte, phys_addr_t addr, kvm_ptw_ctx *ctx);
struct kvm_ptw_ctx {
kvm_pte_ops ops;
unsigned long flag;
/* for kvm_arch_mmu_enable_log_dirty_pt_masked use */
unsigned long mask;
unsigned long gfn;
/* page walk mmu info */
unsigned int level;
unsigned long pgtable_shift;
unsigned long invalid_entry;
unsigned long *invalid_ptes;
unsigned int *pte_shifts;
void *opaque;
/* free pte table page list */
struct list_head list;
};
kvm_pte_t *kvm_pgd_alloc(void);
static inline void kvm_set_pte(kvm_pte_t *ptep, kvm_pte_t val)
{
WRITE_ONCE(*ptep, val);
}
static inline int kvm_pte_write(kvm_pte_t pte) { return pte & _PAGE_WRITE; }
static inline int kvm_pte_dirty(kvm_pte_t pte) { return pte & _PAGE_DIRTY; }
static inline int kvm_pte_young(kvm_pte_t pte) { return pte & _PAGE_ACCESSED; }
static inline int kvm_pte_huge(kvm_pte_t pte) { return pte & _PAGE_HUGE; }
static inline kvm_pte_t kvm_pte_mkyoung(kvm_pte_t pte)
{
return pte | _PAGE_ACCESSED;
}
static inline kvm_pte_t kvm_pte_mkold(kvm_pte_t pte)
{
return pte & ~_PAGE_ACCESSED;
}
static inline kvm_pte_t kvm_pte_mkdirty(kvm_pte_t pte)
{
return pte | _PAGE_DIRTY;
}
static inline kvm_pte_t kvm_pte_mkclean(kvm_pte_t pte)
{
return pte & ~_PAGE_DIRTY;
}
static inline kvm_pte_t kvm_pte_mkhuge(kvm_pte_t pte)
{
return pte | _PAGE_HUGE;
}
static inline kvm_pte_t kvm_pte_mksmall(kvm_pte_t pte)
{
return pte & ~_PAGE_HUGE;
}
static inline int kvm_need_flush(kvm_ptw_ctx *ctx)
{
return ctx->flag & _KVM_FLUSH_PGTABLE;
}
static inline kvm_pte_t *kvm_pgtable_offset(kvm_ptw_ctx *ctx, kvm_pte_t *table,
phys_addr_t addr)
{
return table + ((addr >> ctx->pgtable_shift) & (PTRS_PER_PTE - 1));
}
static inline phys_addr_t kvm_pgtable_addr_end(kvm_ptw_ctx *ctx,
phys_addr_t addr, phys_addr_t end)
{
phys_addr_t boundary, size;
size = 0x1UL << ctx->pgtable_shift;
boundary = (addr + size) & ~(size - 1);
return (boundary - 1 < end - 1) ? boundary : end;
}
static inline int kvm_pte_present(kvm_ptw_ctx *ctx, kvm_pte_t *entry)
{
if (!ctx || ctx->level == 0)
return !!(*entry & _PAGE_PRESENT);
return *entry != ctx->invalid_entry;
}
static inline int kvm_pte_none(kvm_ptw_ctx *ctx, kvm_pte_t *entry)
{
return *entry == ctx->invalid_entry;
}
static inline void kvm_ptw_enter(kvm_ptw_ctx *ctx)
{
ctx->level--;
ctx->pgtable_shift = ctx->pte_shifts[ctx->level];
ctx->invalid_entry = ctx->invalid_ptes[ctx->level];
}
static inline void kvm_ptw_exit(kvm_ptw_ctx *ctx)
{
ctx->level++;
ctx->pgtable_shift = ctx->pte_shifts[ctx->level];
ctx->invalid_entry = ctx->invalid_ptes[ctx->level];
}
#endif /* __ASM_LOONGARCH_KVM_MMU_H__ */

11
arch/loongarch/include/asm/kvm_types.h Normal file
View File

@ -0,0 +1,11 @@
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Copyright (C) 2020-2023 Loongson Technology Corporation Limited
*/
#ifndef _ASM_LOONGARCH_KVM_TYPES_H
#define _ASM_LOONGARCH_KVM_TYPES_H
#define KVM_ARCH_NR_OBJS_PER_MEMORY_CACHE 40
#endif /* _ASM_LOONGARCH_KVM_TYPES_H */

93
arch/loongarch/include/asm/kvm_vcpu.h Normal file
View File

@ -0,0 +1,93 @@
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Copyright (C) 2020-2023 Loongson Technology Corporation Limited
*/
#ifndef __ASM_LOONGARCH_KVM_VCPU_H__
#define __ASM_LOONGARCH_KVM_VCPU_H__
#include <linux/kvm_host.h>
#include <asm/loongarch.h>
/* Controlled by 0x5 guest estat */
#define CPU_SIP0 (_ULCAST_(1))
#define CPU_SIP1 (_ULCAST_(1) << 1)
#define CPU_PMU (_ULCAST_(1) << 10)
#define CPU_TIMER (_ULCAST_(1) << 11)
#define CPU_IPI (_ULCAST_(1) << 12)
/* Controlled by 0x52 guest exception VIP aligned to estat bit 5~12 */
#define CPU_IP0 (_ULCAST_(1))
#define CPU_IP1 (_ULCAST_(1) << 1)
#define CPU_IP2 (_ULCAST_(1) << 2)
#define CPU_IP3 (_ULCAST_(1) << 3)
#define CPU_IP4 (_ULCAST_(1) << 4)
#define CPU_IP5 (_ULCAST_(1) << 5)
#define CPU_IP6 (_ULCAST_(1) << 6)
#define CPU_IP7 (_ULCAST_(1) << 7)
#define MNSEC_PER_SEC (NSEC_PER_SEC >> 20)
/* KVM_IRQ_LINE irq field index values */
#define KVM_LOONGSON_IRQ_TYPE_SHIFT 24
#define KVM_LOONGSON_IRQ_TYPE_MASK 0xff
#define KVM_LOONGSON_IRQ_VCPU_SHIFT 16
#define KVM_LOONGSON_IRQ_VCPU_MASK 0xff
#define KVM_LOONGSON_IRQ_NUM_SHIFT 0
#define KVM_LOONGSON_IRQ_NUM_MASK 0xffff
typedef union loongarch_instruction larch_inst;
typedef int (*exit_handle_fn)(struct kvm_vcpu *);
int kvm_emu_mmio_read(struct kvm_vcpu *vcpu, larch_inst inst);
int kvm_emu_mmio_write(struct kvm_vcpu *vcpu, larch_inst inst);
int kvm_complete_mmio_read(struct kvm_vcpu *vcpu, struct kvm_run *run);
int kvm_complete_iocsr_read(struct kvm_vcpu *vcpu, struct kvm_run *run);
int kvm_emu_idle(struct kvm_vcpu *vcpu);
int kvm_pending_timer(struct kvm_vcpu *vcpu);
int kvm_handle_fault(struct kvm_vcpu *vcpu, int fault);
void kvm_deliver_intr(struct kvm_vcpu *vcpu);
void kvm_deliver_exception(struct kvm_vcpu *vcpu);
void kvm_own_fpu(struct kvm_vcpu *vcpu);
void kvm_lose_fpu(struct kvm_vcpu *vcpu);
void kvm_save_fpu(struct loongarch_fpu *fpu);
void kvm_restore_fpu(struct loongarch_fpu *fpu);
void kvm_restore_fcsr(struct loongarch_fpu *fpu);
void kvm_acquire_timer(struct kvm_vcpu *vcpu);
void kvm_init_timer(struct kvm_vcpu *vcpu, unsigned long hz);
void kvm_reset_timer(struct kvm_vcpu *vcpu);
void kvm_save_timer(struct kvm_vcpu *vcpu);
void kvm_restore_timer(struct kvm_vcpu *vcpu);
int kvm_vcpu_ioctl_interrupt(struct kvm_vcpu *vcpu, struct kvm_interrupt *irq);
/*
* Loongarch KVM guest interrupt handling
*/
static inline void kvm_queue_irq(struct kvm_vcpu *vcpu, unsigned int irq)
{
set_bit(irq, &vcpu->arch.irq_pending);
clear_bit(irq, &vcpu->arch.irq_clear);
}
static inline void kvm_dequeue_irq(struct kvm_vcpu *vcpu, unsigned int irq)
{
clear_bit(irq, &vcpu->arch.irq_pending);
set_bit(irq, &vcpu->arch.irq_clear);
}
static inline int kvm_queue_exception(struct kvm_vcpu *vcpu,
unsigned int code, unsigned int subcode)
{
/* only one exception can be injected */
if (!vcpu->arch.exception_pending) {
set_bit(code, &vcpu->arch.exception_pending);
vcpu->arch.esubcode = subcode;
return 0;
} else
return -1;
}
#endif /* __ASM_LOONGARCH_KVM_VCPU_H__ */

19
arch/loongarch/include/asm/loongarch.h
View File

@ -226,6 +226,7 @@
#define LOONGARCH_CSR_ECFG 0x4 /* Exception config */
#define CSR_ECFG_VS_SHIFT 16
#define CSR_ECFG_VS_WIDTH 3
#define CSR_ECFG_VS_SHIFT_END (CSR_ECFG_VS_SHIFT + CSR_ECFG_VS_WIDTH - 1)
#define CSR_ECFG_VS (_ULCAST_(0x7) << CSR_ECFG_VS_SHIFT)
#define CSR_ECFG_IM_SHIFT 0
#define CSR_ECFG_IM_WIDTH 14
@ -314,13 +315,14 @@
#define CSR_TLBLO1_V (_ULCAST_(0x1) << CSR_TLBLO1_V_SHIFT)
#define LOONGARCH_CSR_GTLBC 0x15 /* Guest TLB control */
#define CSR_GTLBC_RID_SHIFT 16
#define CSR_GTLBC_RID_WIDTH 8
#define CSR_GTLBC_RID (_ULCAST_(0xff) << CSR_GTLBC_RID_SHIFT)
#define CSR_GTLBC_TGID_SHIFT 16
#define CSR_GTLBC_TGID_WIDTH 8
#define CSR_GTLBC_TGID_SHIFT_END (CSR_GTLBC_TGID_SHIFT + CSR_GTLBC_TGID_WIDTH - 1)
#define CSR_GTLBC_TGID (_ULCAST_(0xff) << CSR_GTLBC_TGID_SHIFT)
#define CSR_GTLBC_TOTI_SHIFT 13
#define CSR_GTLBC_TOTI (_ULCAST_(0x1) << CSR_GTLBC_TOTI_SHIFT)
#define CSR_GTLBC_USERID_SHIFT 12
#define CSR_GTLBC_USERID (_ULCAST_(0x1) << CSR_GTLBC_USERID_SHIFT)
#define CSR_GTLBC_USETGID_SHIFT 12
#define CSR_GTLBC_USETGID (_ULCAST_(0x1) << CSR_GTLBC_USETGID_SHIFT)
#define CSR_GTLBC_GMTLBSZ_SHIFT 0
#define CSR_GTLBC_GMTLBSZ_WIDTH 6
#define CSR_GTLBC_GMTLBSZ (_ULCAST_(0x3f) << CSR_GTLBC_GMTLBSZ_SHIFT)
@ -475,6 +477,7 @@
#define LOONGARCH_CSR_GSTAT 0x50 /* Guest status */
#define CSR_GSTAT_GID_SHIFT 16
#define CSR_GSTAT_GID_WIDTH 8
#define CSR_GSTAT_GID_SHIFT_END (CSR_GSTAT_GID_SHIFT + CSR_GSTAT_GID_WIDTH - 1)
#define CSR_GSTAT_GID (_ULCAST_(0xff) << CSR_GSTAT_GID_SHIFT)
#define CSR_GSTAT_GIDBIT_SHIFT 4
#define CSR_GSTAT_GIDBIT_WIDTH 6
@ -525,6 +528,12 @@
#define CSR_GCFG_MATC_GUEST (_ULCAST_(0x0) << CSR_GCFG_MATC_SHITF)
#define CSR_GCFG_MATC_ROOT (_ULCAST_(0x1) << CSR_GCFG_MATC_SHITF)
#define CSR_GCFG_MATC_NEST (_ULCAST_(0x2) << CSR_GCFG_MATC_SHITF)
#define CSR_GCFG_MATP_NEST_SHIFT 2
#define CSR_GCFG_MATP_NEST (_ULCAST_(0x1) << CSR_GCFG_MATP_NEST_SHIFT)
#define CSR_GCFG_MATP_ROOT_SHIFT 1
#define CSR_GCFG_MATP_ROOT (_ULCAST_(0x1) << CSR_GCFG_MATP_ROOT_SHIFT)
#define CSR_GCFG_MATP_GUEST_SHIFT 0
#define CSR_GCFG_MATP_GUEST (_ULCAST_(0x1) << CSR_GCFG_MATP_GUEST_SHIFT)
#define LOONGARCH_CSR_GINTC 0x52 /* Guest interrupt control */
#define CSR_GINTC_HC_SHIFT 16

108
arch/loongarch/include/uapi/asm/kvm.h Normal file
View File

@ -0,0 +1,108 @@
/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
/*
* Copyright (C) 2020-2023 Loongson Technology Corporation Limited
*/
#ifndef __UAPI_ASM_LOONGARCH_KVM_H
#define __UAPI_ASM_LOONGARCH_KVM_H
#include <linux/types.h>
/*
* KVM LoongArch specific structures and definitions.
*
* Some parts derived from the x86 version of this file.
*/
#define __KVM_HAVE_READONLY_MEM
#define KVM_COALESCED_MMIO_PAGE_OFFSET 1
#define KVM_DIRTY_LOG_PAGE_OFFSET 64
/*
* for KVM_GET_REGS and KVM_SET_REGS
*/
struct kvm_regs {
/* out (KVM_GET_REGS) / in (KVM_SET_REGS) */
__u64 gpr[32];
__u64 pc;
};
/*
* for KVM_GET_FPU and KVM_SET_FPU
*/
struct kvm_fpu {
__u32 fcsr;
__u64 fcc; /* 8x8 */
struct kvm_fpureg {
__u64 val64[4];
} fpr[32];
};
/*
* For LoongArch, we use KVM_SET_ONE_REG and KVM_GET_ONE_REG to access various
* registers. The id field is broken down as follows:
*
* bits[63..52] - As per linux/kvm.h
* bits[51..32] - Must be zero.
* bits[31..16] - Register set.
*
* Register set = 0: GP registers from kvm_regs (see definitions below).
*
* Register set = 1: CSR registers.
*
* Register set = 2: KVM specific registers (see definitions below).
*
* Register set = 3: FPU / SIMD registers (see definitions below).
*
* Other sets registers may be added in the future. Each set would
* have its own identifier in bits[31..16].
*/
#define KVM_REG_LOONGARCH_GPR (KVM_REG_LOONGARCH | 0x00000ULL)
#define KVM_REG_LOONGARCH_CSR (KVM_REG_LOONGARCH | 0x10000ULL)
#define KVM_REG_LOONGARCH_KVM (KVM_REG_LOONGARCH | 0x20000ULL)
#define KVM_REG_LOONGARCH_FPSIMD (KVM_REG_LOONGARCH | 0x30000ULL)
#define KVM_REG_LOONGARCH_CPUCFG (KVM_REG_LOONGARCH | 0x40000ULL)
#define KVM_REG_LOONGARCH_MASK (KVM_REG_LOONGARCH | 0x70000ULL)
#define KVM_CSR_IDX_MASK 0x7fff
#define KVM_CPUCFG_IDX_MASK 0x7fff
/*
* KVM_REG_LOONGARCH_KVM - KVM specific control registers.
*/
#define KVM_REG_LOONGARCH_COUNTER (KVM_REG_LOONGARCH_KVM | KVM_REG_SIZE_U64 | 1)
#define KVM_REG_LOONGARCH_VCPU_RESET (KVM_REG_LOONGARCH_KVM | KVM_REG_SIZE_U64 | 2)
#define LOONGARCH_REG_SHIFT 3
#define LOONGARCH_REG_64(TYPE, REG) (TYPE | KVM_REG_SIZE_U64 | (REG << LOONGARCH_REG_SHIFT))
#define KVM_IOC_CSRID(REG) LOONGARCH_REG_64(KVM_REG_LOONGARCH_CSR, REG)
#define KVM_IOC_CPUCFG(REG) LOONGARCH_REG_64(KVM_REG_LOONGARCH_CPUCFG, REG)
struct kvm_debug_exit_arch {
};
/* for KVM_SET_GUEST_DEBUG */
struct kvm_guest_debug_arch {
};
/* definition of registers in kvm_run */
struct kvm_sync_regs {
};
/* dummy definition */
struct kvm_sregs {
};
struct kvm_iocsr_entry {
__u32 addr;
__u32 pad;
__u64 data;
};
#define KVM_NR_IRQCHIPS 1
#define KVM_IRQCHIP_NUM_PINS 64
#define KVM_MAX_CORES 256
#endif /* __UAPI_ASM_LOONGARCH_KVM_H */

32
arch/loongarch/kernel/asm-offsets.c
View File

@ -9,6 +9,7 @@
#include <linux/mm.h>
#include <linux/kbuild.h>
#include <linux/suspend.h>
#include <linux/kvm_host.h>
#include <asm/cpu-info.h>
#include <asm/ptrace.h>
#include <asm/processor.h>
@ -289,3 +290,34 @@ void output_fgraph_ret_regs_defines(void)
BLANK();
}
#endif
void output_kvm_defines(void)
{
COMMENT("KVM/LoongArch Specific offsets.");
OFFSET(VCPU_FCC, kvm_vcpu_arch, fpu.fcc);
OFFSET(VCPU_FCSR0, kvm_vcpu_arch, fpu.fcsr);
BLANK();
OFFSET(KVM_VCPU_ARCH, kvm_vcpu, arch);
OFFSET(KVM_VCPU_KVM, kvm_vcpu, kvm);
OFFSET(KVM_VCPU_RUN, kvm_vcpu, run);
BLANK();
OFFSET(KVM_ARCH_HSP, kvm_vcpu_arch, host_sp);
OFFSET(KVM_ARCH_HTP, kvm_vcpu_arch, host_tp);
OFFSET(KVM_ARCH_HPGD, kvm_vcpu_arch, host_pgd);
OFFSET(KVM_ARCH_HANDLE_EXIT, kvm_vcpu_arch, handle_exit);
OFFSET(KVM_ARCH_HEENTRY, kvm_vcpu_arch, host_eentry);
OFFSET(KVM_ARCH_GEENTRY, kvm_vcpu_arch, guest_eentry);
OFFSET(KVM_ARCH_GPC, kvm_vcpu_arch, pc);
OFFSET(KVM_ARCH_GGPR, kvm_vcpu_arch, gprs);
OFFSET(KVM_ARCH_HBADI, kvm_vcpu_arch, badi);
OFFSET(KVM_ARCH_HBADV, kvm_vcpu_arch, badv);
OFFSET(KVM_ARCH_HECFG, kvm_vcpu_arch, host_ecfg);
OFFSET(KVM_ARCH_HESTAT, kvm_vcpu_arch, host_estat);
OFFSET(KVM_ARCH_HPERCPU, kvm_vcpu_arch, host_percpu);
OFFSET(KVM_GPGD, kvm, arch.pgd);
BLANK();
}

40
arch/loongarch/kvm/Kconfig Normal file
View File

@ -0,0 +1,40 @@
# SPDX-License-Identifier: GPL-2.0
#
# KVM configuration
#
source "virt/kvm/Kconfig"
menuconfig VIRTUALIZATION
bool "Virtualization"
help
Say Y here to get to see options for using your Linux host to run
other operating systems inside virtual machines (guests).
This option alone does not add any kernel code.
If you say N, all options in this submenu will be skipped and
disabled.
if VIRTUALIZATION
config KVM
tristate "Kernel-based Virtual Machine (KVM) support"
depends on AS_HAS_LVZ_EXTENSION
depends on HAVE_KVM
select HAVE_KVM_DIRTY_RING_ACQ_REL
select HAVE_KVM_EVENTFD
select HAVE_KVM_VCPU_ASYNC_IOCTL
select KVM_GENERIC_DIRTYLOG_READ_PROTECT
select KVM_GENERIC_HARDWARE_ENABLING
select KVM_MMIO
select KVM_XFER_TO_GUEST_WORK
select MMU_NOTIFIER
select PREEMPT_NOTIFIERS
help
Support hosting virtualized guest machines using
hardware virtualization extensions. You will need
a processor equipped with virtualization extensions.
If unsure, say N.
endif # VIRTUALIZATION

22
arch/loongarch/kvm/Makefile Normal file
View File

@ -0,0 +1,22 @@
# SPDX-License-Identifier: GPL-2.0
#
# Makefile for LoongArch KVM support
#
ccflags-y += -I $(srctree)/$(src)
include $(srctree)/virt/kvm/Makefile.kvm
obj-$(CONFIG_KVM) += kvm.o
kvm-y += exit.o
kvm-y += interrupt.o
kvm-y += main.o
kvm-y += mmu.o
kvm-y += switch.o
kvm-y += timer.o
kvm-y += tlb.o
kvm-y += vcpu.o
kvm-y += vm.o
CFLAGS_exit.o += $(call cc-option,-Wno-override-init,)

696
arch/loongarch/kvm/exit.c Normal file
View File

@ -0,0 +1,696 @@
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (C) 2020-2023 Loongson Technology Corporation Limited
*/
#include <linux/err.h>
#include <linux/errno.h>
#include <linux/kvm_host.h>
#include <linux/module.h>
#include <linux/preempt.h>
#include <linux/vmalloc.h>
#include <asm/fpu.h>
#include <asm/inst.h>
#include <asm/loongarch.h>
#include <asm/mmzone.h>
#include <asm/numa.h>
#include <asm/time.h>
#include <asm/tlb.h>
#include <asm/kvm_csr.h>
#include <asm/kvm_vcpu.h>
#include "trace.h"
static unsigned long kvm_emu_read_csr(struct kvm_vcpu *vcpu, int csrid)
{
unsigned long val = 0;
struct loongarch_csrs *csr = vcpu->arch.csr;
/*
* From LoongArch Reference Manual Volume 1 Chapter 4.2.1
* For undefined CSR id, return value is 0
*/
if (get_gcsr_flag(csrid) & SW_GCSR)
val = kvm_read_sw_gcsr(csr, csrid);
else
pr_warn_once("Unsupported csrrd 0x%x with pc %lx\n", csrid, vcpu->arch.pc);
return val;
}
static unsigned long kvm_emu_write_csr(struct kvm_vcpu *vcpu, int csrid, unsigned long val)
{
unsigned long old = 0;
struct loongarch_csrs *csr = vcpu->arch.csr;
if (get_gcsr_flag(csrid) & SW_GCSR) {
old = kvm_read_sw_gcsr(csr, csrid);
kvm_write_sw_gcsr(csr, csrid, val);
} else
pr_warn_once("Unsupported csrwr 0x%x with pc %lx\n", csrid, vcpu->arch.pc);
return old;
}
static unsigned long kvm_emu_xchg_csr(struct kvm_vcpu *vcpu, int csrid,
unsigned long csr_mask, unsigned long val)
{
unsigned long old = 0;
struct loongarch_csrs *csr = vcpu->arch.csr;
if (get_gcsr_flag(csrid) & SW_GCSR) {
old = kvm_read_sw_gcsr(csr, csrid);
val = (old & ~csr_mask) | (val & csr_mask);
kvm_write_sw_gcsr(csr, csrid, val);
old = old & csr_mask;
} else
pr_warn_once("Unsupported csrxchg 0x%x with pc %lx\n", csrid, vcpu->arch.pc);
return old;
}
static int kvm_handle_csr(struct kvm_vcpu *vcpu, larch_inst inst)
{
unsigned int rd, rj, csrid;
unsigned long csr_mask, val = 0;
/*
* CSR value mask imm
* rj = 0 means csrrd
* rj = 1 means csrwr
* rj != 0,1 means csrxchg
*/
rd = inst.reg2csr_format.rd;
rj = inst.reg2csr_format.rj;
csrid = inst.reg2csr_format.csr;
/* Process CSR ops */
switch (rj) {
case 0: /* process csrrd */
val = kvm_emu_read_csr(vcpu, csrid);
vcpu->arch.gprs[rd] = val;
break;
case 1: /* process csrwr */
val = vcpu->arch.gprs[rd];
val = kvm_emu_write_csr(vcpu, csrid, val);
vcpu->arch.gprs[rd] = val;
break;
default: /* process csrxchg */
val = vcpu->arch.gprs[rd];
csr_mask = vcpu->arch.gprs[rj];
val = kvm_emu_xchg_csr(vcpu, csrid, csr_mask, val);
vcpu->arch.gprs[rd] = val;
}
return EMULATE_DONE;
}
int kvm_emu_iocsr(larch_inst inst, struct kvm_run *run, struct kvm_vcpu *vcpu)
{
int ret;
unsigned long val;
u32 addr, rd, rj, opcode;
/*
* Each IOCSR with different opcode
*/
rd = inst.reg2_format.rd;
rj = inst.reg2_format.rj;
opcode = inst.reg2_format.opcode;
addr = vcpu->arch.gprs[rj];
ret = EMULATE_DO_IOCSR;
run->iocsr_io.phys_addr = addr;
run->iocsr_io.is_write = 0;
/* LoongArch is Little endian */
switch (opcode) {
case iocsrrdb_op:
run->iocsr_io.len = 1;
break;
case iocsrrdh_op:
run->iocsr_io.len = 2;
break;
case iocsrrdw_op:
run->iocsr_io.len = 4;
break;
case iocsrrdd_op:
run->iocsr_io.len = 8;
break;
case iocsrwrb_op:
run->iocsr_io.len = 1;
run->iocsr_io.is_write = 1;
break;
case iocsrwrh_op:
run->iocsr_io.len = 2;
run->iocsr_io.is_write = 1;
break;
case iocsrwrw_op:
run->iocsr_io.len = 4;
run->iocsr_io.is_write = 1;
break;
case iocsrwrd_op:
run->iocsr_io.len = 8;
run->iocsr_io.is_write = 1;
break;
default:
ret = EMULATE_FAIL;
break;
}
if (ret == EMULATE_DO_IOCSR) {
if (run->iocsr_io.is_write) {
val = vcpu->arch.gprs[rd];
memcpy(run->iocsr_io.data, &val, run->iocsr_io.len);
}
vcpu->arch.io_gpr = rd;
}
return ret;
}
int kvm_complete_iocsr_read(struct kvm_vcpu *vcpu, struct kvm_run *run)
{
enum emulation_result er = EMULATE_DONE;
unsigned long *gpr = &vcpu->arch.gprs[vcpu->arch.io_gpr];
switch (run->iocsr_io.len) {
case 1:
*gpr = *(s8 *)run->iocsr_io.data;
break;
case 2:
*gpr = *(s16 *)run->iocsr_io.data;
break;
case 4:
*gpr = *(s32 *)run->iocsr_io.data;
break;
case 8:
*gpr = *(s64 *)run->iocsr_io.data;
break;
default:
kvm_err("Bad IOCSR length: %d, addr is 0x%lx\n",
run->iocsr_io.len, vcpu->arch.badv);
er = EMULATE_FAIL;
break;
}
return er;
}
int kvm_emu_idle(struct kvm_vcpu *vcpu)
{
++vcpu->stat.idle_exits;
trace_kvm_exit_idle(vcpu, KVM_TRACE_EXIT_IDLE);
if (!kvm_arch_vcpu_runnable(vcpu)) {
/*
* Switch to the software timer before halt-polling/blocking as
* the guest's timer may be a break event for the vCPU, and the
* hypervisor timer runs only when the CPU is in guest mode.
* Switch before halt-polling so that KVM recognizes an expired
* timer before blocking.
*/
kvm_save_timer(vcpu);
kvm_vcpu_block(vcpu);
}
return EMULATE_DONE;
}
static int kvm_trap_handle_gspr(struct kvm_vcpu *vcpu)
{
int rd, rj;
unsigned int index;
unsigned long curr_pc;
larch_inst inst;
enum emulation_result er = EMULATE_DONE;
struct kvm_run *run = vcpu->run;
/* Fetch the instruction */
inst.word = vcpu->arch.badi;
curr_pc = vcpu->arch.pc;
update_pc(&vcpu->arch);
trace_kvm_exit_gspr(vcpu, inst.word);
er = EMULATE_FAIL;
switch (((inst.word >> 24) & 0xff)) {
case 0x0: /* CPUCFG GSPR */
if (inst.reg2_format.opcode == 0x1B) {
rd = inst.reg2_format.rd;
rj = inst.reg2_format.rj;
++vcpu->stat.cpucfg_exits;
index = vcpu->arch.gprs[rj];
er = EMULATE_DONE;
/*
* By LoongArch Reference Manual 2.2.10.5
* return value is 0 for undefined cpucfg index
*/
if (index < KVM_MAX_CPUCFG_REGS)
vcpu->arch.gprs[rd] = vcpu->arch.cpucfg[index];
else
vcpu->arch.gprs[rd] = 0;
}
break;
case 0x4: /* CSR{RD,WR,XCHG} GSPR */
er = kvm_handle_csr(vcpu, inst);
break;
case 0x6: /* Cache, Idle and IOCSR GSPR */
switch (((inst.word >> 22) & 0x3ff)) {
case 0x18: /* Cache GSPR */
er = EMULATE_DONE;
trace_kvm_exit_cache(vcpu, KVM_TRACE_EXIT_CACHE);
break;
case 0x19: /* Idle/IOCSR GSPR */
switch (((inst.word >> 15) & 0x1ffff)) {
case 0xc90: /* IOCSR GSPR */
er = kvm_emu_iocsr(inst, run, vcpu);
break;
case 0xc91: /* Idle GSPR */
er = kvm_emu_idle(vcpu);
break;
default:
er = EMULATE_FAIL;
break;
}
break;
default:
er = EMULATE_FAIL;
break;
}
break;
default:
er = EMULATE_FAIL;
break;
}
/* Rollback PC only if emulation was unsuccessful */
if (er == EMULATE_FAIL) {
kvm_err("[%#lx]%s: unsupported gspr instruction 0x%08x\n",
curr_pc, __func__, inst.word);
kvm_arch_vcpu_dump_regs(vcpu);
vcpu->arch.pc = curr_pc;
}
return er;
}
/*
* Trigger GSPR:
* 1) Execute CPUCFG instruction;
* 2) Execute CACOP/IDLE instructions;
* 3) Access to unimplemented CSRs/IOCSRs.
*/
static int kvm_handle_gspr(struct kvm_vcpu *vcpu)
{
int ret = RESUME_GUEST;
enum emulation_result er = EMULATE_DONE;
er = kvm_trap_handle_gspr(vcpu);
if (er == EMULATE_DONE) {
ret = RESUME_GUEST;
} else if (er == EMULATE_DO_MMIO) {
vcpu->run->exit_reason = KVM_EXIT_MMIO;
ret = RESUME_HOST;
} else if (er == EMULATE_DO_IOCSR) {
vcpu->run->exit_reason = KVM_EXIT_LOONGARCH_IOCSR;
ret = RESUME_HOST;
} else {
kvm_queue_exception(vcpu, EXCCODE_INE, 0);
ret = RESUME_GUEST;
}
return ret;
}
int kvm_emu_mmio_read(struct kvm_vcpu *vcpu, larch_inst inst)
{
int ret;
unsigned int op8, opcode, rd;
struct kvm_run *run = vcpu->run;
run->mmio.phys_addr = vcpu->arch.badv;
vcpu->mmio_needed = 2; /* signed */
op8 = (inst.word >> 24) & 0xff;
ret = EMULATE_DO_MMIO;
switch (op8) {
case 0x24 ... 0x27: /* ldptr.w/d process */
rd = inst.reg2i14_format.rd;
opcode = inst.reg2i14_format.opcode;
switch (opcode) {
case ldptrw_op:
run->mmio.len = 4;
break;
case ldptrd_op:
run->mmio.len = 8;
break;
default:
break;
}
break;
case 0x28 ... 0x2e: /* ld.b/h/w/d, ld.bu/hu/wu process */
rd = inst.reg2i12_format.rd;
opcode = inst.reg2i12_format.opcode;
switch (opcode) {
case ldb_op:
run->mmio.len = 1;
break;
case ldbu_op:
vcpu->mmio_needed = 1; /* unsigned */
run->mmio.len = 1;
break;
case ldh_op:
run->mmio.len = 2;
break;
case ldhu_op:
vcpu->mmio_needed = 1; /* unsigned */
run->mmio.len = 2;
break;
case ldw_op:
run->mmio.len = 4;
break;
case ldwu_op:
vcpu->mmio_needed = 1; /* unsigned */
run->mmio.len = 4;
break;
case ldd_op:
run->mmio.len = 8;
break;
default:
ret = EMULATE_FAIL;
break;
}
break;
case 0x38: /* ldx.b/h/w/d, ldx.bu/hu/wu process */
rd = inst.reg3_format.rd;
opcode = inst.reg3_format.opcode;
switch (opcode) {
case ldxb_op:
run->mmio.len = 1;
break;
case ldxbu_op:
run->mmio.len = 1;
vcpu->mmio_needed = 1; /* unsigned */
break;
case ldxh_op:
run->mmio.len = 2;
break;
case ldxhu_op:
run->mmio.len = 2;
vcpu->mmio_needed = 1; /* unsigned */
break;
case ldxw_op:
run->mmio.len = 4;
break;
case ldxwu_op:
run->mmio.len = 4;
vcpu->mmio_needed = 1; /* unsigned */
break;
case ldxd_op:
run->mmio.len = 8;
break;
default:
ret = EMULATE_FAIL;
break;
}
break;
default:
ret = EMULATE_FAIL;
}
if (ret == EMULATE_DO_MMIO) {
/* Set for kvm_complete_mmio_read() use */
vcpu->arch.io_gpr = rd;
run->mmio.is_write = 0;
vcpu->mmio_is_write = 0;
} else {
kvm_err("Read not supported Inst=0x%08x @%lx BadVaddr:%#lx\n",
inst.word, vcpu->arch.pc, vcpu->arch.badv);
kvm_arch_vcpu_dump_regs(vcpu);
vcpu->mmio_needed = 0;
}
return ret;
}
int kvm_complete_mmio_read(struct kvm_vcpu *vcpu, struct kvm_run *run)
{
enum emulation_result er = EMULATE_DONE;
unsigned long *gpr = &vcpu->arch.gprs[vcpu->arch.io_gpr];
/* Update with new PC */
update_pc(&vcpu->arch);
switch (run->mmio.len) {
case 1:
if (vcpu->mmio_needed == 2)
*gpr = *(s8 *)run->mmio.data;
else
*gpr = *(u8 *)run->mmio.data;
break;
case 2:
if (vcpu->mmio_needed == 2)
*gpr = *(s16 *)run->mmio.data;
else
*gpr = *(u16 *)run->mmio.data;
break;
case 4:
if (vcpu->mmio_needed == 2)
*gpr = *(s32 *)run->mmio.data;
else
*gpr = *(u32 *)run->mmio.data;
break;
case 8:
*gpr = *(s64 *)run->mmio.data;
break;
default:
kvm_err("Bad MMIO length: %d, addr is 0x%lx\n",
run->mmio.len, vcpu->arch.badv);
er = EMULATE_FAIL;
break;
}
return er;
}
int kvm_emu_mmio_write(struct kvm_vcpu *vcpu, larch_inst inst)
{
int ret;
unsigned int rd, op8, opcode;
unsigned long curr_pc, rd_val = 0;
struct kvm_run *run = vcpu->run;
void *data = run->mmio.data;
/*
* Update PC and hold onto current PC in case there is
* an error and we want to rollback the PC
*/
curr_pc = vcpu->arch.pc;
update_pc(&vcpu->arch);
op8 = (inst.word >> 24) & 0xff;
run->mmio.phys_addr = vcpu->arch.badv;
ret = EMULATE_DO_MMIO;
switch (op8) {
case 0x24 ... 0x27: /* stptr.w/d process */
rd = inst.reg2i14_format.rd;
opcode = inst.reg2i14_format.opcode;
switch (opcode) {
case stptrw_op:
run->mmio.len = 4;
*(unsigned int *)data = vcpu->arch.gprs[rd];
break;
case stptrd_op:
run->mmio.len = 8;
*(unsigned long *)data = vcpu->arch.gprs[rd];
break;
default:
ret = EMULATE_FAIL;
break;
}
break;
case 0x28 ... 0x2e: /* st.b/h/w/d process */
rd = inst.reg2i12_format.rd;
opcode = inst.reg2i12_format.opcode;
rd_val = vcpu->arch.gprs[rd];
switch (opcode) {
case stb_op:
run->mmio.len = 1;
*(unsigned char *)data = rd_val;
break;
case sth_op:
run->mmio.len = 2;
*(unsigned short *)data = rd_val;
break;
case stw_op:
run->mmio.len = 4;
*(unsigned int *)data = rd_val;
break;
case std_op:
run->mmio.len = 8;
*(unsigned long *)data = rd_val;
break;
default:
ret = EMULATE_FAIL;
break;
}
break;
case 0x38: /* stx.b/h/w/d process */
rd = inst.reg3_format.rd;
opcode = inst.reg3_format.opcode;
switch (opcode) {
case stxb_op:
run->mmio.len = 1;
*(unsigned char *)data = vcpu->arch.gprs[rd];
break;
case stxh_op:
run->mmio.len = 2;
*(unsigned short *)data = vcpu->arch.gprs[rd];
break;
case stxw_op:
run->mmio.len = 4;
*(unsigned int *)data = vcpu->arch.gprs[rd];
break;
case stxd_op:
run->mmio.len = 8;
*(unsigned long *)data = vcpu->arch.gprs[rd];
break;
default:
ret = EMULATE_FAIL;
break;
}
break;
default:
ret = EMULATE_FAIL;
}
if (ret == EMULATE_DO_MMIO) {
run->mmio.is_write = 1;
vcpu->mmio_needed = 1;
vcpu->mmio_is_write = 1;
} else {
vcpu->arch.pc = curr_pc;
kvm_err("Write not supported Inst=0x%08x @%lx BadVaddr:%#lx\n",
inst.word, vcpu->arch.pc, vcpu->arch.badv);
kvm_arch_vcpu_dump_regs(vcpu);
/* Rollback PC if emulation was unsuccessful */
}
return ret;
}
static int kvm_handle_rdwr_fault(struct kvm_vcpu *vcpu, bool write)
{
int ret;
larch_inst inst;
enum emulation_result er = EMULATE_DONE;
struct kvm_run *run = vcpu->run;
unsigned long badv = vcpu->arch.badv;
ret = kvm_handle_mm_fault(vcpu, badv, write);
if (ret) {
/* Treat as MMIO */
inst.word = vcpu->arch.badi;
if (write) {
er = kvm_emu_mmio_write(vcpu, inst);
} else {
/* A code fetch fault doesn't count as an MMIO */
if (kvm_is_ifetch_fault(&vcpu->arch)) {
kvm_queue_exception(vcpu, EXCCODE_ADE, EXSUBCODE_ADEF);
return RESUME_GUEST;
}
er = kvm_emu_mmio_read(vcpu, inst);
}
}
if (er == EMULATE_DONE) {
ret = RESUME_GUEST;
} else if (er == EMULATE_DO_MMIO) {
run->exit_reason = KVM_EXIT_MMIO;
ret = RESUME_HOST;
} else {
kvm_queue_exception(vcpu, EXCCODE_ADE, EXSUBCODE_ADEM);
ret = RESUME_GUEST;
}
return ret;
}
static int kvm_handle_read_fault(struct kvm_vcpu *vcpu)
{
return kvm_handle_rdwr_fault(vcpu, false);
}
static int kvm_handle_write_fault(struct kvm_vcpu *vcpu)
{
return kvm_handle_rdwr_fault(vcpu, true);
}
/**
* kvm_handle_fpu_disabled() - Guest used fpu however it is disabled at host
* @vcpu: Virtual CPU context.
*
* Handle when the guest attempts to use fpu which hasn't been allowed
* by the root context.
*/
static int kvm_handle_fpu_disabled(struct kvm_vcpu *vcpu)
{
struct kvm_run *run = vcpu->run;
/*
* If guest FPU not present, the FPU operation should have been
* treated as a reserved instruction!
* If FPU already in use, we shouldn't get this at all.
*/
if (WARN_ON(vcpu->arch.aux_inuse & KVM_LARCH_FPU)) {
kvm_err("%s internal error\n", __func__);
run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
return RESUME_HOST;
}
kvm_own_fpu(vcpu);
return RESUME_GUEST;
}
/*
* LoongArch KVM callback handling for unimplemented guest exiting
*/
static int kvm_fault_ni(struct kvm_vcpu *vcpu)
{
unsigned int ecode, inst;
unsigned long estat, badv;
/* Fetch the instruction */
inst = vcpu->arch.badi;
badv = vcpu->arch.badv;
estat = vcpu->arch.host_estat;
ecode = (estat & CSR_ESTAT_EXC) >> CSR_ESTAT_EXC_SHIFT;
kvm_err("ECode: %d PC=%#lx Inst=0x%08x BadVaddr=%#lx ESTAT=%#lx\n",
ecode, vcpu->arch.pc, inst, badv, read_gcsr_estat());
kvm_arch_vcpu_dump_regs(vcpu);
kvm_queue_exception(vcpu, EXCCODE_INE, 0);
return RESUME_GUEST;
}
static exit_handle_fn kvm_fault_tables[EXCCODE_INT_START] = {
[0 ... EXCCODE_INT_START - 1] = kvm_fault_ni,
[EXCCODE_TLBI] = kvm_handle_read_fault,
[EXCCODE_TLBL] = kvm_handle_read_fault,
[EXCCODE_TLBS] = kvm_handle_write_fault,
[EXCCODE_TLBM] = kvm_handle_write_fault,
[EXCCODE_FPDIS] = kvm_handle_fpu_disabled,
[EXCCODE_GSPR] = kvm_handle_gspr,
};
int kvm_handle_fault(struct kvm_vcpu *vcpu, int fault)
{
return kvm_fault_tables[fault](vcpu);
}

183
arch/loongarch/kvm/interrupt.c Normal file
View File

@ -0,0 +1,183 @@
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (C) 2020-2023 Loongson Technology Corporation Limited
*/
#include <linux/err.h>
#include <linux/errno.h>
#include <asm/kvm_csr.h>
#include <asm/kvm_vcpu.h>
static unsigned int priority_to_irq[EXCCODE_INT_NUM] = {
[INT_TI] = CPU_TIMER,
[INT_IPI] = CPU_IPI,
[INT_SWI0] = CPU_SIP0,
[INT_SWI1] = CPU_SIP1,
[INT_HWI0] = CPU_IP0,
[INT_HWI1] = CPU_IP1,
[INT_HWI2] = CPU_IP2,
[INT_HWI3] = CPU_IP3,
[INT_HWI4] = CPU_IP4,
[INT_HWI5] = CPU_IP5,
[INT_HWI6] = CPU_IP6,
[INT_HWI7] = CPU_IP7,
};
static int kvm_irq_deliver(struct kvm_vcpu *vcpu, unsigned int priority)
{
unsigned int irq = 0;
clear_bit(priority, &vcpu->arch.irq_pending);
if (priority < EXCCODE_INT_NUM)
irq = priority_to_irq[priority];
switch (priority) {
case INT_TI:
case INT_IPI:
case INT_SWI0:
case INT_SWI1:
set_gcsr_estat(irq);
break;
case INT_HWI0 ... INT_HWI7:
set_csr_gintc(irq);
break;
default:
break;
}
return 1;
}
static int kvm_irq_clear(struct kvm_vcpu *vcpu, unsigned int priority)
{
unsigned int irq = 0;
clear_bit(priority, &vcpu->arch.irq_clear);
if (priority < EXCCODE_INT_NUM)
irq = priority_to_irq[priority];
switch (priority) {
case INT_TI:
case INT_IPI:
case INT_SWI0:
case INT_SWI1:
clear_gcsr_estat(irq);
break;
case INT_HWI0 ... INT_HWI7:
clear_csr_gintc(irq);
break;
default:
break;
}
return 1;
}
void kvm_deliver_intr(struct kvm_vcpu *vcpu)
{
unsigned int priority;
unsigned long *pending = &vcpu->arch.irq_pending;
unsigned long *pending_clr = &vcpu->arch.irq_clear;
if (!(*pending) && !(*pending_clr))
return;
if (*pending_clr) {
priority = __ffs(*pending_clr);
while (priority <= INT_IPI) {
kvm_irq_clear(vcpu, priority);
priority = find_next_bit(pending_clr,
BITS_PER_BYTE * sizeof(*pending_clr),
priority + 1);
}
}
if (*pending) {
priority = __ffs(*pending);
while (priority <= INT_IPI) {
kvm_irq_deliver(vcpu, priority);
priority = find_next_bit(pending,
BITS_PER_BYTE * sizeof(*pending),
priority + 1);
}
}
}
int kvm_pending_timer(struct kvm_vcpu *vcpu)
{
return test_bit(INT_TI, &vcpu->arch.irq_pending);
}
/*
* Only support illegal instruction or illegal Address Error exception,
* Other exceptions are injected by hardware in kvm mode
*/
static void _kvm_deliver_exception(struct kvm_vcpu *vcpu,
unsigned int code, unsigned int subcode)
{
unsigned long val, vec_size;
/*
* BADV is added for EXCCODE_ADE exception
* Use PC register (GVA address) if it is instruction exeception
* Else use BADV from host side (GPA address) for data exeception
*/
if (code == EXCCODE_ADE) {
if (subcode == EXSUBCODE_ADEF)
val = vcpu->arch.pc;
else
val = vcpu->arch.badv;
kvm_write_hw_gcsr(LOONGARCH_CSR_BADV, val);
}
/* Set exception instruction */
kvm_write_hw_gcsr(LOONGARCH_CSR_BADI, vcpu->arch.badi);
/*
* Save CRMD in PRMD
* Set IRQ disabled and PLV0 with CRMD
*/
val = kvm_read_hw_gcsr(LOONGARCH_CSR_CRMD);
kvm_write_hw_gcsr(LOONGARCH_CSR_PRMD, val);
val = val & ~(CSR_CRMD_PLV | CSR_CRMD_IE);
kvm_write_hw_gcsr(LOONGARCH_CSR_CRMD, val);
/* Set exception PC address */
kvm_write_hw_gcsr(LOONGARCH_CSR_ERA, vcpu->arch.pc);
/*
* Set exception code
* Exception and interrupt can be inject at the same time
* Hardware will handle exception first and then extern interrupt
* Exception code is Ecode in ESTAT[16:21]
* Interrupt code in ESTAT[0:12]
*/
val = kvm_read_hw_gcsr(LOONGARCH_CSR_ESTAT);
val = (val & ~CSR_ESTAT_EXC) | code;
kvm_write_hw_gcsr(LOONGARCH_CSR_ESTAT, val);
/* Calculate expcetion entry address */
val = kvm_read_hw_gcsr(LOONGARCH_CSR_ECFG);
vec_size = (val & CSR_ECFG_VS) >> CSR_ECFG_VS_SHIFT;
if (vec_size)
vec_size = (1 << vec_size) * 4;
val = kvm_read_hw_gcsr(LOONGARCH_CSR_EENTRY);
vcpu->arch.pc = val + code * vec_size;
}
void kvm_deliver_exception(struct kvm_vcpu *vcpu)
{
unsigned int code;
unsigned long *pending = &vcpu->arch.exception_pending;
if (*pending) {
code = __ffs(*pending);
_kvm_deliver_exception(vcpu, code, vcpu->arch.esubcode);
*pending = 0;
vcpu->arch.esubcode = 0;
}
}

420
arch/loongarch/kvm/main.c Normal file
View File

@ -0,0 +1,420 @@
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (C) 2020-2023 Loongson Technology Corporation Limited
*/
#include <linux/err.h>
#include <linux/module.h>
#include <linux/kvm_host.h>
#include <asm/cacheflush.h>
#include <asm/cpufeature.h>
#include <asm/kvm_csr.h>
#include "trace.h"
unsigned long vpid_mask;
struct kvm_world_switch *kvm_loongarch_ops;
static int gcsr_flag[CSR_MAX_NUMS];
static struct kvm_context __percpu *vmcs;
int get_gcsr_flag(int csr)
{
if (csr < CSR_MAX_NUMS)
return gcsr_flag[csr];
return INVALID_GCSR;
}
static inline void set_gcsr_sw_flag(int csr)
{
if (csr < CSR_MAX_NUMS)
gcsr_flag[csr] |= SW_GCSR;
}
static inline void set_gcsr_hw_flag(int csr)
{
if (csr < CSR_MAX_NUMS)
gcsr_flag[csr] |= HW_GCSR;
}
/*
* The default value of gcsr_flag[CSR] is 0, and we use this
* function to set the flag to 1 (SW_GCSR) or 2 (HW_GCSR) if the
* gcsr is software or hardware. It will be used by get/set_gcsr,
* if gcsr_flag is HW we should use gcsrrd/gcsrwr to access it,
* else use software csr to emulate it.
*/
static void kvm_init_gcsr_flag(void)
{
set_gcsr_hw_flag(LOONGARCH_CSR_CRMD);
set_gcsr_hw_flag(LOONGARCH_CSR_PRMD);
set_gcsr_hw_flag(LOONGARCH_CSR_EUEN);
set_gcsr_hw_flag(LOONGARCH_CSR_MISC);
set_gcsr_hw_flag(LOONGARCH_CSR_ECFG);
set_gcsr_hw_flag(LOONGARCH_CSR_ESTAT);
set_gcsr_hw_flag(LOONGARCH_CSR_ERA);
set_gcsr_hw_flag(LOONGARCH_CSR_BADV);
set_gcsr_hw_flag(LOONGARCH_CSR_BADI);
set_gcsr_hw_flag(LOONGARCH_CSR_EENTRY);
set_gcsr_hw_flag(LOONGARCH_CSR_TLBIDX);
set_gcsr_hw_flag(LOONGARCH_CSR_TLBEHI);
set_gcsr_hw_flag(LOONGARCH_CSR_TLBELO0);
set_gcsr_hw_flag(LOONGARCH_CSR_TLBELO1);
set_gcsr_hw_flag(LOONGARCH_CSR_ASID);
set_gcsr_hw_flag(LOONGARCH_CSR_PGDL);
set_gcsr_hw_flag(LOONGARCH_CSR_PGDH);
set_gcsr_hw_flag(LOONGARCH_CSR_PGD);
set_gcsr_hw_flag(LOONGARCH_CSR_PWCTL0);
set_gcsr_hw_flag(LOONGARCH_CSR_PWCTL1);
set_gcsr_hw_flag(LOONGARCH_CSR_STLBPGSIZE);
set_gcsr_hw_flag(LOONGARCH_CSR_RVACFG);
set_gcsr_hw_flag(LOONGARCH_CSR_CPUID);
set_gcsr_hw_flag(LOONGARCH_CSR_PRCFG1);
set_gcsr_hw_flag(LOONGARCH_CSR_PRCFG2);
set_gcsr_hw_flag(LOONGARCH_CSR_PRCFG3);
set_gcsr_hw_flag(LOONGARCH_CSR_KS0);
set_gcsr_hw_flag(LOONGARCH_CSR_KS1);
set_gcsr_hw_flag(LOONGARCH_CSR_KS2);
set_gcsr_hw_flag(LOONGARCH_CSR_KS3);
set_gcsr_hw_flag(LOONGARCH_CSR_KS4);
set_gcsr_hw_flag(LOONGARCH_CSR_KS5);
set_gcsr_hw_flag(LOONGARCH_CSR_KS6);
set_gcsr_hw_flag(LOONGARCH_CSR_KS7);
set_gcsr_hw_flag(LOONGARCH_CSR_TMID);
set_gcsr_hw_flag(LOONGARCH_CSR_TCFG);
set_gcsr_hw_flag(LOONGARCH_CSR_TVAL);
set_gcsr_hw_flag(LOONGARCH_CSR_TINTCLR);
set_gcsr_hw_flag(LOONGARCH_CSR_CNTC);
set_gcsr_hw_flag(LOONGARCH_CSR_LLBCTL);
set_gcsr_hw_flag(LOONGARCH_CSR_TLBRENTRY);
set_gcsr_hw_flag(LOONGARCH_CSR_TLBRBADV);
set_gcsr_hw_flag(LOONGARCH_CSR_TLBRERA);
set_gcsr_hw_flag(LOONGARCH_CSR_TLBRSAVE);
set_gcsr_hw_flag(LOONGARCH_CSR_TLBRELO0);
set_gcsr_hw_flag(LOONGARCH_CSR_TLBRELO1);
set_gcsr_hw_flag(LOONGARCH_CSR_TLBREHI);
set_gcsr_hw_flag(LOONGARCH_CSR_TLBRPRMD);
set_gcsr_hw_flag(LOONGARCH_CSR_DMWIN0);
set_gcsr_hw_flag(LOONGARCH_CSR_DMWIN1);
set_gcsr_hw_flag(LOONGARCH_CSR_DMWIN2);
set_gcsr_hw_flag(LOONGARCH_CSR_DMWIN3);
set_gcsr_sw_flag(LOONGARCH_CSR_IMPCTL1);
set_gcsr_sw_flag(LOONGARCH_CSR_IMPCTL2);
set_gcsr_sw_flag(LOONGARCH_CSR_MERRCTL);
set_gcsr_sw_flag(LOONGARCH_CSR_MERRINFO1);
set_gcsr_sw_flag(LOONGARCH_CSR_MERRINFO2);
set_gcsr_sw_flag(LOONGARCH_CSR_MERRENTRY);
set_gcsr_sw_flag(LOONGARCH_CSR_MERRERA);
set_gcsr_sw_flag(LOONGARCH_CSR_MERRSAVE);
set_gcsr_sw_flag(LOONGARCH_CSR_CTAG);
set_gcsr_sw_flag(LOONGARCH_CSR_DEBUG);
set_gcsr_sw_flag(LOONGARCH_CSR_DERA);
set_gcsr_sw_flag(LOONGARCH_CSR_DESAVE);
set_gcsr_sw_flag(LOONGARCH_CSR_FWPC);
set_gcsr_sw_flag(LOONGARCH_CSR_FWPS);
set_gcsr_sw_flag(LOONGARCH_CSR_MWPC);
set_gcsr_sw_flag(LOONGARCH_CSR_MWPS);
set_gcsr_sw_flag(LOONGARCH_CSR_DB0ADDR);
set_gcsr_sw_flag(LOONGARCH_CSR_DB0MASK);
set_gcsr_sw_flag(LOONGARCH_CSR_DB0CTRL);
set_gcsr_sw_flag(LOONGARCH_CSR_DB0ASID);
set_gcsr_sw_flag(LOONGARCH_CSR_DB1ADDR);
set_gcsr_sw_flag(LOONGARCH_CSR_DB1MASK);
set_gcsr_sw_flag(LOONGARCH_CSR_DB1CTRL);
set_gcsr_sw_flag(LOONGARCH_CSR_DB1ASID);
set_gcsr_sw_flag(LOONGARCH_CSR_DB2ADDR);
set_gcsr_sw_flag(LOONGARCH_CSR_DB2MASK);
set_gcsr_sw_flag(LOONGARCH_CSR_DB2CTRL);
set_gcsr_sw_flag(LOONGARCH_CSR_DB2ASID);
set_gcsr_sw_flag(LOONGARCH_CSR_DB3ADDR);
set_gcsr_sw_flag(LOONGARCH_CSR_DB3MASK);
set_gcsr_sw_flag(LOONGARCH_CSR_DB3CTRL);
set_gcsr_sw_flag(LOONGARCH_CSR_DB3ASID);
set_gcsr_sw_flag(LOONGARCH_CSR_DB4ADDR);
set_gcsr_sw_flag(LOONGARCH_CSR_DB4MASK);
set_gcsr_sw_flag(LOONGARCH_CSR_DB4CTRL);
set_gcsr_sw_flag(LOONGARCH_CSR_DB4ASID);
set_gcsr_sw_flag(LOONGARCH_CSR_DB5ADDR);
set_gcsr_sw_flag(LOONGARCH_CSR_DB5MASK);
set_gcsr_sw_flag(LOONGARCH_CSR_DB5CTRL);
set_gcsr_sw_flag(LOONGARCH_CSR_DB5ASID);
set_gcsr_sw_flag(LOONGARCH_CSR_DB6ADDR);
set_gcsr_sw_flag(LOONGARCH_CSR_DB6MASK);
set_gcsr_sw_flag(LOONGARCH_CSR_DB6CTRL);
set_gcsr_sw_flag(LOONGARCH_CSR_DB6ASID);
set_gcsr_sw_flag(LOONGARCH_CSR_DB7ADDR);
set_gcsr_sw_flag(LOONGARCH_CSR_DB7MASK);
set_gcsr_sw_flag(LOONGARCH_CSR_DB7CTRL);
set_gcsr_sw_flag(LOONGARCH_CSR_DB7ASID);
set_gcsr_sw_flag(LOONGARCH_CSR_IB0ADDR);
set_gcsr_sw_flag(LOONGARCH_CSR_IB0MASK);
set_gcsr_sw_flag(LOONGARCH_CSR_IB0CTRL);
set_gcsr_sw_flag(LOONGARCH_CSR_IB0ASID);
set_gcsr_sw_flag(LOONGARCH_CSR_IB1ADDR);
set_gcsr_sw_flag(LOONGARCH_CSR_IB1MASK);
set_gcsr_sw_flag(LOONGARCH_CSR_IB1CTRL);
set_gcsr_sw_flag(LOONGARCH_CSR_IB1ASID);
set_gcsr_sw_flag(LOONGARCH_CSR_IB2ADDR);
set_gcsr_sw_flag(LOONGARCH_CSR_IB2MASK);
set_gcsr_sw_flag(LOONGARCH_CSR_IB2CTRL);
set_gcsr_sw_flag(LOONGARCH_CSR_IB2ASID);
set_gcsr_sw_flag(LOONGARCH_CSR_IB3ADDR);
set_gcsr_sw_flag(LOONGARCH_CSR_IB3MASK);
set_gcsr_sw_flag(LOONGARCH_CSR_IB3CTRL);
set_gcsr_sw_flag(LOONGARCH_CSR_IB3ASID);
set_gcsr_sw_flag(LOONGARCH_CSR_IB4ADDR);
set_gcsr_sw_flag(LOONGARCH_CSR_IB4MASK);
set_gcsr_sw_flag(LOONGARCH_CSR_IB4CTRL);
set_gcsr_sw_flag(LOONGARCH_CSR_IB4ASID);
set_gcsr_sw_flag(LOONGARCH_CSR_IB5ADDR);
set_gcsr_sw_flag(LOONGARCH_CSR_IB5MASK);
set_gcsr_sw_flag(LOONGARCH_CSR_IB5CTRL);
set_gcsr_sw_flag(LOONGARCH_CSR_IB5ASID);
set_gcsr_sw_flag(LOONGARCH_CSR_IB6ADDR);
set_gcsr_sw_flag(LOONGARCH_CSR_IB6MASK);
set_gcsr_sw_flag(LOONGARCH_CSR_IB6CTRL);
set_gcsr_sw_flag(LOONGARCH_CSR_IB6ASID);
set_gcsr_sw_flag(LOONGARCH_CSR_IB7ADDR);
set_gcsr_sw_flag(LOONGARCH_CSR_IB7MASK);
set_gcsr_sw_flag(LOONGARCH_CSR_IB7CTRL);
set_gcsr_sw_flag(LOONGARCH_CSR_IB7ASID);
set_gcsr_sw_flag(LOONGARCH_CSR_PERFCTRL0);
set_gcsr_sw_flag(LOONGARCH_CSR_PERFCNTR0);
set_gcsr_sw_flag(LOONGARCH_CSR_PERFCTRL1);
set_gcsr_sw_flag(LOONGARCH_CSR_PERFCNTR1);
set_gcsr_sw_flag(LOONGARCH_CSR_PERFCTRL2);
set_gcsr_sw_flag(LOONGARCH_CSR_PERFCNTR2);
set_gcsr_sw_flag(LOONGARCH_CSR_PERFCTRL3);
set_gcsr_sw_flag(LOONGARCH_CSR_PERFCNTR3);
}
static void kvm_update_vpid(struct kvm_vcpu *vcpu, int cpu)
{
unsigned long vpid;
struct kvm_context *context;
context = per_cpu_ptr(vcpu->kvm->arch.vmcs, cpu);
vpid = context->vpid_cache + 1;
if (!(vpid & vpid_mask)) {
/* finish round of vpid loop */
if (unlikely(!vpid))
vpid = vpid_mask + 1;
++vpid; /* vpid 0 reserved for root */
/* start new vpid cycle */
kvm_flush_tlb_all();
}
context->vpid_cache = vpid;
vcpu->arch.vpid = vpid;
}
void kvm_check_vpid(struct kvm_vcpu *vcpu)
{
int cpu;
bool migrated;
unsigned long ver, old, vpid;
struct kvm_context *context;
cpu = smp_processor_id();
/*
* Are we entering guest context on a different CPU to last time?
* If so, the vCPU's guest TLB state on this CPU may be stale.
*/
context = per_cpu_ptr(vcpu->kvm->arch.vmcs, cpu);
migrated = (vcpu->cpu != cpu);
/*
* Check if our vpid is of an older version
*
* We also discard the stored vpid if we've executed on
* another CPU, as the guest mappings may have changed without
* hypervisor knowledge.
*/
ver = vcpu->arch.vpid & ~vpid_mask;
old = context->vpid_cache & ~vpid_mask;
if (migrated || (ver != old)) {
kvm_update_vpid(vcpu, cpu);
trace_kvm_vpid_change(vcpu, vcpu->arch.vpid);
vcpu->cpu = cpu;
}
/* Restore GSTAT(0x50).vpid */
vpid = (vcpu->arch.vpid & vpid_mask) << CSR_GSTAT_GID_SHIFT;
change_csr_gstat(vpid_mask << CSR_GSTAT_GID_SHIFT, vpid);
}
void kvm_init_vmcs(struct kvm *kvm)
{
kvm->arch.vmcs = vmcs;
}
long kvm_arch_dev_ioctl(struct file *filp,
unsigned int ioctl, unsigned long arg)
{
return -ENOIOCTLCMD;
}
int kvm_arch_hardware_enable(void)
{
unsigned long env, gcfg = 0;
env = read_csr_gcfg();
/* First init gcfg, gstat, gintc, gtlbc. All guest use the same config */
write_csr_gcfg(0);
write_csr_gstat(0);
write_csr_gintc(0);
clear_csr_gtlbc(CSR_GTLBC_USETGID | CSR_GTLBC_TOTI);
/*
* Enable virtualization features granting guest direct control of
* certain features:
* GCI=2: Trap on init or unimplement cache instruction.
* TORU=0: Trap on Root Unimplement.
* CACTRL=1: Root control cache.
* TOP=0: Trap on Previlege.
* TOE=0: Trap on Exception.
* TIT=0: Trap on Timer.
*/
if (env & CSR_GCFG_GCIP_ALL)
gcfg |= CSR_GCFG_GCI_SECURE;
if (env & CSR_GCFG_MATC_ROOT)
gcfg |= CSR_GCFG_MATC_ROOT;
gcfg |= CSR_GCFG_TIT;
write_csr_gcfg(gcfg);
kvm_flush_tlb_all();
/* Enable using TGID */
set_csr_gtlbc(CSR_GTLBC_USETGID);
kvm_debug("GCFG:%lx GSTAT:%lx GINTC:%lx GTLBC:%lx",
read_csr_gcfg(), read_csr_gstat(), read_csr_gintc(), read_csr_gtlbc());
return 0;
}
void kvm_arch_hardware_disable(void)
{
write_csr_gcfg(0);
write_csr_gstat(0);
write_csr_gintc(0);
clear_csr_gtlbc(CSR_GTLBC_USETGID | CSR_GTLBC_TOTI);
/* Flush any remaining guest TLB entries */
kvm_flush_tlb_all();
}
static int kvm_loongarch_env_init(void)
{
int cpu, order;
void *addr;
struct kvm_context *context;
vmcs = alloc_percpu(struct kvm_context);
if (!vmcs) {
pr_err("kvm: failed to allocate percpu kvm_context\n");
return -ENOMEM;
}
kvm_loongarch_ops = kzalloc(sizeof(*kvm_loongarch_ops), GFP_KERNEL);
if (!kvm_loongarch_ops) {
free_percpu(vmcs);
vmcs = NULL;
return -ENOMEM;
}
/*
* PGD register is shared between root kernel and kvm hypervisor.
* So world switch entry should be in DMW area rather than TLB area
* to avoid page fault reenter.
*
* In future if hardware pagetable walking is supported, we won't
* need to copy world switch code to DMW area.
*/
order = get_order(kvm_exception_size + kvm_enter_guest_size);
addr = (void *)__get_free_pages(GFP_KERNEL, order);
if (!addr) {
free_percpu(vmcs);
vmcs = NULL;
kfree(kvm_loongarch_ops);
kvm_loongarch_ops = NULL;
return -ENOMEM;
}
memcpy(addr, kvm_exc_entry, kvm_exception_size);
memcpy(addr + kvm_exception_size, kvm_enter_guest, kvm_enter_guest_size);
flush_icache_range((unsigned long)addr, (unsigned long)addr + kvm_exception_size + kvm_enter_guest_size);
kvm_loongarch_ops->exc_entry = addr;
kvm_loongarch_ops->enter_guest = addr + kvm_exception_size;
kvm_loongarch_ops->page_order = order;
vpid_mask = read_csr_gstat();
vpid_mask = (vpid_mask & CSR_GSTAT_GIDBIT) >> CSR_GSTAT_GIDBIT_SHIFT;
if (vpid_mask)
vpid_mask = GENMASK(vpid_mask - 1, 0);
for_each_possible_cpu(cpu) {
context = per_cpu_ptr(vmcs, cpu);
context->vpid_cache = vpid_mask + 1;
context->last_vcpu = NULL;
}
kvm_init_gcsr_flag();
return 0;
}
static void kvm_loongarch_env_exit(void)
{
unsigned long addr;
if (vmcs)
free_percpu(vmcs);
if (kvm_loongarch_ops) {
if (kvm_loongarch_ops->exc_entry) {
addr = (unsigned long)kvm_loongarch_ops->exc_entry;
free_pages(addr, kvm_loongarch_ops->page_order);
}
kfree(kvm_loongarch_ops);
}
}
static int kvm_loongarch_init(void)
{
int r;
if (!cpu_has_lvz) {
kvm_info("Hardware virtualization not available\n");
return -ENODEV;
}
r = kvm_loongarch_env_init();
if (r)
return r;
return kvm_init(sizeof(struct kvm_vcpu), 0, THIS_MODULE);
}
static void kvm_loongarch_exit(void)
{
kvm_exit();
kvm_loongarch_env_exit();
}
module_init(kvm_loongarch_init);
module_exit(kvm_loongarch_exit);
#ifdef MODULE
static const struct cpu_feature kvm_feature[] = {
{ .feature = cpu_feature(LOONGARCH_LVZ) },
{},
};
MODULE_DEVICE_TABLE(cpu, kvm_feature);
#endif

914
arch/loongarch/kvm/mmu.c Normal file
View File

@ -0,0 +1,914 @@
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (C) 2020-2023 Loongson Technology Corporation Limited
*/
#include <linux/highmem.h>
#include <linux/hugetlb.h>
#include <linux/kvm_host.h>
#include <linux/page-flags.h>
#include <linux/uaccess.h>
#include <asm/mmu_context.h>
#include <asm/pgalloc.h>
#include <asm/tlb.h>
#include <asm/kvm_mmu.h>
static inline void kvm_ptw_prepare(struct kvm *kvm, kvm_ptw_ctx *ctx)
{
ctx->level = kvm->arch.root_level;
/* pte table */
ctx->invalid_ptes = kvm->arch.invalid_ptes;
ctx->pte_shifts = kvm->arch.pte_shifts;
ctx->pgtable_shift = ctx->pte_shifts[ctx->level];
ctx->invalid_entry = ctx->invalid_ptes[ctx->level];
ctx->opaque = kvm;
}
/*
* Mark a range of guest physical address space old (all accesses fault) in the
* VM's GPA page table to allow detection of commonly used pages.
*/
static int kvm_mkold_pte(kvm_pte_t *pte, phys_addr_t addr, kvm_ptw_ctx *ctx)
{
if (kvm_pte_young(*pte)) {
*pte = kvm_pte_mkold(*pte);
return 1;
}
return 0;
}
/*
* Mark a range of guest physical address space clean (writes fault) in the VM's
* GPA page table to allow dirty page tracking.
*/
static int kvm_mkclean_pte(kvm_pte_t *pte, phys_addr_t addr, kvm_ptw_ctx *ctx)
{
gfn_t offset;
kvm_pte_t val;
val = *pte;
/*
* For kvm_arch_mmu_enable_log_dirty_pt_masked with mask, start and end
* may cross hugepage, for first huge page parameter addr is equal to
* start, however for the second huge page addr is base address of
* this huge page, rather than start or end address
*/
if ((ctx->flag & _KVM_HAS_PGMASK) && !kvm_pte_huge(val)) {
offset = (addr >> PAGE_SHIFT) - ctx->gfn;
if (!(BIT(offset) & ctx->mask))
return 0;
}
/*
* Need not split huge page now, just set write-proect pte bit
* Split huge page until next write fault
*/
if (kvm_pte_dirty(val)) {
*pte = kvm_pte_mkclean(val);
return 1;
}
return 0;
}
/*
* Clear pte entry
*/
static int kvm_flush_pte(kvm_pte_t *pte, phys_addr_t addr, kvm_ptw_ctx *ctx)
{
struct kvm *kvm;
kvm = ctx->opaque;
if (ctx->level)
kvm->stat.hugepages--;
else
kvm->stat.pages--;
*pte = ctx->invalid_entry;
return 1;
}
/*
* kvm_pgd_alloc() - Allocate and initialise a KVM GPA page directory.
*
* Allocate a blank KVM GPA page directory (PGD) for representing guest physical
* to host physical page mappings.
*
* Returns: Pointer to new KVM GPA page directory.
* NULL on allocation failure.
*/
kvm_pte_t *kvm_pgd_alloc(void)
{
kvm_pte_t *pgd;
pgd = (kvm_pte_t *)__get_free_pages(GFP_KERNEL, 0);
if (pgd)
pgd_init((void *)pgd);
return pgd;
}
static void _kvm_pte_init(void *addr, unsigned long val)
{
unsigned long *p, *end;
p = (unsigned long *)addr;
end = p + PTRS_PER_PTE;
do {
p[0] = val;
p[1] = val;
p[2] = val;
p[3] = val;
p[4] = val;
p += 8;
p[-3] = val;
p[-2] = val;
p[-1] = val;
} while (p != end);
}
/*
* Caller must hold kvm->mm_lock
*
* Walk the page tables of kvm to find the PTE corresponding to the
* address @addr. If page tables don't exist for @addr, they will be created
* from the MMU cache if @cache is not NULL.
*/
static kvm_pte_t *kvm_populate_gpa(struct kvm *kvm,
struct kvm_mmu_memory_cache *cache,
unsigned long addr, int level)
{
kvm_ptw_ctx ctx;
kvm_pte_t *entry, *child;
kvm_ptw_prepare(kvm, &ctx);
child = kvm->arch.pgd;
while (ctx.level > level) {
entry = kvm_pgtable_offset(&ctx, child, addr);
if (kvm_pte_none(&ctx, entry)) {
if (!cache)
return NULL;
child = kvm_mmu_memory_cache_alloc(cache);
_kvm_pte_init(child, ctx.invalid_ptes[ctx.level - 1]);
kvm_set_pte(entry, __pa(child));
} else if (kvm_pte_huge(*entry)) {
return entry;
} else
child = (kvm_pte_t *)__va(PHYSADDR(*entry));
kvm_ptw_enter(&ctx);
}
entry = kvm_pgtable_offset(&ctx, child, addr);
return entry;
}
/*
* Page walker for VM shadow mmu at last level
* The last level is small pte page or huge pmd page
*/
static int kvm_ptw_leaf(kvm_pte_t *dir, phys_addr_t addr, phys_addr_t end, kvm_ptw_ctx *ctx)
{
int ret;
phys_addr_t next, start, size;
struct list_head *list;
kvm_pte_t *entry, *child;
ret = 0;
start = addr;
child = (kvm_pte_t *)__va(PHYSADDR(*dir));
entry = kvm_pgtable_offset(ctx, child, addr);
do {
next = addr + (0x1UL << ctx->pgtable_shift);
if (!kvm_pte_present(ctx, entry))
continue;
ret |= ctx->ops(entry, addr, ctx);
} while (entry++, addr = next, addr < end);
if (kvm_need_flush(ctx)) {
size = 0x1UL << (ctx->pgtable_shift + PAGE_SHIFT - 3);
if (start + size == end) {
list = (struct list_head *)child;
list_add_tail(list, &ctx->list);
*dir = ctx->invalid_ptes[ctx->level + 1];
}
}
return ret;
}
/*
* Page walker for VM shadow mmu at page table dir level
*/
static int kvm_ptw_dir(kvm_pte_t *dir, phys_addr_t addr, phys_addr_t end, kvm_ptw_ctx *ctx)
{
int ret;
phys_addr_t next, start, size;
struct list_head *list;
kvm_pte_t *entry, *child;
ret = 0;
start = addr;
child = (kvm_pte_t *)__va(PHYSADDR(*dir));
entry = kvm_pgtable_offset(ctx, child, addr);
do {
next = kvm_pgtable_addr_end(ctx, addr, end);
if (!kvm_pte_present(ctx, entry))
continue;
if (kvm_pte_huge(*entry)) {
ret |= ctx->ops(entry, addr, ctx);
continue;
}
kvm_ptw_enter(ctx);
if (ctx->level == 0)
ret |= kvm_ptw_leaf(entry, addr, next, ctx);
else
ret |= kvm_ptw_dir(entry, addr, next, ctx);
kvm_ptw_exit(ctx);
} while (entry++, addr = next, addr < end);
if (kvm_need_flush(ctx)) {
size = 0x1UL << (ctx->pgtable_shift + PAGE_SHIFT - 3);
if (start + size == end) {
list = (struct list_head *)child;
list_add_tail(list, &ctx->list);
*dir = ctx->invalid_ptes[ctx->level + 1];
}
}
return ret;
}
/*
* Page walker for VM shadow mmu at page root table
*/
static int kvm_ptw_top(kvm_pte_t *dir, phys_addr_t addr, phys_addr_t end, kvm_ptw_ctx *ctx)
{
int ret;
phys_addr_t next;
kvm_pte_t *entry;
ret = 0;
entry = kvm_pgtable_offset(ctx, dir, addr);
do {
next = kvm_pgtable_addr_end(ctx, addr, end);
if (!kvm_pte_present(ctx, entry))
continue;
kvm_ptw_enter(ctx);
ret |= kvm_ptw_dir(entry, addr, next, ctx);
kvm_ptw_exit(ctx);
} while (entry++, addr = next, addr < end);
return ret;
}
/*
* kvm_flush_range() - Flush a range of guest physical addresses.
* @kvm: KVM pointer.
* @start_gfn: Guest frame number of first page in GPA range to flush.
* @end_gfn: Guest frame number of last page in GPA range to flush.
* @lock: Whether to hold mmu_lock or not
*
* Flushes a range of GPA mappings from the GPA page tables.
*/
static void kvm_flush_range(struct kvm *kvm, gfn_t start_gfn, gfn_t end_gfn, int lock)
{
int ret;
kvm_ptw_ctx ctx;
struct list_head *pos, *temp;
ctx.ops = kvm_flush_pte;
ctx.flag = _KVM_FLUSH_PGTABLE;
kvm_ptw_prepare(kvm, &ctx);
INIT_LIST_HEAD(&ctx.list);
if (lock) {
spin_lock(&kvm->mmu_lock);
ret = kvm_ptw_top(kvm->arch.pgd, start_gfn << PAGE_SHIFT,
end_gfn << PAGE_SHIFT, &ctx);
spin_unlock(&kvm->mmu_lock);
} else
ret = kvm_ptw_top(kvm->arch.pgd, start_gfn << PAGE_SHIFT,
end_gfn << PAGE_SHIFT, &ctx);
/* Flush vpid for each vCPU individually */
if (ret)
kvm_flush_remote_tlbs(kvm);
/*
* free pte table page after mmu_lock
* the pte table page is linked together with ctx.list
*/
list_for_each_safe(pos, temp, &ctx.list) {
list_del(pos);
free_page((unsigned long)pos);
}
}
/*
* kvm_mkclean_gpa_pt() - Make a range of guest physical addresses clean.
* @kvm: KVM pointer.
* @start_gfn: Guest frame number of first page in GPA range to flush.
* @end_gfn: Guest frame number of last page in GPA range to flush.
*
* Make a range of GPA mappings clean so that guest writes will fault and
* trigger dirty page logging.
*
* The caller must hold the @kvm->mmu_lock spinlock.
*
* Returns: Whether any GPA mappings were modified, which would require
* derived mappings (GVA page tables & TLB enties) to be
* invalidated.
*/
static int kvm_mkclean_gpa_pt(struct kvm *kvm, gfn_t start_gfn, gfn_t end_gfn)
{
kvm_ptw_ctx ctx;
ctx.ops = kvm_mkclean_pte;
ctx.flag = 0;
kvm_ptw_prepare(kvm, &ctx);
return kvm_ptw_top(kvm->arch.pgd, start_gfn << PAGE_SHIFT, end_gfn << PAGE_SHIFT, &ctx);
}
/*
* kvm_arch_mmu_enable_log_dirty_pt_masked() - write protect dirty pages
* @kvm: The KVM pointer
* @slot: The memory slot associated with mask
* @gfn_offset: The gfn offset in memory slot
* @mask: The mask of dirty pages at offset 'gfn_offset' in this memory
* slot to be write protected
*
* Walks bits set in mask write protects the associated pte's. Caller must
* acquire @kvm->mmu_lock.
*/
void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm,
struct kvm_memory_slot *slot, gfn_t gfn_offset, unsigned long mask)
{
kvm_ptw_ctx ctx;
gfn_t base_gfn = slot->base_gfn + gfn_offset;
gfn_t start = base_gfn + __ffs(mask);
gfn_t end = base_gfn + __fls(mask) + 1;
ctx.ops = kvm_mkclean_pte;
ctx.flag = _KVM_HAS_PGMASK;
ctx.mask = mask;
ctx.gfn = base_gfn;
kvm_ptw_prepare(kvm, &ctx);
kvm_ptw_top(kvm->arch.pgd, start << PAGE_SHIFT, end << PAGE_SHIFT, &ctx);
}
void kvm_arch_commit_memory_region(struct kvm *kvm,
struct kvm_memory_slot *old,
const struct kvm_memory_slot *new,
enum kvm_mr_change change)
{
int needs_flush;
/*
* If dirty page logging is enabled, write protect all pages in the slot
* ready for dirty logging.
*
* There is no need to do this in any of the following cases:
* CREATE: No dirty mappings will already exist.
* MOVE/DELETE: The old mappings will already have been cleaned up by
* kvm_arch_flush_shadow_memslot()
*/
if (change == KVM_MR_FLAGS_ONLY &&
(!(old->flags & KVM_MEM_LOG_DIRTY_PAGES) &&
new->flags & KVM_MEM_LOG_DIRTY_PAGES)) {
spin_lock(&kvm->mmu_lock);
/* Write protect GPA page table entries */
needs_flush = kvm_mkclean_gpa_pt(kvm, new->base_gfn,
new->base_gfn + new->npages);
spin_unlock(&kvm->mmu_lock);
if (needs_flush)
kvm_flush_remote_tlbs(kvm);
}
}
void kvm_arch_flush_shadow_all(struct kvm *kvm)
{
kvm_flush_range(kvm, 0, kvm->arch.gpa_size >> PAGE_SHIFT, 0);
}
void kvm_arch_flush_shadow_memslot(struct kvm *kvm, struct kvm_memory_slot *slot)
{
/*
* The slot has been made invalid (ready for moving or deletion), so we
* need to ensure that it can no longer be accessed by any guest vCPUs.
*/
kvm_flush_range(kvm, slot->base_gfn, slot->base_gfn + slot->npages, 1);
}
bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
{
kvm_ptw_ctx ctx;
ctx.flag = 0;
ctx.ops = kvm_flush_pte;
kvm_ptw_prepare(kvm, &ctx);
INIT_LIST_HEAD(&ctx.list);
return kvm_ptw_top(kvm->arch.pgd, range->start << PAGE_SHIFT,
range->end << PAGE_SHIFT, &ctx);
}
bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
{
unsigned long prot_bits;
kvm_pte_t *ptep;
kvm_pfn_t pfn = pte_pfn(range->arg.pte);
gpa_t gpa = range->start << PAGE_SHIFT;
ptep = kvm_populate_gpa(kvm, NULL, gpa, 0);
if (!ptep)
return false;
/* Replacing an absent or old page doesn't need flushes */
if (!kvm_pte_present(NULL, ptep) || !kvm_pte_young(*ptep)) {
kvm_set_pte(ptep, 0);
return false;
}
/* Fill new pte if write protected or page migrated */
prot_bits = _PAGE_PRESENT | __READABLE;
prot_bits |= _CACHE_MASK & pte_val(range->arg.pte);
/*
* Set _PAGE_WRITE or _PAGE_DIRTY iff old and new pte both support
* _PAGE_WRITE for map_page_fast if next page write fault
* _PAGE_DIRTY since gpa has already recorded as dirty page
*/
prot_bits |= __WRITEABLE & *ptep & pte_val(range->arg.pte);
kvm_set_pte(ptep, kvm_pfn_pte(pfn, __pgprot(prot_bits)));
return true;
}
bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
{
kvm_ptw_ctx ctx;
ctx.flag = 0;
ctx.ops = kvm_mkold_pte;
kvm_ptw_prepare(kvm, &ctx);
return kvm_ptw_top(kvm->arch.pgd, range->start << PAGE_SHIFT,
range->end << PAGE_SHIFT, &ctx);
}
bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
{
gpa_t gpa = range->start << PAGE_SHIFT;
kvm_pte_t *ptep = kvm_populate_gpa(kvm, NULL, gpa, 0);
if (ptep && kvm_pte_present(NULL, ptep) && kvm_pte_young(*ptep))
return true;
return false;
}
/*
* kvm_map_page_fast() - Fast path GPA fault handler.
* @vcpu: vCPU pointer.
* @gpa: Guest physical address of fault.
* @write: Whether the fault was due to a write.
*
* Perform fast path GPA fault handling, doing all that can be done without
* calling into KVM. This handles marking old pages young (for idle page
* tracking), and dirtying of clean pages (for dirty page logging).
*
* Returns: 0 on success, in which case we can update derived mappings and
* resume guest execution.
* -EFAULT on failure due to absent GPA mapping or write to
* read-only page, in which case KVM must be consulted.
*/
static int kvm_map_page_fast(struct kvm_vcpu *vcpu, unsigned long gpa, bool write)
{
int ret = 0;
kvm_pfn_t pfn = 0;
kvm_pte_t *ptep, changed, new;
gfn_t gfn = gpa >> PAGE_SHIFT;
struct kvm *kvm = vcpu->kvm;
struct kvm_memory_slot *slot;
spin_lock(&kvm->mmu_lock);
/* Fast path - just check GPA page table for an existing entry */
ptep = kvm_populate_gpa(kvm, NULL, gpa, 0);
if (!ptep || !kvm_pte_present(NULL, ptep)) {
ret = -EFAULT;
goto out;
}
/* Track access to pages marked old */
new = *ptep;
if (!kvm_pte_young(new))
new = kvm_pte_mkyoung(new);
/* call kvm_set_pfn_accessed() after unlock */
if (write && !kvm_pte_dirty(new)) {
if (!kvm_pte_write(new)) {
ret = -EFAULT;
goto out;
}
if (kvm_pte_huge(new)) {
/*
* Do not set write permission when dirty logging is
* enabled for HugePages
*/
slot = gfn_to_memslot(kvm, gfn);
if (kvm_slot_dirty_track_enabled(slot)) {
ret = -EFAULT;
goto out;
}
}
/* Track dirtying of writeable pages */
new = kvm_pte_mkdirty(new);
}
changed = new ^ (*ptep);
if (changed) {
kvm_set_pte(ptep, new);
pfn = kvm_pte_pfn(new);
}
spin_unlock(&kvm->mmu_lock);
/*
* Fixme: pfn may be freed after mmu_lock
* kvm_try_get_pfn(pfn)/kvm_release_pfn pair to prevent this?
*/
if (kvm_pte_young(changed))
kvm_set_pfn_accessed(pfn);
if (kvm_pte_dirty(changed)) {
mark_page_dirty(kvm, gfn);
kvm_set_pfn_dirty(pfn);
}
return ret;
out:
spin_unlock(&kvm->mmu_lock);
return ret;
}
static bool fault_supports_huge_mapping(struct kvm_memory_slot *memslot,
unsigned long hva, unsigned long map_size, bool write)
{
size_t size;
gpa_t gpa_start;
hva_t uaddr_start, uaddr_end;
/* Disable dirty logging on HugePages */
if (kvm_slot_dirty_track_enabled(memslot) && write)
return false;
size = memslot->npages * PAGE_SIZE;
gpa_start = memslot->base_gfn << PAGE_SHIFT;
uaddr_start = memslot->userspace_addr;
uaddr_end = uaddr_start + size;
/*
* Pages belonging to memslots that don't have the same alignment
* within a PMD for userspace and GPA cannot be mapped with stage-2
* PMD entries, because we'll end up mapping the wrong pages.
*
* Consider a layout like the following:
*
* memslot->userspace_addr:
* +-----+--------------------+--------------------+---+
* |abcde|fgh Stage-1 block | Stage-1 block tv|xyz|
* +-----+--------------------+--------------------+---+
*
* memslot->base_gfn << PAGE_SIZE:
* +---+--------------------+--------------------+-----+
* |abc|def Stage-2 block | Stage-2 block |tvxyz|
* +---+--------------------+--------------------+-----+
*
* If we create those stage-2 blocks, we'll end up with this incorrect
* mapping:
* d -> f
* e -> g
* f -> h
*/
if ((gpa_start & (map_size - 1)) != (uaddr_start & (map_size - 1)))
return false;
/*
* Next, let's make sure we're not trying to map anything not covered
* by the memslot. This means we have to prohibit block size mappings
* for the beginning and end of a non-block aligned and non-block sized
* memory slot (illustrated by the head and tail parts of the
* userspace view above containing pages 'abcde' and 'xyz',
* respectively).
*
* Note that it doesn't matter if we do the check using the
* userspace_addr or the base_gfn, as both are equally aligned (per
* the check above) and equally sized.
*/
return (hva & ~(map_size - 1)) >= uaddr_start &&
(hva & ~(map_size - 1)) + map_size <= uaddr_end;
}
/*
* Lookup the mapping level for @gfn in the current mm.
*
* WARNING! Use of host_pfn_mapping_level() requires the caller and the end
* consumer to be tied into KVM's handlers for MMU notifier events!
*
* There are several ways to safely use this helper:
*
* - Check mmu_invalidate_retry_hva() after grabbing the mapping level, before
* consuming it. In this case, mmu_lock doesn't need to be held during the
* lookup, but it does need to be held while checking the MMU notifier.
*
* - Hold mmu_lock AND ensure there is no in-progress MMU notifier invalidation
* event for the hva. This can be done by explicit checking the MMU notifier
* or by ensuring that KVM already has a valid mapping that covers the hva.
*
* - Do not use the result to install new mappings, e.g. use the host mapping
* level only to decide whether or not to zap an entry. In this case, it's
* not required to hold mmu_lock (though it's highly likely the caller will
* want to hold mmu_lock anyways, e.g. to modify SPTEs).
*
* Note! The lookup can still race with modifications to host page tables, but
* the above "rules" ensure KVM will not _consume_ the result of the walk if a
* race with the primary MMU occurs.
*/
static int host_pfn_mapping_level(struct kvm *kvm, gfn_t gfn,
const struct kvm_memory_slot *slot)
{
int level = 0;
unsigned long hva;
unsigned long flags;
pgd_t pgd;
p4d_t p4d;
pud_t pud;
pmd_t pmd;
/*
* Note, using the already-retrieved memslot and __gfn_to_hva_memslot()
* is not solely for performance, it's also necessary to avoid the
* "writable" check in __gfn_to_hva_many(), which will always fail on
* read-only memslots due to gfn_to_hva() assuming writes. Earlier
* page fault steps have already verified the guest isn't writing a
* read-only memslot.
*/
hva = __gfn_to_hva_memslot(slot, gfn);
/*
* Disable IRQs to prevent concurrent tear down of host page tables,
* e.g. if the primary MMU promotes a P*D to a huge page and then frees
* the original page table.
*/
local_irq_save(flags);
/*
* Read each entry once. As above, a non-leaf entry can be promoted to
* a huge page _during_ this walk. Re-reading the entry could send the
* walk into the weeks, e.g. p*d_large() returns false (sees the old
* value) and then p*d_offset() walks into the target huge page instead
* of the old page table (sees the new value).
*/
pgd = READ_ONCE(*pgd_offset(kvm->mm, hva));
if (pgd_none(pgd))
goto out;
p4d = READ_ONCE(*p4d_offset(&pgd, hva));
if (p4d_none(p4d) || !p4d_present(p4d))
goto out;
pud = READ_ONCE(*pud_offset(&p4d, hva));
if (pud_none(pud) || !pud_present(pud))
goto out;
pmd = READ_ONCE(*pmd_offset(&pud, hva));
if (pmd_none(pmd) || !pmd_present(pmd))
goto out;
if (kvm_pte_huge(pmd_val(pmd)))
level = 1;
out:
local_irq_restore(flags);
return level;
}
/*
* Split huge page
*/
static kvm_pte_t *kvm_split_huge(struct kvm_vcpu *vcpu, kvm_pte_t *ptep, gfn_t gfn)
{
int i;
kvm_pte_t val, *child;
struct kvm *kvm = vcpu->kvm;
struct kvm_mmu_memory_cache *memcache;
memcache = &vcpu->arch.mmu_page_cache;
child = kvm_mmu_memory_cache_alloc(memcache);
val = kvm_pte_mksmall(*ptep);
for (i = 0; i < PTRS_PER_PTE; i++) {
kvm_set_pte(child + i, val);
val += PAGE_SIZE;
}
/* The later kvm_flush_tlb_gpa() will flush hugepage tlb */
kvm_set_pte(ptep, __pa(child));
kvm->stat.hugepages--;
kvm->stat.pages += PTRS_PER_PTE;
return child + (gfn & (PTRS_PER_PTE - 1));
}
/*
* kvm_map_page() - Map a guest physical page.
* @vcpu: vCPU pointer.
* @gpa: Guest physical address of fault.
* @write: Whether the fault was due to a write.
*
* Handle GPA faults by creating a new GPA mapping (or updating an existing
* one).
*
* This takes care of marking pages young or dirty (idle/dirty page tracking),
* asking KVM for the corresponding PFN, and creating a mapping in the GPA page
* tables. Derived mappings (GVA page tables and TLBs) must be handled by the
* caller.
*
* Returns: 0 on success
* -EFAULT if there is no memory region at @gpa or a write was
* attempted to a read-only memory region. This is usually handled
* as an MMIO access.
*/
static int kvm_map_page(struct kvm_vcpu *vcpu, unsigned long gpa, bool write)
{
bool writeable;
int srcu_idx, err, retry_no = 0, level;
unsigned long hva, mmu_seq, prot_bits;
kvm_pfn_t pfn;
kvm_pte_t *ptep, new_pte;
gfn_t gfn = gpa >> PAGE_SHIFT;
struct kvm *kvm = vcpu->kvm;
struct kvm_memory_slot *memslot;
struct kvm_mmu_memory_cache *memcache = &vcpu->arch.mmu_page_cache;
/* Try the fast path to handle old / clean pages */
srcu_idx = srcu_read_lock(&kvm->srcu);
err = kvm_map_page_fast(vcpu, gpa, write);
if (!err)
goto out;
memslot = gfn_to_memslot(kvm, gfn);
hva = gfn_to_hva_memslot_prot(memslot, gfn, &writeable);
if (kvm_is_error_hva(hva) || (write && !writeable)) {
err = -EFAULT;
goto out;
}
/* We need a minimum of cached pages ready for page table creation */
err = kvm_mmu_topup_memory_cache(memcache, KVM_MMU_CACHE_MIN_PAGES);
if (err)
goto out;
retry:
/*
* Used to check for invalidations in progress, of the pfn that is
* returned by pfn_to_pfn_prot below.
*/
mmu_seq = kvm->mmu_invalidate_seq;
/*
* Ensure the read of mmu_invalidate_seq isn't reordered with PTE reads in
* gfn_to_pfn_prot() (which calls get_user_pages()), so that we don't
* risk the page we get a reference to getting unmapped before we have a
* chance to grab the mmu_lock without mmu_invalidate_retry() noticing.
*
* This smp_rmb() pairs with the effective smp_wmb() of the combination
* of the pte_unmap_unlock() after the PTE is zapped, and the
* spin_lock() in kvm_mmu_invalidate_invalidate_<page|range_end>() before
* mmu_invalidate_seq is incremented.
*/
smp_rmb();
/* Slow path - ask KVM core whether we can access this GPA */
pfn = gfn_to_pfn_prot(kvm, gfn, write, &writeable);
if (is_error_noslot_pfn(pfn)) {
err = -EFAULT;
goto out;
}
/* Check if an invalidation has taken place since we got pfn */
spin_lock(&kvm->mmu_lock);
if (mmu_invalidate_retry_hva(kvm, mmu_seq, hva)) {
/*
* This can happen when mappings are changed asynchronously, but
* also synchronously if a COW is triggered by
* gfn_to_pfn_prot().
*/
spin_unlock(&kvm->mmu_lock);
kvm_release_pfn_clean(pfn);
if (retry_no > 100) {
retry_no = 0;
schedule();
}
retry_no++;
goto retry;
}
/*
* For emulated devices such virtio device, actual cache attribute is
* determined by physical machine.
* For pass through physical device, it should be uncachable
*/
prot_bits = _PAGE_PRESENT | __READABLE;
if (pfn_valid(pfn))
prot_bits |= _CACHE_CC;
else
prot_bits |= _CACHE_SUC;
if (writeable) {
prot_bits |= _PAGE_WRITE;
if (write)
prot_bits |= __WRITEABLE;
}
/* Disable dirty logging on HugePages */
level = 0;
if (!fault_supports_huge_mapping(memslot, hva, PMD_SIZE, write)) {
level = 0;
} else {
level = host_pfn_mapping_level(kvm, gfn, memslot);
if (level == 1) {
gfn = gfn & ~(PTRS_PER_PTE - 1);
pfn = pfn & ~(PTRS_PER_PTE - 1);
}
}
/* Ensure page tables are allocated */
ptep = kvm_populate_gpa(kvm, memcache, gpa, level);
new_pte = kvm_pfn_pte(pfn, __pgprot(prot_bits));
if (level == 1) {
new_pte = kvm_pte_mkhuge(new_pte);
/*
* previous pmd entry is invalid_pte_table
* there is invalid tlb with small page
* need flush these invalid tlbs for current vcpu
*/
kvm_make_request(KVM_REQ_TLB_FLUSH, vcpu);
++kvm->stat.hugepages;
} else if (kvm_pte_huge(*ptep) && write)
ptep = kvm_split_huge(vcpu, ptep, gfn);
else
++kvm->stat.pages;
kvm_set_pte(ptep, new_pte);
spin_unlock(&kvm->mmu_lock);
if (prot_bits & _PAGE_DIRTY) {
mark_page_dirty_in_slot(kvm, memslot, gfn);
kvm_set_pfn_dirty(pfn);
}
kvm_set_pfn_accessed(pfn);
kvm_release_pfn_clean(pfn);
out:
srcu_read_unlock(&kvm->srcu, srcu_idx);
return err;
}
int kvm_handle_mm_fault(struct kvm_vcpu *vcpu, unsigned long gpa, bool write)
{
int ret;
ret = kvm_map_page(vcpu, gpa, write);
if (ret)
return ret;
/* Invalidate this entry in the TLB */
kvm_flush_tlb_gpa(vcpu, gpa);
return 0;
}
void kvm_arch_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot)
{
}
int kvm_arch_prepare_memory_region(struct kvm *kvm, const struct kvm_memory_slot *old,
struct kvm_memory_slot *new, enum kvm_mr_change change)
{
return 0;
}
void kvm_arch_flush_remote_tlbs_memslot(struct kvm *kvm,
const struct kvm_memory_slot *memslot)
{
kvm_flush_remote_tlbs(kvm);
}

250
arch/loongarch/kvm/switch.S Normal file
View File

@ -0,0 +1,250 @@
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Copyright (C) 2020-2023 Loongson Technology Corporation Limited
*/
#include <linux/linkage.h>
#include <asm/asm.h>
#include <asm/asmmacro.h>
#include <asm/loongarch.h>
#include <asm/regdef.h>
#include <asm/stackframe.h>
#define HGPR_OFFSET(x) (PT_R0 + 8*x)
#define GGPR_OFFSET(x) (KVM_ARCH_GGPR + 8*x)
.macro kvm_save_host_gpr base
.irp n,1,2,3,22,23,24,25,26,27,28,29,30,31
st.d $r\n, \base, HGPR_OFFSET(\n)
.endr
.endm
.macro kvm_restore_host_gpr base
.irp n,1,2,3,22,23,24,25,26,27,28,29,30,31
ld.d $r\n, \base, HGPR_OFFSET(\n)
.endr
.endm
/*
* Save and restore all GPRs except base register,
* and default value of base register is a2.
*/
.macro kvm_save_guest_gprs base
.irp n,1,2,3,4,5,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31
st.d $r\n, \base, GGPR_OFFSET(\n)
.endr
.endm
.macro kvm_restore_guest_gprs base
.irp n,1,2,3,4,5,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31
ld.d $r\n, \base, GGPR_OFFSET(\n)
.endr
.endm
/*
* Prepare switch to guest, save host regs and restore guest regs.
* a2: kvm_vcpu_arch, don't touch it until 'ertn'
* t0, t1: temp register
*/
.macro kvm_switch_to_guest
/* Set host ECFG.VS=0, all exceptions share one exception entry */
csrrd t0, LOONGARCH_CSR_ECFG
bstrins.w t0, zero, CSR_ECFG_VS_SHIFT_END, CSR_ECFG_VS_SHIFT
csrwr t0, LOONGARCH_CSR_ECFG
/* Load up the new EENTRY */
ld.d t0, a2, KVM_ARCH_GEENTRY
csrwr t0, LOONGARCH_CSR_EENTRY
/* Set Guest ERA */
ld.d t0, a2, KVM_ARCH_GPC
csrwr t0, LOONGARCH_CSR_ERA
/* Save host PGDL */
csrrd t0, LOONGARCH_CSR_PGDL
st.d t0, a2, KVM_ARCH_HPGD
/* Switch to kvm */
ld.d t1, a2, KVM_VCPU_KVM - KVM_VCPU_ARCH
/* Load guest PGDL */
li.w t0, KVM_GPGD
ldx.d t0, t1, t0
csrwr t0, LOONGARCH_CSR_PGDL
/* Mix GID and RID */
csrrd t1, LOONGARCH_CSR_GSTAT
bstrpick.w t1, t1, CSR_GSTAT_GID_SHIFT_END, CSR_GSTAT_GID_SHIFT
csrrd t0, LOONGARCH_CSR_GTLBC
bstrins.w t0, t1, CSR_GTLBC_TGID_SHIFT_END, CSR_GTLBC_TGID_SHIFT
csrwr t0, LOONGARCH_CSR_GTLBC
/*
* Enable intr in root mode with future ertn so that host interrupt
* can be responsed during VM runs
* Guest CRMD comes from separate GCSR_CRMD register
*/
ori t0, zero, CSR_PRMD_PIE
csrxchg t0, t0, LOONGARCH_CSR_PRMD
/* Set PVM bit to setup ertn to guest context */
ori t0, zero, CSR_GSTAT_PVM
csrxchg t0, t0, LOONGARCH_CSR_GSTAT
/* Load Guest GPRs */
kvm_restore_guest_gprs a2
/* Load KVM_ARCH register */
ld.d a2, a2, (KVM_ARCH_GGPR + 8 * REG_A2)
ertn /* Switch to guest: GSTAT.PGM = 1, ERRCTL.ISERR = 0, TLBRPRMD.ISTLBR = 0 */
.endm
/*
* Exception entry for general exception from guest mode
* - IRQ is disabled
* - kernel privilege in root mode
* - page mode keep unchanged from previous PRMD in root mode
* - Fixme: tlb exception cannot happen since registers relative with TLB
* - is still in guest mode, such as pgd table/vmid registers etc,
* - will fix with hw page walk enabled in future
* load kvm_vcpu from reserved CSR KVM_VCPU_KS, and save a2 to KVM_TEMP_KS
*/
.text
.cfi_sections .debug_frame
SYM_CODE_START(kvm_exc_entry)
csrwr a2, KVM_TEMP_KS
csrrd a2, KVM_VCPU_KS
addi.d a2, a2, KVM_VCPU_ARCH
/* After save GPRs, free to use any GPR */
kvm_save_guest_gprs a2
/* Save guest A2 */
csrrd t0, KVM_TEMP_KS
st.d t0, a2, (KVM_ARCH_GGPR + 8 * REG_A2)
/* A2 is kvm_vcpu_arch, A1 is free to use */
csrrd s1, KVM_VCPU_KS
ld.d s0, s1, KVM_VCPU_RUN
csrrd t0, LOONGARCH_CSR_ESTAT
st.d t0, a2, KVM_ARCH_HESTAT
csrrd t0, LOONGARCH_CSR_ERA
st.d t0, a2, KVM_ARCH_GPC
csrrd t0, LOONGARCH_CSR_BADV
st.d t0, a2, KVM_ARCH_HBADV
csrrd t0, LOONGARCH_CSR_BADI
st.d t0, a2, KVM_ARCH_HBADI
/* Restore host ECFG.VS */
csrrd t0, LOONGARCH_CSR_ECFG
ld.d t1, a2, KVM_ARCH_HECFG
or t0, t0, t1
csrwr t0, LOONGARCH_CSR_ECFG
/* Restore host EENTRY */
ld.d t0, a2, KVM_ARCH_HEENTRY
csrwr t0, LOONGARCH_CSR_EENTRY
/* Restore host pgd table */
ld.d t0, a2, KVM_ARCH_HPGD
csrwr t0, LOONGARCH_CSR_PGDL
/*
* Disable PGM bit to enter root mode by default with next ertn
*/
ori t0, zero, CSR_GSTAT_PVM
csrxchg zero, t0, LOONGARCH_CSR_GSTAT
/*
* Clear GTLBC.TGID field
* 0: for root tlb update in future tlb instr
* others: for guest tlb update like gpa to hpa in future tlb instr
*/
csrrd t0, LOONGARCH_CSR_GTLBC
bstrins.w t0, zero, CSR_GTLBC_TGID_SHIFT_END, CSR_GTLBC_TGID_SHIFT
csrwr t0, LOONGARCH_CSR_GTLBC
ld.d tp, a2, KVM_ARCH_HTP
ld.d sp, a2, KVM_ARCH_HSP
/* restore per cpu register */
ld.d u0, a2, KVM_ARCH_HPERCPU
addi.d sp, sp, -PT_SIZE
/* Prepare handle exception */
or a0, s0, zero
or a1, s1, zero
ld.d t8, a2, KVM_ARCH_HANDLE_EXIT
jirl ra, t8, 0
or a2, s1, zero
addi.d a2, a2, KVM_VCPU_ARCH
/* Resume host when ret <= 0 */
blez a0, ret_to_host
/*
* Return to guest
* Save per cpu register again, maybe switched to another cpu
*/
st.d u0, a2, KVM_ARCH_HPERCPU
/* Save kvm_vcpu to kscratch */
csrwr s1, KVM_VCPU_KS
kvm_switch_to_guest
ret_to_host:
ld.d a2, a2, KVM_ARCH_HSP
addi.d a2, a2, -PT_SIZE
kvm_restore_host_gpr a2
jr ra
SYM_INNER_LABEL(kvm_exc_entry_end, SYM_L_LOCAL)
SYM_CODE_END(kvm_exc_entry)
/*
* int kvm_enter_guest(struct kvm_run *run, struct kvm_vcpu *vcpu)
*
* @register_param:
* a0: kvm_run* run
* a1: kvm_vcpu* vcpu
*/
SYM_FUNC_START(kvm_enter_guest)
/* Allocate space in stack bottom */
addi.d a2, sp, -PT_SIZE
/* Save host GPRs */
kvm_save_host_gpr a2
/* Save host CRMD, PRMD to stack */
csrrd a3, LOONGARCH_CSR_CRMD
st.d a3, a2, PT_CRMD
csrrd a3, LOONGARCH_CSR_PRMD
st.d a3, a2, PT_PRMD
addi.d a2, a1, KVM_VCPU_ARCH
st.d sp, a2, KVM_ARCH_HSP
st.d tp, a2, KVM_ARCH_HTP
/* Save per cpu register */
st.d u0, a2, KVM_ARCH_HPERCPU
/* Save kvm_vcpu to kscratch */
csrwr a1, KVM_VCPU_KS
kvm_switch_to_guest
SYM_INNER_LABEL(kvm_enter_guest_end, SYM_L_LOCAL)
SYM_FUNC_END(kvm_enter_guest)
SYM_FUNC_START(kvm_save_fpu)
fpu_save_csr a0 t1
fpu_save_double a0 t1
fpu_save_cc a0 t1 t2
jr ra
SYM_FUNC_END(kvm_save_fpu)
SYM_FUNC_START(kvm_restore_fpu)
fpu_restore_double a0 t1
fpu_restore_csr a0 t1 t2
fpu_restore_cc a0 t1 t2
jr ra
SYM_FUNC_END(kvm_restore_fpu)
.section ".rodata"
SYM_DATA(kvm_exception_size, .quad kvm_exc_entry_end - kvm_exc_entry)
SYM_DATA(kvm_enter_guest_size, .quad kvm_enter_guest_end - kvm_enter_guest)

197
arch/loongarch/kvm/timer.c Normal file
View File

@ -0,0 +1,197 @@
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (C) 2020-2023 Loongson Technology Corporation Limited
*/
#include <linux/kvm_host.h>
#include <asm/kvm_csr.h>
#include <asm/kvm_vcpu.h>
/*
* ktime_to_tick() - Scale ktime_t to timer tick value.
*/
static inline u64 ktime_to_tick(struct kvm_vcpu *vcpu, ktime_t now)
{
u64 delta;
delta = ktime_to_ns(now);
return div_u64(delta * vcpu->arch.timer_mhz, MNSEC_PER_SEC);
}
static inline u64 tick_to_ns(struct kvm_vcpu *vcpu, u64 tick)
{
return div_u64(tick * MNSEC_PER_SEC, vcpu->arch.timer_mhz);
}
/*
* Push timer forward on timeout.
* Handle an hrtimer event by push the hrtimer forward a period.
*/
static enum hrtimer_restart kvm_count_timeout(struct kvm_vcpu *vcpu)
{
unsigned long cfg, period;
/* Add periodic tick to current expire time */
cfg = kvm_read_sw_gcsr(vcpu->arch.csr, LOONGARCH_CSR_TCFG);
if (cfg & CSR_TCFG_PERIOD) {
period = tick_to_ns(vcpu, cfg & CSR_TCFG_VAL);
hrtimer_add_expires_ns(&vcpu->arch.swtimer, period);
return HRTIMER_RESTART;
} else
return HRTIMER_NORESTART;
}
/* Low level hrtimer wake routine */
enum hrtimer_restart kvm_swtimer_wakeup(struct hrtimer *timer)
{
struct kvm_vcpu *vcpu;
vcpu = container_of(timer, struct kvm_vcpu, arch.swtimer);
kvm_queue_irq(vcpu, INT_TI);
rcuwait_wake_up(&vcpu->wait);
return kvm_count_timeout(vcpu);
}
/*
* Initialise the timer to the specified frequency, zero it
*/
void kvm_init_timer(struct kvm_vcpu *vcpu, unsigned long timer_hz)
{
vcpu->arch.timer_mhz = timer_hz >> 20;
/* Starting at 0 */
kvm_write_sw_gcsr(vcpu->arch.csr, LOONGARCH_CSR_TVAL, 0);
}
/*
* Restore hard timer state and enable guest to access timer registers
* without trap, should be called with irq disabled
*/
void kvm_acquire_timer(struct kvm_vcpu *vcpu)
{
unsigned long cfg;
cfg = read_csr_gcfg();
if (!(cfg & CSR_GCFG_TIT))
return;
/* Enable guest access to hard timer */
write_csr_gcfg(cfg & ~CSR_GCFG_TIT);
/*
* Freeze the soft-timer and sync the guest stable timer with it. We do
* this with interrupts disabled to avoid latency.
*/
hrtimer_cancel(&vcpu->arch.swtimer);
}
/*
* Restore soft timer state from saved context.
*/
void kvm_restore_timer(struct kvm_vcpu *vcpu)
{
unsigned long cfg, delta, period;
ktime_t expire, now;
struct loongarch_csrs *csr = vcpu->arch.csr;
/*
* Set guest stable timer cfg csr
*/
cfg = kvm_read_sw_gcsr(csr, LOONGARCH_CSR_TCFG);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_ESTAT);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_TCFG);
if (!(cfg & CSR_TCFG_EN)) {
/* Guest timer is disabled, just restore timer registers */
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_TVAL);
return;
}
/*
* Set remainder tick value if not expired
*/
now = ktime_get();
expire = vcpu->arch.expire;
if (ktime_before(now, expire))
delta = ktime_to_tick(vcpu, ktime_sub(expire, now));
else {
if (cfg & CSR_TCFG_PERIOD) {
period = cfg & CSR_TCFG_VAL;
delta = ktime_to_tick(vcpu, ktime_sub(now, expire));
delta = period - (delta % period);
} else
delta = 0;
/*
* Inject timer here though sw timer should inject timer
* interrupt async already, since sw timer may be cancelled
* during injecting intr async in function kvm_acquire_timer
*/
kvm_queue_irq(vcpu, INT_TI);
}
write_gcsr_timertick(delta);
}
/*
* Save guest timer state and switch to software emulation of guest
* timer. The hard timer must already be in use, so preemption should be
* disabled.
*/
static void _kvm_save_timer(struct kvm_vcpu *vcpu)
{
unsigned long ticks, delta;
ktime_t expire;
struct loongarch_csrs *csr = vcpu->arch.csr;
ticks = kvm_read_sw_gcsr(csr, LOONGARCH_CSR_TVAL);
delta = tick_to_ns(vcpu, ticks);
expire = ktime_add_ns(ktime_get(), delta);
vcpu->arch.expire = expire;
if (ticks) {
/*
* Update hrtimer to use new timeout
* HRTIMER_MODE_PINNED is suggested since vcpu may run in
* the same physical cpu in next time
*/
hrtimer_cancel(&vcpu->arch.swtimer);
hrtimer_start(&vcpu->arch.swtimer, expire, HRTIMER_MODE_ABS_PINNED);
} else
/*
* Inject timer interrupt so that hall polling can dectect and exit
*/
kvm_queue_irq(vcpu, INT_TI);
}
/*
* Save guest timer state and switch to soft guest timer if hard timer was in
* use.
*/
void kvm_save_timer(struct kvm_vcpu *vcpu)
{
unsigned long cfg;
struct loongarch_csrs *csr = vcpu->arch.csr;
preempt_disable();
cfg = read_csr_gcfg();
if (!(cfg & CSR_GCFG_TIT)) {
/* Disable guest use of hard timer */
write_csr_gcfg(cfg | CSR_GCFG_TIT);
/* Save hard timer state */
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_TCFG);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_TVAL);
if (kvm_read_sw_gcsr(csr, LOONGARCH_CSR_TCFG) & CSR_TCFG_EN)
_kvm_save_timer(vcpu);
}
/* Save timer-related state to vCPU context */
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_ESTAT);
preempt_enable();
}
void kvm_reset_timer(struct kvm_vcpu *vcpu)
{
write_gcsr_timercfg(0);
kvm_write_sw_gcsr(vcpu->arch.csr, LOONGARCH_CSR_TCFG, 0);
hrtimer_cancel(&vcpu->arch.swtimer);
}

32
arch/loongarch/kvm/tlb.c Normal file
View File

@ -0,0 +1,32 @@
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (C) 2020-2023 Loongson Technology Corporation Limited
*/
#include <linux/kvm_host.h>
#include <asm/tlb.h>
#include <asm/kvm_csr.h>
/*
* kvm_flush_tlb_all() - Flush all root TLB entries for guests.
*
* Invalidate all entries including GVA-->GPA and GPA-->HPA mappings.
*/
void kvm_flush_tlb_all(void)
{
unsigned long flags;
local_irq_save(flags);
invtlb_all(INVTLB_ALLGID, 0, 0);
local_irq_restore(flags);
}
void kvm_flush_tlb_gpa(struct kvm_vcpu *vcpu, unsigned long gpa)
{
unsigned long flags;
local_irq_save(flags);
gpa &= (PAGE_MASK << 1);
invtlb(INVTLB_GID_ADDR, read_csr_gstat() & CSR_GSTAT_GID, gpa);
local_irq_restore(flags);
}

162
arch/loongarch/kvm/trace.h Normal file
View File

@ -0,0 +1,162 @@
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Copyright (C) 2020-2023 Loongson Technology Corporation Limited
*/
#if !defined(_TRACE_KVM_H) || defined(TRACE_HEADER_MULTI_READ)
#define _TRACE_KVM_H
#include <linux/tracepoint.h>
#include <asm/kvm_csr.h>
#undef TRACE_SYSTEM
#define TRACE_SYSTEM kvm
/*
* Tracepoints for VM enters
*/
DECLARE_EVENT_CLASS(kvm_transition,
TP_PROTO(struct kvm_vcpu *vcpu),
TP_ARGS(vcpu),
TP_STRUCT__entry(
__field(unsigned long, pc)
),
TP_fast_assign(
__entry->pc = vcpu->arch.pc;
),
TP_printk("PC: 0x%08lx", __entry->pc)
);
DEFINE_EVENT(kvm_transition, kvm_enter,
TP_PROTO(struct kvm_vcpu *vcpu),
TP_ARGS(vcpu));
DEFINE_EVENT(kvm_transition, kvm_reenter,
TP_PROTO(struct kvm_vcpu *vcpu),
TP_ARGS(vcpu));
DEFINE_EVENT(kvm_transition, kvm_out,
TP_PROTO(struct kvm_vcpu *vcpu),
TP_ARGS(vcpu));
/* Further exit reasons */
#define KVM_TRACE_EXIT_IDLE 64
#define KVM_TRACE_EXIT_CACHE 65
/* Tracepoints for VM exits */
#define kvm_trace_symbol_exit_types \
{ KVM_TRACE_EXIT_IDLE, "IDLE" }, \
{ KVM_TRACE_EXIT_CACHE, "CACHE" }
DECLARE_EVENT_CLASS(kvm_exit,
TP_PROTO(struct kvm_vcpu *vcpu, unsigned int reason),
TP_ARGS(vcpu, reason),
TP_STRUCT__entry(
__field(unsigned long, pc)
__field(unsigned int, reason)
),
TP_fast_assign(
__entry->pc = vcpu->arch.pc;
__entry->reason = reason;
),
TP_printk("[%s]PC: 0x%08lx",
__print_symbolic(__entry->reason,
kvm_trace_symbol_exit_types),
__entry->pc)
);
DEFINE_EVENT(kvm_exit, kvm_exit_idle,
TP_PROTO(struct kvm_vcpu *vcpu, unsigned int reason),
TP_ARGS(vcpu, reason));
DEFINE_EVENT(kvm_exit, kvm_exit_cache,
TP_PROTO(struct kvm_vcpu *vcpu, unsigned int reason),
TP_ARGS(vcpu, reason));
DEFINE_EVENT(kvm_exit, kvm_exit,
TP_PROTO(struct kvm_vcpu *vcpu, unsigned int reason),
TP_ARGS(vcpu, reason));
TRACE_EVENT(kvm_exit_gspr,
TP_PROTO(struct kvm_vcpu *vcpu, unsigned int inst_word),
TP_ARGS(vcpu, inst_word),
TP_STRUCT__entry(
__field(unsigned int, inst_word)
),
TP_fast_assign(
__entry->inst_word = inst_word;
),
TP_printk("Inst word: 0x%08x", __entry->inst_word)
);
#define KVM_TRACE_AUX_SAVE 0
#define KVM_TRACE_AUX_RESTORE 1
#define KVM_TRACE_AUX_ENABLE 2
#define KVM_TRACE_AUX_DISABLE 3
#define KVM_TRACE_AUX_DISCARD 4
#define KVM_TRACE_AUX_FPU 1
#define kvm_trace_symbol_aux_op \
{ KVM_TRACE_AUX_SAVE, "save" }, \
{ KVM_TRACE_AUX_RESTORE, "restore" }, \
{ KVM_TRACE_AUX_ENABLE, "enable" }, \
{ KVM_TRACE_AUX_DISABLE, "disable" }, \
{ KVM_TRACE_AUX_DISCARD, "discard" }
#define kvm_trace_symbol_aux_state \
{ KVM_TRACE_AUX_FPU, "FPU" }
TRACE_EVENT(kvm_aux,
TP_PROTO(struct kvm_vcpu *vcpu, unsigned int op,
unsigned int state),
TP_ARGS(vcpu, op, state),
TP_STRUCT__entry(
__field(unsigned long, pc)
__field(u8, op)
__field(u8, state)
),
TP_fast_assign(
__entry->pc = vcpu->arch.pc;
__entry->op = op;
__entry->state = state;
),
TP_printk("%s %s PC: 0x%08lx",
__print_symbolic(__entry->op,
kvm_trace_symbol_aux_op),
__print_symbolic(__entry->state,
kvm_trace_symbol_aux_state),
__entry->pc)
);
TRACE_EVENT(kvm_vpid_change,
TP_PROTO(struct kvm_vcpu *vcpu, unsigned long vpid),
TP_ARGS(vcpu, vpid),
TP_STRUCT__entry(
__field(unsigned long, vpid)
),
TP_fast_assign(
__entry->vpid = vpid;
),
TP_printk("VPID: 0x%08lx", __entry->vpid)
);
#endif /* _TRACE_KVM_H */
#undef TRACE_INCLUDE_PATH
#define TRACE_INCLUDE_PATH ../../arch/loongarch/kvm
#undef TRACE_INCLUDE_FILE
#define TRACE_INCLUDE_FILE trace
/* This part must be outside protection */
#include <trace/define_trace.h>

939
arch/loongarch/kvm/vcpu.c Normal file
View File

@ -0,0 +1,939 @@
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (C) 2020-2023 Loongson Technology Corporation Limited
*/
#include <linux/kvm_host.h>
#include <linux/entry-kvm.h>
#include <asm/fpu.h>
#include <asm/loongarch.h>
#include <asm/setup.h>
#include <asm/time.h>
#define CREATE_TRACE_POINTS
#include "trace.h"
const struct _kvm_stats_desc kvm_vcpu_stats_desc[] = {
KVM_GENERIC_VCPU_STATS(),
STATS_DESC_COUNTER(VCPU, int_exits),
STATS_DESC_COUNTER(VCPU, idle_exits),
STATS_DESC_COUNTER(VCPU, cpucfg_exits),
STATS_DESC_COUNTER(VCPU, signal_exits),
};
const struct kvm_stats_header kvm_vcpu_stats_header = {
.name_size = KVM_STATS_NAME_SIZE,
.num_desc = ARRAY_SIZE(kvm_vcpu_stats_desc),
.id_offset = sizeof(struct kvm_stats_header),
.desc_offset = sizeof(struct kvm_stats_header) + KVM_STATS_NAME_SIZE,
.data_offset = sizeof(struct kvm_stats_header) + KVM_STATS_NAME_SIZE +
sizeof(kvm_vcpu_stats_desc),
};
/*
* kvm_check_requests - check and handle pending vCPU requests
*
* Return: RESUME_GUEST if we should enter the guest
* RESUME_HOST if we should exit to userspace
*/
static int kvm_check_requests(struct kvm_vcpu *vcpu)
{
if (!kvm_request_pending(vcpu))
return RESUME_GUEST;
if (kvm_check_request(KVM_REQ_TLB_FLUSH, vcpu))
vcpu->arch.vpid = 0; /* Drop vpid for this vCPU */
if (kvm_dirty_ring_check_request(vcpu))
return RESUME_HOST;
return RESUME_GUEST;
}
/*
* Check and handle pending signal and vCPU requests etc
* Run with irq enabled and preempt enabled
*
* Return: RESUME_GUEST if we should enter the guest
* RESUME_HOST if we should exit to userspace
* < 0 if we should exit to userspace, where the return value
* indicates an error
*/
static int kvm_enter_guest_check(struct kvm_vcpu *vcpu)
{
int ret;
/*
* Check conditions before entering the guest
*/
ret = xfer_to_guest_mode_handle_work(vcpu);
if (ret < 0)
return ret;
ret = kvm_check_requests(vcpu);
return ret;
}
/*
* Called with irq enabled
*
* Return: RESUME_GUEST if we should enter the guest, and irq disabled
* Others if we should exit to userspace
*/
static int kvm_pre_enter_guest(struct kvm_vcpu *vcpu)
{
int ret;
do {
ret = kvm_enter_guest_check(vcpu);
if (ret != RESUME_GUEST)
break;
/*
* Handle vcpu timer, interrupts, check requests and
* check vmid before vcpu enter guest
*/
local_irq_disable();
kvm_acquire_timer(vcpu);
kvm_deliver_intr(vcpu);
kvm_deliver_exception(vcpu);
/* Make sure the vcpu mode has been written */
smp_store_mb(vcpu->mode, IN_GUEST_MODE);
kvm_check_vpid(vcpu);
vcpu->arch.host_eentry = csr_read64(LOONGARCH_CSR_EENTRY);
/* Clear KVM_LARCH_SWCSR_LATEST as CSR will change when enter guest */
vcpu->arch.aux_inuse &= ~KVM_LARCH_SWCSR_LATEST;
if (kvm_request_pending(vcpu) || xfer_to_guest_mode_work_pending()) {
/* make sure the vcpu mode has been written */
smp_store_mb(vcpu->mode, OUTSIDE_GUEST_MODE);
local_irq_enable();
ret = -EAGAIN;
}
} while (ret != RESUME_GUEST);
return ret;
}
/*
* Return 1 for resume guest and "<= 0" for resume host.
*/
static int kvm_handle_exit(struct kvm_run *run, struct kvm_vcpu *vcpu)
{
int ret = RESUME_GUEST;
unsigned long estat = vcpu->arch.host_estat;
u32 intr = estat & 0x1fff; /* Ignore NMI */
u32 ecode = (estat & CSR_ESTAT_EXC) >> CSR_ESTAT_EXC_SHIFT;
vcpu->mode = OUTSIDE_GUEST_MODE;
/* Set a default exit reason */
run->exit_reason = KVM_EXIT_UNKNOWN;
guest_timing_exit_irqoff();
guest_state_exit_irqoff();
local_irq_enable();
trace_kvm_exit(vcpu, ecode);
if (ecode) {
ret = kvm_handle_fault(vcpu, ecode);
} else {
WARN(!intr, "vm exiting with suspicious irq\n");
++vcpu->stat.int_exits;
}
if (ret == RESUME_GUEST)
ret = kvm_pre_enter_guest(vcpu);
if (ret != RESUME_GUEST) {
local_irq_disable();
return ret;
}
guest_timing_enter_irqoff();
guest_state_enter_irqoff();
trace_kvm_reenter(vcpu);
return RESUME_GUEST;
}
int kvm_arch_vcpu_runnable(struct kvm_vcpu *vcpu)
{
return !!(vcpu->arch.irq_pending) &&
vcpu->arch.mp_state.mp_state == KVM_MP_STATE_RUNNABLE;
}
int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu)
{
return kvm_vcpu_exiting_guest_mode(vcpu) == IN_GUEST_MODE;
}
bool kvm_arch_vcpu_in_kernel(struct kvm_vcpu *vcpu)
{
return false;
}
vm_fault_t kvm_arch_vcpu_fault(struct kvm_vcpu *vcpu, struct vm_fault *vmf)
{
return VM_FAULT_SIGBUS;
}
int kvm_arch_vcpu_ioctl_translate(struct kvm_vcpu *vcpu,
struct kvm_translation *tr)
{
return -EINVAL;
}
int kvm_cpu_has_pending_timer(struct kvm_vcpu *vcpu)
{
return kvm_pending_timer(vcpu) ||
kvm_read_hw_gcsr(LOONGARCH_CSR_ESTAT) & (1 << INT_TI);
}
int kvm_arch_vcpu_dump_regs(struct kvm_vcpu *vcpu)
{
int i;
kvm_debug("vCPU Register Dump:\n");
kvm_debug("\tPC = 0x%08lx\n", vcpu->arch.pc);
kvm_debug("\tExceptions: %08lx\n", vcpu->arch.irq_pending);
for (i = 0; i < 32; i += 4) {
kvm_debug("\tGPR%02d: %08lx %08lx %08lx %08lx\n", i,
vcpu->arch.gprs[i], vcpu->arch.gprs[i + 1],
vcpu->arch.gprs[i + 2], vcpu->arch.gprs[i + 3]);
}
kvm_debug("\tCRMD: 0x%08lx, ESTAT: 0x%08lx\n",
kvm_read_hw_gcsr(LOONGARCH_CSR_CRMD),
kvm_read_hw_gcsr(LOONGARCH_CSR_ESTAT));
kvm_debug("\tERA: 0x%08lx\n", kvm_read_hw_gcsr(LOONGARCH_CSR_ERA));
return 0;
}
int kvm_arch_vcpu_ioctl_get_mpstate(struct kvm_vcpu *vcpu,
struct kvm_mp_state *mp_state)
{
*mp_state = vcpu->arch.mp_state;
return 0;
}
int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu,
struct kvm_mp_state *mp_state)
{
int ret = 0;
switch (mp_state->mp_state) {
case KVM_MP_STATE_RUNNABLE:
vcpu->arch.mp_state = *mp_state;
break;
default:
ret = -EINVAL;
}
return ret;
}
int kvm_arch_vcpu_ioctl_set_guest_debug(struct kvm_vcpu *vcpu,
struct kvm_guest_debug *dbg)
{
return -EINVAL;
}
/**
* kvm_migrate_count() - Migrate timer.
* @vcpu: Virtual CPU.
*
* Migrate hrtimer to the current CPU by cancelling and restarting it
* if the hrtimer is active.
*
* Must be called when the vCPU is migrated to a different CPU, so that
* the timer can interrupt the guest at the new CPU, and the timer irq can
* be delivered to the vCPU.
*/
static void kvm_migrate_count(struct kvm_vcpu *vcpu)
{
if (hrtimer_cancel(&vcpu->arch.swtimer))
hrtimer_restart(&vcpu->arch.swtimer);
}
static int _kvm_getcsr(struct kvm_vcpu *vcpu, unsigned int id, u64 *val)
{
unsigned long gintc;
struct loongarch_csrs *csr = vcpu->arch.csr;
if (get_gcsr_flag(id) & INVALID_GCSR)
return -EINVAL;
if (id == LOONGARCH_CSR_ESTAT) {
/* ESTAT IP0~IP7 get from GINTC */
gintc = kvm_read_sw_gcsr(csr, LOONGARCH_CSR_GINTC) & 0xff;
*val = kvm_read_sw_gcsr(csr, LOONGARCH_CSR_ESTAT) | (gintc << 2);
return 0;
}
/*
* Get software CSR state since software state is consistent
* with hardware for synchronous ioctl
*/
*val = kvm_read_sw_gcsr(csr, id);
return 0;
}
static int _kvm_setcsr(struct kvm_vcpu *vcpu, unsigned int id, u64 val)
{
int ret = 0, gintc;
struct loongarch_csrs *csr = vcpu->arch.csr;
if (get_gcsr_flag(id) & INVALID_GCSR)
return -EINVAL;
if (id == LOONGARCH_CSR_ESTAT) {
/* ESTAT IP0~IP7 inject through GINTC */
gintc = (val >> 2) & 0xff;
kvm_set_sw_gcsr(csr, LOONGARCH_CSR_GINTC, gintc);
gintc = val & ~(0xffUL << 2);
kvm_set_sw_gcsr(csr, LOONGARCH_CSR_ESTAT, gintc);
return ret;
}
kvm_write_sw_gcsr(csr, id, val);
return ret;
}
static int kvm_get_one_reg(struct kvm_vcpu *vcpu,
const struct kvm_one_reg *reg, u64 *v)
{
int id, ret = 0;
u64 type = reg->id & KVM_REG_LOONGARCH_MASK;
switch (type) {
case KVM_REG_LOONGARCH_CSR:
id = KVM_GET_IOC_CSR_IDX(reg->id);
ret = _kvm_getcsr(vcpu, id, v);
break;
case KVM_REG_LOONGARCH_CPUCFG:
id = KVM_GET_IOC_CPUCFG_IDX(reg->id);
if (id >= 0 && id < KVM_MAX_CPUCFG_REGS)
*v = vcpu->arch.cpucfg[id];
else
ret = -EINVAL;
break;
case KVM_REG_LOONGARCH_KVM:
switch (reg->id) {
case KVM_REG_LOONGARCH_COUNTER:
*v = drdtime() + vcpu->kvm->arch.time_offset;
break;
default:
ret = -EINVAL;
break;
}
break;
default:
ret = -EINVAL;
break;
}
return ret;
}
static int kvm_get_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
{
int ret = 0;
u64 v, size = reg->id & KVM_REG_SIZE_MASK;
switch (size) {
case KVM_REG_SIZE_U64:
ret = kvm_get_one_reg(vcpu, reg, &v);
if (ret)
return ret;
ret = put_user(v, (u64 __user *)(long)reg->addr);
break;
default:
ret = -EINVAL;
break;
}
return ret;
}
static int kvm_set_one_reg(struct kvm_vcpu *vcpu,
const struct kvm_one_reg *reg, u64 v)
{
int id, ret = 0;
u64 type = reg->id & KVM_REG_LOONGARCH_MASK;
switch (type) {
case KVM_REG_LOONGARCH_CSR:
id = KVM_GET_IOC_CSR_IDX(reg->id);
ret = _kvm_setcsr(vcpu, id, v);
break;
case KVM_REG_LOONGARCH_CPUCFG:
id = KVM_GET_IOC_CPUCFG_IDX(reg->id);
if (id >= 0 && id < KVM_MAX_CPUCFG_REGS)
vcpu->arch.cpucfg[id] = (u32)v;
else
ret = -EINVAL;
break;
case KVM_REG_LOONGARCH_KVM:
switch (reg->id) {
case KVM_REG_LOONGARCH_COUNTER:
/*
* gftoffset is relative with board, not vcpu
* only set for the first time for smp system
*/
if (vcpu->vcpu_id == 0)
vcpu->kvm->arch.time_offset = (signed long)(v - drdtime());
break;
case KVM_REG_LOONGARCH_VCPU_RESET:
kvm_reset_timer(vcpu);
memset(&vcpu->arch.irq_pending, 0, sizeof(vcpu->arch.irq_pending));
memset(&vcpu->arch.irq_clear, 0, sizeof(vcpu->arch.irq_clear));
break;
default:
ret = -EINVAL;
break;
}
break;
default:
ret = -EINVAL;
break;
}
return ret;
}
static int kvm_set_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
{
int ret = 0;
u64 v, size = reg->id & KVM_REG_SIZE_MASK;
switch (size) {
case KVM_REG_SIZE_U64:
ret = get_user(v, (u64 __user *)(long)reg->addr);
if (ret)
return ret;
break;
default:
return -EINVAL;
}
return kvm_set_one_reg(vcpu, reg, v);
}
int kvm_arch_vcpu_ioctl_get_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs)
{
return -ENOIOCTLCMD;
}
int kvm_arch_vcpu_ioctl_set_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs)
{
return -ENOIOCTLCMD;
}
int kvm_arch_vcpu_ioctl_get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs)
{
int i;
for (i = 0; i < ARRAY_SIZE(vcpu->arch.gprs); i++)
regs->gpr[i] = vcpu->arch.gprs[i];
regs->pc = vcpu->arch.pc;
return 0;
}
int kvm_arch_vcpu_ioctl_set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs)
{
int i;
for (i = 1; i < ARRAY_SIZE(vcpu->arch.gprs); i++)
vcpu->arch.gprs[i] = regs->gpr[i];
vcpu->arch.gprs[0] = 0; /* zero is special, and cannot be set. */
vcpu->arch.pc = regs->pc;
return 0;
}
static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,
struct kvm_enable_cap *cap)
{
/* FPU is enabled by default, will support LSX/LASX later. */
return -EINVAL;
}
long kvm_arch_vcpu_ioctl(struct file *filp,
unsigned int ioctl, unsigned long arg)
{
long r;
void __user *argp = (void __user *)arg;
struct kvm_vcpu *vcpu = filp->private_data;
/*
* Only software CSR should be modified
*
* If any hardware CSR register is modified, vcpu_load/vcpu_put pair
* should be used. Since CSR registers owns by this vcpu, if switch
* to other vcpus, other vcpus need reload CSR registers.
*
* If software CSR is modified, bit KVM_LARCH_HWCSR_USABLE should
* be clear in vcpu->arch.aux_inuse, and vcpu_load will check
* aux_inuse flag and reload CSR registers form software.
*/
switch (ioctl) {
case KVM_SET_ONE_REG:
case KVM_GET_ONE_REG: {
struct kvm_one_reg reg;
r = -EFAULT;
if (copy_from_user(&reg, argp, sizeof(reg)))
break;
if (ioctl == KVM_SET_ONE_REG) {
r = kvm_set_reg(vcpu, &reg);
vcpu->arch.aux_inuse &= ~KVM_LARCH_HWCSR_USABLE;
} else
r = kvm_get_reg(vcpu, &reg);
break;
}
case KVM_ENABLE_CAP: {
struct kvm_enable_cap cap;
r = -EFAULT;
if (copy_from_user(&cap, argp, sizeof(cap)))
break;
r = kvm_vcpu_ioctl_enable_cap(vcpu, &cap);
break;
}
default:
r = -ENOIOCTLCMD;
break;
}
return r;
}
int kvm_arch_vcpu_ioctl_get_fpu(struct kvm_vcpu *vcpu, struct kvm_fpu *fpu)
{
int i = 0;
fpu->fcc = vcpu->arch.fpu.fcc;
fpu->fcsr = vcpu->arch.fpu.fcsr;
for (i = 0; i < NUM_FPU_REGS; i++)
memcpy(&fpu->fpr[i], &vcpu->arch.fpu.fpr[i], FPU_REG_WIDTH / 64);
return 0;
}
int kvm_arch_vcpu_ioctl_set_fpu(struct kvm_vcpu *vcpu, struct kvm_fpu *fpu)
{
int i = 0;
vcpu->arch.fpu.fcc = fpu->fcc;
vcpu->arch.fpu.fcsr = fpu->fcsr;
for (i = 0; i < NUM_FPU_REGS; i++)
memcpy(&vcpu->arch.fpu.fpr[i], &fpu->fpr[i], FPU_REG_WIDTH / 64);
return 0;
}
/* Enable FPU and restore context */
void kvm_own_fpu(struct kvm_vcpu *vcpu)
{
preempt_disable();
/* Enable FPU */
set_csr_euen(CSR_EUEN_FPEN);
kvm_restore_fpu(&vcpu->arch.fpu);
vcpu->arch.aux_inuse |= KVM_LARCH_FPU;
trace_kvm_aux(vcpu, KVM_TRACE_AUX_RESTORE, KVM_TRACE_AUX_FPU);
preempt_enable();
}
/* Save context and disable FPU */
void kvm_lose_fpu(struct kvm_vcpu *vcpu)
{
preempt_disable();
if (vcpu->arch.aux_inuse & KVM_LARCH_FPU) {
kvm_save_fpu(&vcpu->arch.fpu);
vcpu->arch.aux_inuse &= ~KVM_LARCH_FPU;
trace_kvm_aux(vcpu, KVM_TRACE_AUX_SAVE, KVM_TRACE_AUX_FPU);
/* Disable FPU */
clear_csr_euen(CSR_EUEN_FPEN);
}
preempt_enable();
}
int kvm_vcpu_ioctl_interrupt(struct kvm_vcpu *vcpu, struct kvm_interrupt *irq)
{
int intr = (int)irq->irq;
if (intr > 0)
kvm_queue_irq(vcpu, intr);
else if (intr < 0)
kvm_dequeue_irq(vcpu, -intr);
else {
kvm_err("%s: invalid interrupt ioctl %d\n", __func__, irq->irq);
return -EINVAL;
}
kvm_vcpu_kick(vcpu);
return 0;
}
long kvm_arch_vcpu_async_ioctl(struct file *filp,
unsigned int ioctl, unsigned long arg)
{
void __user *argp = (void __user *)arg;
struct kvm_vcpu *vcpu = filp->private_data;
if (ioctl == KVM_INTERRUPT) {
struct kvm_interrupt irq;
if (copy_from_user(&irq, argp, sizeof(irq)))
return -EFAULT;
kvm_debug("[%d] %s: irq: %d\n", vcpu->vcpu_id, __func__, irq.irq);
return kvm_vcpu_ioctl_interrupt(vcpu, &irq);
}
return -ENOIOCTLCMD;
}
int kvm_arch_vcpu_precreate(struct kvm *kvm, unsigned int id)
{
return 0;
}
int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
{
unsigned long timer_hz;
struct loongarch_csrs *csr;
vcpu->arch.vpid = 0;
hrtimer_init(&vcpu->arch.swtimer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS_PINNED);
vcpu->arch.swtimer.function = kvm_swtimer_wakeup;
vcpu->arch.handle_exit = kvm_handle_exit;
vcpu->arch.guest_eentry = (unsigned long)kvm_loongarch_ops->exc_entry;
vcpu->arch.csr = kzalloc(sizeof(struct loongarch_csrs), GFP_KERNEL);
if (!vcpu->arch.csr)
return -ENOMEM;
/*
* All kvm exceptions share one exception entry, and host <-> guest
* switch also switch ECFG.VS field, keep host ECFG.VS info here.
*/
vcpu->arch.host_ecfg = (read_csr_ecfg() & CSR_ECFG_VS);
/* Init */
vcpu->arch.last_sched_cpu = -1;
/*
* Initialize guest register state to valid architectural reset state.
*/
timer_hz = calc_const_freq();
kvm_init_timer(vcpu, timer_hz);
/* Set Initialize mode for guest */
csr = vcpu->arch.csr;
kvm_write_sw_gcsr(csr, LOONGARCH_CSR_CRMD, CSR_CRMD_DA);
/* Set cpuid */
kvm_write_sw_gcsr(csr, LOONGARCH_CSR_TMID, vcpu->vcpu_id);
/* Start with no pending virtual guest interrupts */
csr->csrs[LOONGARCH_CSR_GINTC] = 0;
return 0;
}
void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)
{
}
void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu)
{
int cpu;
struct kvm_context *context;
hrtimer_cancel(&vcpu->arch.swtimer);
kvm_mmu_free_memory_cache(&vcpu->arch.mmu_page_cache);
kfree(vcpu->arch.csr);
/*
* If the vCPU is freed and reused as another vCPU, we don't want the
* matching pointer wrongly hanging around in last_vcpu.
*/
for_each_possible_cpu(cpu) {
context = per_cpu_ptr(vcpu->kvm->arch.vmcs, cpu);
if (context->last_vcpu == vcpu)
context->last_vcpu = NULL;
}
}
static int _kvm_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
{
bool migrated;
struct kvm_context *context;
struct loongarch_csrs *csr = vcpu->arch.csr;
/*
* Have we migrated to a different CPU?
* If so, any old guest TLB state may be stale.
*/
migrated = (vcpu->arch.last_sched_cpu != cpu);
/*
* Was this the last vCPU to run on this CPU?
* If not, any old guest state from this vCPU will have been clobbered.
*/
context = per_cpu_ptr(vcpu->kvm->arch.vmcs, cpu);
if (migrated || (context->last_vcpu != vcpu))
vcpu->arch.aux_inuse &= ~KVM_LARCH_HWCSR_USABLE;
context->last_vcpu = vcpu;
/* Restore timer state regardless */
kvm_restore_timer(vcpu);
/* Control guest page CCA attribute */
change_csr_gcfg(CSR_GCFG_MATC_MASK, CSR_GCFG_MATC_ROOT);
/* Don't bother restoring registers multiple times unless necessary */
if (vcpu->arch.aux_inuse & KVM_LARCH_HWCSR_USABLE)
return 0;
write_csr_gcntc((ulong)vcpu->kvm->arch.time_offset);
/* Restore guest CSR registers */
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_CRMD);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_PRMD);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_EUEN);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_MISC);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_ECFG);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_ERA);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_BADV);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_BADI);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_EENTRY);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_TLBIDX);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_TLBEHI);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_TLBELO0);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_TLBELO1);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_ASID);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_PGDL);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_PGDH);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_PWCTL0);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_PWCTL1);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_STLBPGSIZE);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_RVACFG);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_CPUID);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_KS0);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_KS1);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_KS2);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_KS3);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_KS4);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_KS5);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_KS6);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_KS7);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_TMID);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_CNTC);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_TLBRENTRY);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_TLBRBADV);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_TLBRERA);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_TLBRSAVE);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_TLBRELO0);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_TLBRELO1);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_TLBREHI);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_TLBRPRMD);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_DMWIN0);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_DMWIN1);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_DMWIN2);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_DMWIN3);
kvm_restore_hw_gcsr(csr, LOONGARCH_CSR_LLBCTL);
/* Restore Root.GINTC from unused Guest.GINTC register */
write_csr_gintc(csr->csrs[LOONGARCH_CSR_GINTC]);
/*
* We should clear linked load bit to break interrupted atomics. This
* prevents a SC on the next vCPU from succeeding by matching a LL on
* the previous vCPU.
*/
if (vcpu->kvm->created_vcpus > 1)
set_gcsr_llbctl(CSR_LLBCTL_WCLLB);
vcpu->arch.aux_inuse |= KVM_LARCH_HWCSR_USABLE;
return 0;
}
void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
{
unsigned long flags;
local_irq_save(flags);
if (vcpu->arch.last_sched_cpu != cpu) {
kvm_debug("[%d->%d]KVM vCPU[%d] switch\n",
vcpu->arch.last_sched_cpu, cpu, vcpu->vcpu_id);
/*
* Migrate the timer interrupt to the current CPU so that it
* always interrupts the guest and synchronously triggers a
* guest timer interrupt.
*/
kvm_migrate_count(vcpu);
}
/* Restore guest state to registers */
_kvm_vcpu_load(vcpu, cpu);
local_irq_restore(flags);
}
static int _kvm_vcpu_put(struct kvm_vcpu *vcpu, int cpu)
{
struct loongarch_csrs *csr = vcpu->arch.csr;
kvm_lose_fpu(vcpu);
/*
* Update CSR state from hardware if software CSR state is stale,
* most CSR registers are kept unchanged during process context
* switch except CSR registers like remaining timer tick value and
* injected interrupt state.
*/
if (vcpu->arch.aux_inuse & KVM_LARCH_SWCSR_LATEST)
goto out;
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_CRMD);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_PRMD);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_EUEN);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_MISC);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_ECFG);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_ERA);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_BADV);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_BADI);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_EENTRY);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_TLBIDX);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_TLBEHI);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_TLBELO0);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_TLBELO1);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_ASID);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_PGDL);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_PGDH);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_PWCTL0);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_PWCTL1);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_STLBPGSIZE);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_RVACFG);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_CPUID);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_PRCFG1);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_PRCFG2);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_PRCFG3);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_KS0);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_KS1);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_KS2);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_KS3);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_KS4);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_KS5);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_KS6);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_KS7);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_TMID);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_CNTC);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_LLBCTL);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_TLBRENTRY);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_TLBRBADV);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_TLBRERA);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_TLBRSAVE);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_TLBRELO0);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_TLBRELO1);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_TLBREHI);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_TLBRPRMD);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_DMWIN0);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_DMWIN1);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_DMWIN2);
kvm_save_hw_gcsr(csr, LOONGARCH_CSR_DMWIN3);
vcpu->arch.aux_inuse |= KVM_LARCH_SWCSR_LATEST;
out:
kvm_save_timer(vcpu);
/* Save Root.GINTC into unused Guest.GINTC register */
csr->csrs[LOONGARCH_CSR_GINTC] = read_csr_gintc();
return 0;
}
void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
{
int cpu;
unsigned long flags;
local_irq_save(flags);
cpu = smp_processor_id();
vcpu->arch.last_sched_cpu = cpu;
/* Save guest state in registers */
_kvm_vcpu_put(vcpu, cpu);
local_irq_restore(flags);
}
int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
{
int r = -EINTR;
struct kvm_run *run = vcpu->run;
if (vcpu->mmio_needed) {
if (!vcpu->mmio_is_write)
kvm_complete_mmio_read(vcpu, run);
vcpu->mmio_needed = 0;
}
if (run->exit_reason == KVM_EXIT_LOONGARCH_IOCSR) {
if (!run->iocsr_io.is_write)
kvm_complete_iocsr_read(vcpu, run);
}
if (run->immediate_exit)
return r;
/* Clear exit_reason */
run->exit_reason = KVM_EXIT_UNKNOWN;
lose_fpu(1);
vcpu_load(vcpu);
kvm_sigset_activate(vcpu);
r = kvm_pre_enter_guest(vcpu);
if (r != RESUME_GUEST)
goto out;
guest_timing_enter_irqoff();
guest_state_enter_irqoff();
trace_kvm_enter(vcpu);
r = kvm_loongarch_ops->enter_guest(run, vcpu);
trace_kvm_out(vcpu);
/*
* Guest exit is already recorded at kvm_handle_exit()
* return value must not be RESUME_GUEST
*/
local_irq_enable();
out:
kvm_sigset_deactivate(vcpu);
vcpu_put(vcpu);
return r;
}

94
arch/loongarch/kvm/vm.c Normal file
View File

@ -0,0 +1,94 @@
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (C) 2020-2023 Loongson Technology Corporation Limited
*/
#include <linux/kvm_host.h>
#include <asm/kvm_mmu.h>
const struct _kvm_stats_desc kvm_vm_stats_desc[] = {
KVM_GENERIC_VM_STATS(),
STATS_DESC_ICOUNTER(VM, pages),
STATS_DESC_ICOUNTER(VM, hugepages),
};
const struct kvm_stats_header kvm_vm_stats_header = {
.name_size = KVM_STATS_NAME_SIZE,
.num_desc = ARRAY_SIZE(kvm_vm_stats_desc),
.id_offset = sizeof(struct kvm_stats_header),
.desc_offset = sizeof(struct kvm_stats_header) + KVM_STATS_NAME_SIZE,
.data_offset = sizeof(struct kvm_stats_header) + KVM_STATS_NAME_SIZE +
sizeof(kvm_vm_stats_desc),
};
int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
{
int i;
/* Allocate page table to map GPA -> RPA */
kvm->arch.pgd = kvm_pgd_alloc();
if (!kvm->arch.pgd)
return -ENOMEM;
kvm_init_vmcs(kvm);
kvm->arch.gpa_size = BIT(cpu_vabits - 1);
kvm->arch.root_level = CONFIG_PGTABLE_LEVELS - 1;
kvm->arch.invalid_ptes[0] = 0;
kvm->arch.invalid_ptes[1] = (unsigned long)invalid_pte_table;
#if CONFIG_PGTABLE_LEVELS > 2
kvm->arch.invalid_ptes[2] = (unsigned long)invalid_pmd_table;
#endif
#if CONFIG_PGTABLE_LEVELS > 3
kvm->arch.invalid_ptes[3] = (unsigned long)invalid_pud_table;
#endif
for (i = 0; i <= kvm->arch.root_level; i++)
kvm->arch.pte_shifts[i] = PAGE_SHIFT + i * (PAGE_SHIFT - 3);
return 0;
}
void kvm_arch_destroy_vm(struct kvm *kvm)
{
kvm_destroy_vcpus(kvm);
free_page((unsigned long)kvm->arch.pgd);
kvm->arch.pgd = NULL;
}
int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
{
int r;
switch (ext) {
case KVM_CAP_ONE_REG:
case KVM_CAP_ENABLE_CAP:
case KVM_CAP_READONLY_MEM:
case KVM_CAP_SYNC_MMU:
case KVM_CAP_IMMEDIATE_EXIT:
case KVM_CAP_IOEVENTFD:
case KVM_CAP_MP_STATE:
r = 1;
break;
case KVM_CAP_NR_VCPUS:
r = num_online_cpus();
break;
case KVM_CAP_MAX_VCPUS:
r = KVM_MAX_VCPUS;
break;
case KVM_CAP_MAX_VCPU_ID:
r = KVM_MAX_VCPU_IDS;
break;
case KVM_CAP_NR_MEMSLOTS:
r = KVM_USER_MEM_SLOTS;
break;
default:
r = 0;
break;
}
return r;
}
int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg)
{
return -ENOIOCTLCMD;
}

18
arch/riscv/include/asm/csr.h
View File

@ -203,6 +203,18 @@
#define ENVCFG_CBIE_INV _AC(0x3, UL)
#define ENVCFG_FIOM _AC(0x1, UL)
/* Smstateen bits */
#define SMSTATEEN0_AIA_IMSIC_SHIFT 58
#define SMSTATEEN0_AIA_IMSIC (_ULL(1) << SMSTATEEN0_AIA_IMSIC_SHIFT)
#define SMSTATEEN0_AIA_SHIFT 59
#define SMSTATEEN0_AIA (_ULL(1) << SMSTATEEN0_AIA_SHIFT)
#define SMSTATEEN0_AIA_ISEL_SHIFT 60
#define SMSTATEEN0_AIA_ISEL (_ULL(1) << SMSTATEEN0_AIA_ISEL_SHIFT)
#define SMSTATEEN0_HSENVCFG_SHIFT 62
#define SMSTATEEN0_HSENVCFG (_ULL(1) << SMSTATEEN0_HSENVCFG_SHIFT)
#define SMSTATEEN0_SSTATEEN0_SHIFT 63
#define SMSTATEEN0_SSTATEEN0 (_ULL(1) << SMSTATEEN0_SSTATEEN0_SHIFT)
/* symbolic CSR names: */
#define CSR_CYCLE 0xc00
#define CSR_TIME 0xc01
@ -275,6 +287,8 @@
#define CSR_SIE 0x104
#define CSR_STVEC 0x105
#define CSR_SCOUNTEREN 0x106
#define CSR_SENVCFG 0x10a
#define CSR_SSTATEEN0 0x10c
#define CSR_SSCRATCH 0x140
#define CSR_SEPC 0x141
#define CSR_SCAUSE 0x142
@ -349,6 +363,10 @@
#define CSR_VSIEH 0x214
#define CSR_VSIPH 0x254
/* Hypervisor stateen CSRs */
#define CSR_HSTATEEN0 0x60c
#define CSR_HSTATEEN0H 0x61c
#define CSR_MSTATUS 0x300
#define CSR_MISA 0x301
#define CSR_MIDELEG 0x303

2
arch/riscv/include/asm/hwcap.h
View File

@ -58,6 +58,8 @@
#define RISCV_ISA_EXT_ZICSR 40
#define RISCV_ISA_EXT_ZIFENCEI 41
#define RISCV_ISA_EXT_ZIHPM 42
#define RISCV_ISA_EXT_SMSTATEEN 43
#define RISCV_ISA_EXT_ZICOND 44
#define RISCV_ISA_EXT_MAX 64

18
arch/riscv/include/asm/kvm_host.h
View File

@ -162,6 +162,16 @@ struct kvm_vcpu_csr {
unsigned long hvip;
unsigned long vsatp;
unsigned long scounteren;
unsigned long senvcfg;
};
struct kvm_vcpu_config {
u64 henvcfg;
u64 hstateen0;
};
struct kvm_vcpu_smstateen_csr {
unsigned long sstateen0;
};
struct kvm_vcpu_arch {
@ -183,6 +193,8 @@ struct kvm_vcpu_arch {
unsigned long host_sscratch;
unsigned long host_stvec;
unsigned long host_scounteren;
unsigned long host_senvcfg;
unsigned long host_sstateen0;
/* CPU context of Host */
struct kvm_cpu_context host_context;
@ -193,6 +205,9 @@ struct kvm_vcpu_arch {
/* CPU CSR context of Guest VCPU */
struct kvm_vcpu_csr guest_csr;
/* CPU Smstateen CSR context of Guest VCPU */
struct kvm_vcpu_smstateen_csr smstateen_csr;
/* CPU context upon Guest VCPU reset */
struct kvm_cpu_context guest_reset_context;
@ -244,6 +259,9 @@ struct kvm_vcpu_arch {
/* Performance monitoring context */
struct kvm_pmu pmu_context;
/* 'static' configurations which are set only once */
struct kvm_vcpu_config cfg;
};
static inline void kvm_arch_sync_events(struct kvm *kvm) {}

7
arch/riscv/include/asm/kvm_vcpu_sbi.h
View File

@ -11,7 +11,7 @@
#define KVM_SBI_IMPID 3
#define KVM_SBI_VERSION_MAJOR 1
#define KVM_SBI_VERSION_MAJOR 2
#define KVM_SBI_VERSION_MINOR 0
enum kvm_riscv_sbi_ext_status {
@ -35,6 +35,9 @@ struct kvm_vcpu_sbi_return {
struct kvm_vcpu_sbi_extension {
unsigned long extid_start;
unsigned long extid_end;
bool default_unavail;
/**
* SBI extension handler. It can be defined for a given extension or group of
* extension. But it should always return linux error codes rather than SBI
@ -59,6 +62,7 @@ int kvm_riscv_vcpu_get_reg_sbi_ext(struct kvm_vcpu *vcpu,
const struct kvm_vcpu_sbi_extension *kvm_vcpu_sbi_find_ext(
struct kvm_vcpu *vcpu, unsigned long extid);
int kvm_riscv_vcpu_sbi_ecall(struct kvm_vcpu *vcpu, struct kvm_run *run);
void kvm_riscv_vcpu_sbi_init(struct kvm_vcpu *vcpu);
#ifdef CONFIG_RISCV_SBI_V01
extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_v01;
@ -69,6 +73,7 @@ extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_ipi;
extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_rfence;
extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_srst;
extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_hsm;
extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_dbcn;
extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_experimental;
extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_vendor;

7
arch/riscv/include/asm/sbi.h
View File

@ -30,6 +30,7 @@ enum sbi_ext_id {
SBI_EXT_HSM = 0x48534D,
SBI_EXT_SRST = 0x53525354,
SBI_EXT_PMU = 0x504D55,
SBI_EXT_DBCN = 0x4442434E,
/* Experimentals extensions must lie within this range */
SBI_EXT_EXPERIMENTAL_START = 0x08000000,
@ -236,6 +237,12 @@ enum sbi_pmu_ctr_type {
/* Flags defined for counter stop function */
#define SBI_PMU_STOP_FLAG_RESET (1 << 0)
enum sbi_ext_dbcn_fid {
SBI_EXT_DBCN_CONSOLE_WRITE = 0,
SBI_EXT_DBCN_CONSOLE_READ = 1,
SBI_EXT_DBCN_CONSOLE_WRITE_BYTE = 2,
};
#define SBI_SPEC_VERSION_DEFAULT 0x1
#define SBI_SPEC_VERSION_MAJOR_SHIFT 24
#define SBI_SPEC_VERSION_MAJOR_MASK 0x7f

12
arch/riscv/include/uapi/asm/kvm.h
View File

@ -80,6 +80,7 @@ struct kvm_riscv_csr {
unsigned long sip;
unsigned long satp;
unsigned long scounteren;
unsigned long senvcfg;
};
/* AIA CSR registers for KVM_GET_ONE_REG and KVM_SET_ONE_REG */
@ -93,6 +94,11 @@ struct kvm_riscv_aia_csr {
unsigned long iprio2h;
};
/* Smstateen CSR for KVM_GET_ONE_REG and KVM_SET_ONE_REG */
struct kvm_riscv_smstateen_csr {
unsigned long sstateen0;
};
/* TIMER registers for KVM_GET_ONE_REG and KVM_SET_ONE_REG */
struct kvm_riscv_timer {
__u64 frequency;
@ -131,6 +137,8 @@ enum KVM_RISCV_ISA_EXT_ID {
KVM_RISCV_ISA_EXT_ZICSR,
KVM_RISCV_ISA_EXT_ZIFENCEI,
KVM_RISCV_ISA_EXT_ZIHPM,
KVM_RISCV_ISA_EXT_SMSTATEEN,
KVM_RISCV_ISA_EXT_ZICOND,
KVM_RISCV_ISA_EXT_MAX,
};
@ -148,6 +156,7 @@ enum KVM_RISCV_SBI_EXT_ID {
KVM_RISCV_SBI_EXT_PMU,
KVM_RISCV_SBI_EXT_EXPERIMENTAL,
KVM_RISCV_SBI_EXT_VENDOR,
KVM_RISCV_SBI_EXT_DBCN,
KVM_RISCV_SBI_EXT_MAX,
};
@ -178,10 +187,13 @@ enum KVM_RISCV_SBI_EXT_ID {
#define KVM_REG_RISCV_CSR (0x03 << KVM_REG_RISCV_TYPE_SHIFT)
#define KVM_REG_RISCV_CSR_GENERAL (0x0 << KVM_REG_RISCV_SUBTYPE_SHIFT)
#define KVM_REG_RISCV_CSR_AIA (0x1 << KVM_REG_RISCV_SUBTYPE_SHIFT)
#define KVM_REG_RISCV_CSR_SMSTATEEN (0x2 << KVM_REG_RISCV_SUBTYPE_SHIFT)
#define KVM_REG_RISCV_CSR_REG(name) \
(offsetof(struct kvm_riscv_csr, name) / sizeof(unsigned long))
#define KVM_REG_RISCV_CSR_AIA_REG(name) \
(offsetof(struct kvm_riscv_aia_csr, name) / sizeof(unsigned long))
#define KVM_REG_RISCV_CSR_SMSTATEEN_REG(name) \
(offsetof(struct kvm_riscv_smstateen_csr, name) / sizeof(unsigned long))
/* Timer registers are mapped as type 4 */
#define KVM_REG_RISCV_TIMER (0x04 << KVM_REG_RISCV_TYPE_SHIFT)

2
arch/riscv/kernel/cpufeature.c
View File

@ -167,6 +167,7 @@ const struct riscv_isa_ext_data riscv_isa_ext[] = {
__RISCV_ISA_EXT_DATA(zicbom, RISCV_ISA_EXT_ZICBOM),
__RISCV_ISA_EXT_DATA(zicboz, RISCV_ISA_EXT_ZICBOZ),
__RISCV_ISA_EXT_DATA(zicntr, RISCV_ISA_EXT_ZICNTR),
__RISCV_ISA_EXT_DATA(zicond, RISCV_ISA_EXT_ZICOND),
__RISCV_ISA_EXT_DATA(zicsr, RISCV_ISA_EXT_ZICSR),
__RISCV_ISA_EXT_DATA(zifencei, RISCV_ISA_EXT_ZIFENCEI),
__RISCV_ISA_EXT_DATA(zihintpause, RISCV_ISA_EXT_ZIHINTPAUSE),
@ -175,6 +176,7 @@ const struct riscv_isa_ext_data riscv_isa_ext[] = {
__RISCV_ISA_EXT_DATA(zbb, RISCV_ISA_EXT_ZBB),
__RISCV_ISA_EXT_DATA(zbs, RISCV_ISA_EXT_ZBS),
__RISCV_ISA_EXT_DATA(smaia, RISCV_ISA_EXT_SMAIA),
__RISCV_ISA_EXT_DATA(smstateen, RISCV_ISA_EXT_SMSTATEEN),
__RISCV_ISA_EXT_DATA(ssaia, RISCV_ISA_EXT_SSAIA),
__RISCV_ISA_EXT_DATA(sscofpmf, RISCV_ISA_EXT_SSCOFPMF),
__RISCV_ISA_EXT_DATA(sstc, RISCV_ISA_EXT_SSTC),

74
arch/riscv/kvm/vcpu.c
View File

@ -141,6 +141,12 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
if (rc)
return rc;
/*
* Setup SBI extensions
* NOTE: This must be the last thing to be initialized.
*/
kvm_riscv_vcpu_sbi_init(vcpu);
/* Reset VCPU */
kvm_riscv_reset_vcpu(vcpu);
@ -471,31 +477,38 @@ int kvm_arch_vcpu_ioctl_set_guest_debug(struct kvm_vcpu *vcpu,
return -EINVAL;
}
static void kvm_riscv_vcpu_update_config(const unsigned long *isa)
static void kvm_riscv_vcpu_setup_config(struct kvm_vcpu *vcpu)
{
u64 henvcfg = 0;
const unsigned long *isa = vcpu->arch.isa;
struct kvm_vcpu_config *cfg = &vcpu->arch.cfg;
if (riscv_isa_extension_available(isa, SVPBMT))
henvcfg |= ENVCFG_PBMTE;
cfg->henvcfg |= ENVCFG_PBMTE;
if (riscv_isa_extension_available(isa, SSTC))
henvcfg |= ENVCFG_STCE;
cfg->henvcfg |= ENVCFG_STCE;
if (riscv_isa_extension_available(isa, ZICBOM))
henvcfg |= (ENVCFG_CBIE | ENVCFG_CBCFE);
cfg->henvcfg |= (ENVCFG_CBIE | ENVCFG_CBCFE);
if (riscv_isa_extension_available(isa, ZICBOZ))
henvcfg |= ENVCFG_CBZE;
cfg->henvcfg |= ENVCFG_CBZE;
csr_write(CSR_HENVCFG, henvcfg);
#ifdef CONFIG_32BIT
csr_write(CSR_HENVCFGH, henvcfg >> 32);
#endif
if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SMSTATEEN)) {
cfg->hstateen0 |= SMSTATEEN0_HSENVCFG;
if (riscv_isa_extension_available(isa, SSAIA))
cfg->hstateen0 |= SMSTATEEN0_AIA_IMSIC |
SMSTATEEN0_AIA |
SMSTATEEN0_AIA_ISEL;
if (riscv_isa_extension_available(isa, SMSTATEEN))
cfg->hstateen0 |= SMSTATEEN0_SSTATEEN0;
}
}
void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
{
struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;
struct kvm_vcpu_config *cfg = &vcpu->arch.cfg;
csr_write(CSR_VSSTATUS, csr->vsstatus);
csr_write(CSR_VSIE, csr->vsie);
@ -506,8 +519,14 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
csr_write(CSR_VSTVAL, csr->vstval);
csr_write(CSR_HVIP, csr->hvip);
csr_write(CSR_VSATP, csr->vsatp);
kvm_riscv_vcpu_update_config(vcpu->arch.isa);
csr_write(CSR_HENVCFG, cfg->henvcfg);
if (IS_ENABLED(CONFIG_32BIT))
csr_write(CSR_HENVCFGH, cfg->henvcfg >> 32);
if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SMSTATEEN)) {
csr_write(CSR_HSTATEEN0, cfg->hstateen0);
if (IS_ENABLED(CONFIG_32BIT))
csr_write(CSR_HSTATEEN0H, cfg->hstateen0 >> 32);
}
kvm_riscv_gstage_update_hgatp(vcpu);
@ -606,6 +625,32 @@ static void kvm_riscv_update_hvip(struct kvm_vcpu *vcpu)
kvm_riscv_vcpu_aia_update_hvip(vcpu);
}
static __always_inline void kvm_riscv_vcpu_swap_in_guest_state(struct kvm_vcpu *vcpu)
{
struct kvm_vcpu_smstateen_csr *smcsr = &vcpu->arch.smstateen_csr;
struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;
struct kvm_vcpu_config *cfg = &vcpu->arch.cfg;
vcpu->arch.host_senvcfg = csr_swap(CSR_SENVCFG, csr->senvcfg);
if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SMSTATEEN) &&
(cfg->hstateen0 & SMSTATEEN0_SSTATEEN0))
vcpu->arch.host_sstateen0 = csr_swap(CSR_SSTATEEN0,
smcsr->sstateen0);
}
static __always_inline void kvm_riscv_vcpu_swap_in_host_state(struct kvm_vcpu *vcpu)
{
struct kvm_vcpu_smstateen_csr *smcsr = &vcpu->arch.smstateen_csr;
struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;
struct kvm_vcpu_config *cfg = &vcpu->arch.cfg;
csr->senvcfg = csr_swap(CSR_SENVCFG, vcpu->arch.host_senvcfg);
if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SMSTATEEN) &&
(cfg->hstateen0 & SMSTATEEN0_SSTATEEN0))
smcsr->sstateen0 = csr_swap(CSR_SSTATEEN0,
vcpu->arch.host_sstateen0);
}
/*
* Actually run the vCPU, entering an RCU extended quiescent state (EQS) while
* the vCPU is running.
@ -615,10 +660,12 @@ static void kvm_riscv_update_hvip(struct kvm_vcpu *vcpu)
*/
static void noinstr kvm_riscv_vcpu_enter_exit(struct kvm_vcpu *vcpu)
{
kvm_riscv_vcpu_swap_in_guest_state(vcpu);
guest_state_enter_irqoff();
__kvm_riscv_switch_to(&vcpu->arch);
vcpu->arch.last_exit_cpu = vcpu->cpu;
guest_state_exit_irqoff();
kvm_riscv_vcpu_swap_in_host_state(vcpu);
}
int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
@ -627,6 +674,9 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
struct kvm_cpu_trap trap;
struct kvm_run *run = vcpu->run;
if (!vcpu->arch.ran_atleast_once)
kvm_riscv_vcpu_setup_config(vcpu);
/* Mark this VCPU ran at least once */
vcpu->arch.ran_atleast_once = true;

72
arch/riscv/kvm/vcpu_onereg.c
View File

@ -34,6 +34,7 @@ static const unsigned long kvm_isa_ext_arr[] = {
[KVM_RISCV_ISA_EXT_M] = RISCV_ISA_EXT_m,
[KVM_RISCV_ISA_EXT_V] = RISCV_ISA_EXT_v,
/* Multi letter extensions (alphabetically sorted) */
KVM_ISA_EXT_ARR(SMSTATEEN),
KVM_ISA_EXT_ARR(SSAIA),
KVM_ISA_EXT_ARR(SSTC),
KVM_ISA_EXT_ARR(SVINVAL),
@ -45,6 +46,7 @@ static const unsigned long kvm_isa_ext_arr[] = {
KVM_ISA_EXT_ARR(ZICBOM),
KVM_ISA_EXT_ARR(ZICBOZ),
KVM_ISA_EXT_ARR(ZICNTR),
KVM_ISA_EXT_ARR(ZICOND),
KVM_ISA_EXT_ARR(ZICSR),
KVM_ISA_EXT_ARR(ZIFENCEI),
KVM_ISA_EXT_ARR(ZIHINTPAUSE),
@ -80,11 +82,11 @@ static bool kvm_riscv_vcpu_isa_enable_allowed(unsigned long ext)
static bool kvm_riscv_vcpu_isa_disable_allowed(unsigned long ext)
{
switch (ext) {
/* Extensions which don't have any mechanism to disable */
case KVM_RISCV_ISA_EXT_A:
case KVM_RISCV_ISA_EXT_C:
case KVM_RISCV_ISA_EXT_I:
case KVM_RISCV_ISA_EXT_M:
case KVM_RISCV_ISA_EXT_SSAIA:
case KVM_RISCV_ISA_EXT_SSTC:
case KVM_RISCV_ISA_EXT_SVINVAL:
case KVM_RISCV_ISA_EXT_SVNAPOT:
@ -92,11 +94,15 @@ static bool kvm_riscv_vcpu_isa_disable_allowed(unsigned long ext)
case KVM_RISCV_ISA_EXT_ZBB:
case KVM_RISCV_ISA_EXT_ZBS:
case KVM_RISCV_ISA_EXT_ZICNTR:
case KVM_RISCV_ISA_EXT_ZICOND:
case KVM_RISCV_ISA_EXT_ZICSR:
case KVM_RISCV_ISA_EXT_ZIFENCEI:
case KVM_RISCV_ISA_EXT_ZIHINTPAUSE:
case KVM_RISCV_ISA_EXT_ZIHPM:
return false;
/* Extensions which can be disabled using Smstateen */
case KVM_RISCV_ISA_EXT_SSAIA:
return riscv_has_extension_unlikely(RISCV_ISA_EXT_SMSTATEEN);
default:
break;
}
@ -378,6 +384,34 @@ static int kvm_riscv_vcpu_general_set_csr(struct kvm_vcpu *vcpu,
return 0;
}
static inline int kvm_riscv_vcpu_smstateen_set_csr(struct kvm_vcpu *vcpu,
unsigned long reg_num,
unsigned long reg_val)
{
struct kvm_vcpu_smstateen_csr *csr = &vcpu->arch.smstateen_csr;
if (reg_num >= sizeof(struct kvm_riscv_smstateen_csr) /
sizeof(unsigned long))
return -EINVAL;
((unsigned long *)csr)[reg_num] = reg_val;
return 0;
}
static int kvm_riscv_vcpu_smstateen_get_csr(struct kvm_vcpu *vcpu,
unsigned long reg_num,
unsigned long *out_val)
{
struct kvm_vcpu_smstateen_csr *csr = &vcpu->arch.smstateen_csr;
if (reg_num >= sizeof(struct kvm_riscv_smstateen_csr) /
sizeof(unsigned long))
return -EINVAL;
*out_val = ((unsigned long *)csr)[reg_num];
return 0;
}
static int kvm_riscv_vcpu_get_reg_csr(struct kvm_vcpu *vcpu,
const struct kvm_one_reg *reg)
{
@ -401,6 +435,12 @@ static int kvm_riscv_vcpu_get_reg_csr(struct kvm_vcpu *vcpu,
case KVM_REG_RISCV_CSR_AIA:
rc = kvm_riscv_vcpu_aia_get_csr(vcpu, reg_num, &reg_val);
break;
case KVM_REG_RISCV_CSR_SMSTATEEN:
rc = -EINVAL;
if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SMSTATEEN))
rc = kvm_riscv_vcpu_smstateen_get_csr(vcpu, reg_num,
&reg_val);
break;
default:
rc = -ENOENT;
break;
@ -440,6 +480,12 @@ static int kvm_riscv_vcpu_set_reg_csr(struct kvm_vcpu *vcpu,
case KVM_REG_RISCV_CSR_AIA:
rc = kvm_riscv_vcpu_aia_set_csr(vcpu, reg_num, reg_val);
break;
case KVM_REG_RISCV_CSR_SMSTATEEN:
rc = -EINVAL;
if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SMSTATEEN))
rc = kvm_riscv_vcpu_smstateen_set_csr(vcpu, reg_num,
reg_val);
break;
default:
rc = -ENOENT;
break;
@ -696,6 +742,8 @@ static inline unsigned long num_csr_regs(const struct kvm_vcpu *vcpu)
if (riscv_isa_extension_available(vcpu->arch.isa, SSAIA))
n += sizeof(struct kvm_riscv_aia_csr) / sizeof(unsigned long);
if (riscv_isa_extension_available(vcpu->arch.isa, SMSTATEEN))
n += sizeof(struct kvm_riscv_smstateen_csr) / sizeof(unsigned long);
return n;
}
@ -704,7 +752,7 @@ static int copy_csr_reg_indices(const struct kvm_vcpu *vcpu,
u64 __user *uindices)
{
int n1 = sizeof(struct kvm_riscv_csr) / sizeof(unsigned long);
int n2 = 0;
int n2 = 0, n3 = 0;
/* copy general csr regs */
for (int i = 0; i < n1; i++) {
@ -738,7 +786,25 @@ static int copy_csr_reg_indices(const struct kvm_vcpu *vcpu,
}
}
return n1 + n2;
/* copy Smstateen csr regs */
if (riscv_isa_extension_available(vcpu->arch.isa, SMSTATEEN)) {
n3 = sizeof(struct kvm_riscv_smstateen_csr) / sizeof(unsigned long);
for (int i = 0; i < n3; i++) {
u64 size = IS_ENABLED(CONFIG_32BIT) ?
KVM_REG_SIZE_U32 : KVM_REG_SIZE_U64;
u64 reg = KVM_REG_RISCV | size | KVM_REG_RISCV_CSR |
KVM_REG_RISCV_CSR_SMSTATEEN | i;
if (uindices) {
if (put_user(reg, uindices))
return -EFAULT;
uindices++;
}
}
}
return n1 + n2 + n3;
}
static inline unsigned long num_timer_regs(void)

61
arch/riscv/kvm/vcpu_sbi.c
View File

@ -66,6 +66,10 @@ static const struct kvm_riscv_sbi_extension_entry sbi_ext[] = {
.ext_idx = KVM_RISCV_SBI_EXT_PMU,
.ext_ptr = &vcpu_sbi_ext_pmu,
},
{
.ext_idx = KVM_RISCV_SBI_EXT_DBCN,
.ext_ptr = &vcpu_sbi_ext_dbcn,
},
{
.ext_idx = KVM_RISCV_SBI_EXT_EXPERIMENTAL,
.ext_ptr = &vcpu_sbi_ext_experimental,
@ -155,14 +159,8 @@ static int riscv_vcpu_set_sbi_ext_single(struct kvm_vcpu *vcpu,
if (!sext)
return -ENOENT;
/*
* We can't set the extension status to available here, since it may
* have a probe() function which needs to confirm availability first,
* but it may be too early to call that here. We can set the status to
* unavailable, though.
*/
if (!reg_val)
scontext->ext_status[sext->ext_idx] =
scontext->ext_status[sext->ext_idx] = (reg_val) ?
KVM_RISCV_SBI_EXT_AVAILABLE :
KVM_RISCV_SBI_EXT_UNAVAILABLE;
return 0;
@ -188,16 +186,8 @@ static int riscv_vcpu_get_sbi_ext_single(struct kvm_vcpu *vcpu,
if (!sext)
return -ENOENT;
/*
* If the extension status is still uninitialized, then we should probe
* to determine if it's available, but it may be too early to do that
* here. The best we can do is report that the extension has not been
* disabled, i.e. we return 1 when the extension is available and also
* when it only may be available.
*/
*reg_val = scontext->ext_status[sext->ext_idx] !=
KVM_RISCV_SBI_EXT_UNAVAILABLE;
*reg_val = scontext->ext_status[sext->ext_idx] ==
KVM_RISCV_SBI_EXT_AVAILABLE;
return 0;
}
@ -337,18 +327,8 @@ const struct kvm_vcpu_sbi_extension *kvm_vcpu_sbi_find_ext(
scontext->ext_status[entry->ext_idx] ==
KVM_RISCV_SBI_EXT_AVAILABLE)
return ext;
if (scontext->ext_status[entry->ext_idx] ==
KVM_RISCV_SBI_EXT_UNAVAILABLE)
return NULL;
if (ext->probe && !ext->probe(vcpu)) {
scontext->ext_status[entry->ext_idx] =
KVM_RISCV_SBI_EXT_UNAVAILABLE;
return NULL;
}
scontext->ext_status[entry->ext_idx] =
KVM_RISCV_SBI_EXT_AVAILABLE;
return ext;
return NULL;
}
}
@ -419,3 +399,26 @@ ecall_done:
return ret;
}
void kvm_riscv_vcpu_sbi_init(struct kvm_vcpu *vcpu)
{
struct kvm_vcpu_sbi_context *scontext = &vcpu->arch.sbi_context;
const struct kvm_riscv_sbi_extension_entry *entry;
const struct kvm_vcpu_sbi_extension *ext;
int i;
for (i = 0; i < ARRAY_SIZE(sbi_ext); i++) {
entry = &sbi_ext[i];
ext = entry->ext_ptr;
if (ext->probe && !ext->probe(vcpu)) {
scontext->ext_status[entry->ext_idx] =
KVM_RISCV_SBI_EXT_UNAVAILABLE;
continue;
}
scontext->ext_status[entry->ext_idx] = ext->default_unavail ?
KVM_RISCV_SBI_EXT_UNAVAILABLE :
KVM_RISCV_SBI_EXT_AVAILABLE;
}
}

32
arch/riscv/kvm/vcpu_sbi_replace.c
View File

@ -175,3 +175,35 @@ const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_srst = {
.extid_end = SBI_EXT_SRST,
.handler = kvm_sbi_ext_srst_handler,
};
static int kvm_sbi_ext_dbcn_handler(struct kvm_vcpu *vcpu,
struct kvm_run *run,
struct kvm_vcpu_sbi_return *retdata)
{
struct kvm_cpu_context *cp = &vcpu->arch.guest_context;
unsigned long funcid = cp->a6;
switch (funcid) {
case SBI_EXT_DBCN_CONSOLE_WRITE:
case SBI_EXT_DBCN_CONSOLE_READ:
case SBI_EXT_DBCN_CONSOLE_WRITE_BYTE:
/*
* The SBI debug console functions are unconditionally
* forwarded to the userspace.
*/
kvm_riscv_vcpu_sbi_forward(vcpu, run);
retdata->uexit = true;
break;
default:
retdata->err_val = SBI_ERR_NOT_SUPPORTED;
}
return 0;
}
const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_dbcn = {
.extid_start = SBI_EXT_DBCN,
.extid_end = SBI_EXT_DBCN,
.default_unavail = true,
.handler = kvm_sbi_ext_dbcn_handler,
};

7
arch/s390/include/asm/kvm_host.h
View File

@ -777,6 +777,13 @@ struct kvm_vm_stat {
u64 inject_service_signal;
u64 inject_virtio;
u64 aen_forward;
u64 gmap_shadow_create;
u64 gmap_shadow_reuse;
u64 gmap_shadow_r1_entry;
u64 gmap_shadow_r2_entry;
u64 gmap_shadow_r3_entry;
u64 gmap_shadow_sg_entry;
u64 gmap_shadow_pg_entry;
};
struct kvm_arch_memory_slot {

7
arch/s390/kvm/gaccess.c
View File

@ -1382,6 +1382,7 @@ static int kvm_s390_shadow_tables(struct gmap *sg, unsigned long saddr,
unsigned long *pgt, int *dat_protection,
int *fake)
{
struct kvm *kvm;
struct gmap *parent;
union asce asce;
union vaddress vaddr;
@ -1390,6 +1391,7 @@ static int kvm_s390_shadow_tables(struct gmap *sg, unsigned long saddr,
*fake = 0;
*dat_protection = 0;
kvm = sg->private;
parent = sg->parent;
vaddr.addr = saddr;
asce.val = sg->orig_asce;
@ -1450,6 +1452,7 @@ shadow_r2t:
rc = gmap_shadow_r2t(sg, saddr, rfte.val, *fake);
if (rc)
return rc;
kvm->stat.gmap_shadow_r1_entry++;
}
fallthrough;
case ASCE_TYPE_REGION2: {
@ -1478,6 +1481,7 @@ shadow_r3t:
rc = gmap_shadow_r3t(sg, saddr, rste.val, *fake);
if (rc)
return rc;
kvm->stat.gmap_shadow_r2_entry++;
}
fallthrough;
case ASCE_TYPE_REGION3: {
@ -1515,6 +1519,7 @@ shadow_sgt:
rc = gmap_shadow_sgt(sg, saddr, rtte.val, *fake);
if (rc)
return rc;
kvm->stat.gmap_shadow_r3_entry++;
}
fallthrough;
case ASCE_TYPE_SEGMENT: {
@ -1548,6 +1553,7 @@ shadow_pgt:
rc = gmap_shadow_pgt(sg, saddr, ste.val, *fake);
if (rc)
return rc;
kvm->stat.gmap_shadow_sg_entry++;
}
}
/* Return the parent address of the page table */
@ -1618,6 +1624,7 @@ shadow_page:
pte.p |= dat_protection;
if (!rc)
rc = gmap_shadow_page(sg, saddr, __pte(pte.val));
vcpu->kvm->stat.gmap_shadow_pg_entry++;
ipte_unlock(vcpu->kvm);
mmap_read_unlock(sg->mm);
return rc;

11
arch/s390/kvm/kvm-s390.c
View File

@ -66,7 +66,14 @@ const struct _kvm_stats_desc kvm_vm_stats_desc[] = {
STATS_DESC_COUNTER(VM, inject_pfault_done),
STATS_DESC_COUNTER(VM, inject_service_signal),
STATS_DESC_COUNTER(VM, inject_virtio),
STATS_DESC_COUNTER(VM, aen_forward)
STATS_DESC_COUNTER(VM, aen_forward),
STATS_DESC_COUNTER(VM, gmap_shadow_reuse),
STATS_DESC_COUNTER(VM, gmap_shadow_create),
STATS_DESC_COUNTER(VM, gmap_shadow_r1_entry),
STATS_DESC_COUNTER(VM, gmap_shadow_r2_entry),
STATS_DESC_COUNTER(VM, gmap_shadow_r3_entry),
STATS_DESC_COUNTER(VM, gmap_shadow_sg_entry),
STATS_DESC_COUNTER(VM, gmap_shadow_pg_entry),
};
const struct kvm_stats_header kvm_vm_stats_header = {
@ -4053,6 +4060,8 @@ static void kvm_gmap_notifier(struct gmap *gmap, unsigned long start,
unsigned long prefix;
unsigned long i;
trace_kvm_s390_gmap_notifier(start, end, gmap_is_shadow(gmap));
if (gmap_is_shadow(gmap))
return;
if (start >= 1UL << 31)

23
arch/s390/kvm/trace-s390.h
View File

@ -333,6 +333,29 @@ TRACE_EVENT(kvm_s390_airq_suppressed,
__entry->id, __entry->isc)
);
/*
* Trace point for gmap notifier calls.
*/
TRACE_EVENT(kvm_s390_gmap_notifier,
TP_PROTO(unsigned long start, unsigned long end, unsigned int shadow),
TP_ARGS(start, end, shadow),
TP_STRUCT__entry(
__field(unsigned long, start)
__field(unsigned long, end)
__field(unsigned int, shadow)
),
TP_fast_assign(
__entry->start = start;
__entry->end = end;
__entry->shadow = shadow;
),
TP_printk("gmap notified (start:0x%lx end:0x%lx shadow:%d)",
__entry->start, __entry->end, __entry->shadow)
);
#endif /* _TRACE_KVMS390_H */

5
arch/s390/kvm/vsie.c
View File

@ -1214,8 +1214,10 @@ static int acquire_gmap_shadow(struct kvm_vcpu *vcpu,
* we're holding has been unshadowed. If the gmap is still valid,
* we can safely reuse it.
*/
if (vsie_page->gmap && gmap_shadow_valid(vsie_page->gmap, asce, edat))
if (vsie_page->gmap && gmap_shadow_valid(vsie_page->gmap, asce, edat)) {
vcpu->kvm->stat.gmap_shadow_reuse++;
return 0;
}
/* release the old shadow - if any, and mark the prefix as unmapped */
release_gmap_shadow(vsie_page);
@ -1223,6 +1225,7 @@ static int acquire_gmap_shadow(struct kvm_vcpu *vcpu,
if (IS_ERR(gmap))
return PTR_ERR(gmap);
gmap->private = vcpu->kvm;
vcpu->kvm->stat.gmap_shadow_create++;
WRITE_ONCE(vsie_page->gmap, gmap);
return 0;
}

1
arch/x86/include/asm/cpufeatures.h
View File

@ -443,6 +443,7 @@
/* AMD-defined Extended Feature 2 EAX, CPUID level 0x80000021 (EAX), word 20 */
#define X86_FEATURE_NO_NESTED_DATA_BP (20*32+ 0) /* "" No Nested Data Breakpoints */
#define X86_FEATURE_WRMSR_XX_BASE_NS (20*32+ 1) /* "" WRMSR to {FS,GS,KERNEL_GS}_BASE is non-serializing */
#define X86_FEATURE_LFENCE_RDTSC (20*32+ 2) /* "" LFENCE always serializing / synchronizes RDTSC */
#define X86_FEATURE_NULL_SEL_CLR_BASE (20*32+ 6) /* "" Null Selector Clears Base */
#define X86_FEATURE_AUTOIBRS (20*32+ 8) /* "" Automatic IBRS */

3
arch/x86/include/asm/kvm-x86-ops.h
View File

@ -108,6 +108,7 @@ KVM_X86_OP_OPTIONAL(vcpu_blocking)
KVM_X86_OP_OPTIONAL(vcpu_unblocking)
KVM_X86_OP_OPTIONAL(pi_update_irte)
KVM_X86_OP_OPTIONAL(pi_start_assignment)
KVM_X86_OP_OPTIONAL(apicv_pre_state_restore)
KVM_X86_OP_OPTIONAL(apicv_post_state_restore)
KVM_X86_OP_OPTIONAL_RET0(dy_apicv_has_pending_interrupt)
KVM_X86_OP_OPTIONAL(set_hv_timer)
@ -126,7 +127,7 @@ KVM_X86_OP_OPTIONAL(vm_copy_enc_context_from)
KVM_X86_OP_OPTIONAL(vm_move_enc_context_from)
KVM_X86_OP_OPTIONAL(guest_memory_reclaimed)
KVM_X86_OP(get_msr_feature)
KVM_X86_OP(can_emulate_instruction)
KVM_X86_OP(check_emulate_instruction)
KVM_X86_OP(apic_init_signal_blocked)
KVM_X86_OP_OPTIONAL(enable_l2_tlb_flush)
KVM_X86_OP_OPTIONAL(migrate_timers)

22
arch/x86/include/asm/kvm_host.h
View File

@ -39,7 +39,15 @@
#define __KVM_HAVE_ARCH_VCPU_DEBUGFS
/*
* CONFIG_KVM_MAX_NR_VCPUS is defined iff CONFIG_KVM!=n, provide a dummy max if
* KVM is disabled (arbitrarily use the default from CONFIG_KVM_MAX_NR_VCPUS).
*/
#ifdef CONFIG_KVM_MAX_NR_VCPUS
#define KVM_MAX_VCPUS CONFIG_KVM_MAX_NR_VCPUS
#else
#define KVM_MAX_VCPUS 1024
#endif
/*
* In x86, the VCPU ID corresponds to the APIC ID, and APIC IDs
@ -679,6 +687,7 @@ struct kvm_hypervisor_cpuid {
u32 limit;
};
#ifdef CONFIG_KVM_XEN
/* Xen HVM per vcpu emulation context */
struct kvm_vcpu_xen {
u64 hypercall_rip;
@ -701,6 +710,7 @@ struct kvm_vcpu_xen {
struct timer_list poll_timer;
struct kvm_hypervisor_cpuid cpuid;
};
#endif
struct kvm_queued_exception {
bool pending;
@ -929,8 +939,9 @@ struct kvm_vcpu_arch {
bool hyperv_enabled;
struct kvm_vcpu_hv *hyperv;
#ifdef CONFIG_KVM_XEN
struct kvm_vcpu_xen xen;
#endif
cpumask_var_t wbinvd_dirty_mask;
unsigned long last_retry_eip;
@ -1275,7 +1286,6 @@ struct kvm_arch {
*/
spinlock_t mmu_unsync_pages_lock;
struct list_head assigned_dev_head;
struct iommu_domain *iommu_domain;
bool iommu_noncoherent;
#define __KVM_HAVE_ARCH_NONCOHERENT_DMA
@ -1323,6 +1333,7 @@ struct kvm_arch {
int nr_vcpus_matched_tsc;
u32 default_tsc_khz;
bool user_set_tsc;
seqcount_raw_spinlock_t pvclock_sc;
bool use_master_clock;
@ -1691,7 +1702,7 @@ struct kvm_x86_ops {
void (*request_immediate_exit)(struct kvm_vcpu *vcpu);
void (*sched_in)(struct kvm_vcpu *kvm, int cpu);
void (*sched_in)(struct kvm_vcpu *vcpu, int cpu);
/*
* Size of the CPU's dirty log buffer, i.e. VMX's PML buffer. A zero
@ -1708,6 +1719,7 @@ struct kvm_x86_ops {
int (*pi_update_irte)(struct kvm *kvm, unsigned int host_irq,
uint32_t guest_irq, bool set);
void (*pi_start_assignment)(struct kvm *kvm);
void (*apicv_pre_state_restore)(struct kvm_vcpu *vcpu);
void (*apicv_post_state_restore)(struct kvm_vcpu *vcpu);
bool (*dy_apicv_has_pending_interrupt)(struct kvm_vcpu *vcpu);
@ -1733,8 +1745,8 @@ struct kvm_x86_ops {
int (*get_msr_feature)(struct kvm_msr_entry *entry);
bool (*can_emulate_instruction)(struct kvm_vcpu *vcpu, int emul_type,
void *insn, int insn_len);
int (*check_emulate_instruction)(struct kvm_vcpu *vcpu, int emul_type,
void *insn, int insn_len);
bool (*apic_init_signal_blocked)(struct kvm_vcpu *vcpu);
int (*enable_l2_tlb_flush)(struct kvm_vcpu *vcpu);

1
arch/x86/include/asm/msr-index.h
View File

@ -554,6 +554,7 @@
#define MSR_AMD64_CPUID_FN_1 0xc0011004
#define MSR_AMD64_LS_CFG 0xc0011020
#define MSR_AMD64_DC_CFG 0xc0011022
#define MSR_AMD64_TW_CFG 0xc0011023
#define MSR_AMD64_DE_CFG 0xc0011029
#define MSR_AMD64_DE_CFG_LFENCE_SERIALIZE_BIT 1

11
arch/x86/kvm/Kconfig
View File

@ -154,4 +154,15 @@ config KVM_PROVE_MMU
config KVM_EXTERNAL_WRITE_TRACKING
bool
config KVM_MAX_NR_VCPUS
int "Maximum number of vCPUs per KVM guest"
depends on KVM
range 1024 4096
default 4096 if MAXSMP
default 1024
help
Set the maximum number of vCPUs per KVM guest. Larger values will increase
the memory footprint of each KVM guest, regardless of how many vCPUs are
created for a given VM.
endif # VIRTUALIZATION

10
arch/x86/kvm/cpuid.c
View File

@ -448,7 +448,9 @@ static int kvm_set_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid_entry2 *e2,
vcpu->arch.cpuid_nent = nent;
vcpu->arch.kvm_cpuid = kvm_get_hypervisor_cpuid(vcpu, KVM_SIGNATURE);
#ifdef CONFIG_KVM_XEN
vcpu->arch.xen.cpuid = kvm_get_hypervisor_cpuid(vcpu, XEN_SIGNATURE);
#endif
kvm_vcpu_after_set_cpuid(vcpu);
return 0;
@ -753,11 +755,13 @@ void kvm_set_cpu_caps(void)
kvm_cpu_cap_mask(CPUID_8000_0021_EAX,
F(NO_NESTED_DATA_BP) | F(LFENCE_RDTSC) | 0 /* SmmPgCfgLock */ |
F(NULL_SEL_CLR_BASE) | F(AUTOIBRS) | 0 /* PrefetchCtlMsr */
F(NULL_SEL_CLR_BASE) | F(AUTOIBRS) | 0 /* PrefetchCtlMsr */ |
F(WRMSR_XX_BASE_NS)
);
if (cpu_feature_enabled(X86_FEATURE_SRSO_NO))
kvm_cpu_cap_set(X86_FEATURE_SRSO_NO);
kvm_cpu_cap_check_and_set(X86_FEATURE_SBPB);
kvm_cpu_cap_check_and_set(X86_FEATURE_IBPB_BRTYPE);
kvm_cpu_cap_check_and_set(X86_FEATURE_SRSO_NO);
kvm_cpu_cap_init_kvm_defined(CPUID_8000_0022_EAX,
F(PERFMON_V2)

3
arch/x86/kvm/cpuid.h
View File

@ -174,7 +174,8 @@ static inline bool guest_has_spec_ctrl_msr(struct kvm_vcpu *vcpu)
static inline bool guest_has_pred_cmd_msr(struct kvm_vcpu *vcpu)
{
return (guest_cpuid_has(vcpu, X86_FEATURE_SPEC_CTRL) ||
guest_cpuid_has(vcpu, X86_FEATURE_AMD_IBPB));
guest_cpuid_has(vcpu, X86_FEATURE_AMD_IBPB) ||
guest_cpuid_has(vcpu, X86_FEATURE_SBPB));
}
static inline bool supports_cpuid_fault(struct kvm_vcpu *vcpu)

10
arch/x86/kvm/hyperv.c
View File

@ -727,10 +727,12 @@ static int stimer_set_count(struct kvm_vcpu_hv_stimer *stimer, u64 count,
stimer_cleanup(stimer);
stimer->count = count;
if (stimer->count == 0)
stimer->config.enable = 0;
else if (stimer->config.auto_enable)
stimer->config.enable = 1;
if (!host) {
if (stimer->count == 0)
stimer->config.enable = 0;
else if (stimer->config.auto_enable)
stimer->config.enable = 1;
}
if (stimer->config.enable)
stimer_mark_pending(stimer, false);

30
arch/x86/kvm/lapic.c
View File

@ -2444,22 +2444,22 @@ EXPORT_SYMBOL_GPL(kvm_lapic_set_eoi);
void kvm_apic_write_nodecode(struct kvm_vcpu *vcpu, u32 offset)
{
struct kvm_lapic *apic = vcpu->arch.apic;
u64 val;
/*
* ICR is a single 64-bit register when x2APIC is enabled. For legacy
* xAPIC, ICR writes need to go down the common (slightly slower) path
* to get the upper half from ICR2.
* ICR is a single 64-bit register when x2APIC is enabled, all others
* registers hold 32-bit values. For legacy xAPIC, ICR writes need to
* go down the common path to get the upper half from ICR2.
*
* Note, using the write helpers may incur an unnecessary write to the
* virtual APIC state, but KVM needs to conditionally modify the value
* in certain cases, e.g. to clear the ICR busy bit. The cost of extra
* conditional branches is likely a wash relative to the cost of the
* maybe-unecessary write, and both are in the noise anyways.
*/
if (apic_x2apic_mode(apic) && offset == APIC_ICR) {
val = kvm_lapic_get_reg64(apic, APIC_ICR);
kvm_apic_send_ipi(apic, (u32)val, (u32)(val >> 32));
trace_kvm_apic_write(APIC_ICR, val);
} else {
/* TODO: optimize to just emulate side effect w/o one more write */
val = kvm_lapic_get_reg(apic, offset);
kvm_lapic_reg_write(apic, offset, (u32)val);
}
if (apic_x2apic_mode(apic) && offset == APIC_ICR)
kvm_x2apic_icr_write(apic, kvm_lapic_get_reg64(apic, APIC_ICR));
else
kvm_lapic_reg_write(apic, offset, kvm_lapic_get_reg(apic, offset));
}
EXPORT_SYMBOL_GPL(kvm_apic_write_nodecode);
@ -2670,6 +2670,8 @@ void kvm_lapic_reset(struct kvm_vcpu *vcpu, bool init_event)
u64 msr_val;
int i;
static_call_cond(kvm_x86_apicv_pre_state_restore)(vcpu);
if (!init_event) {
msr_val = APIC_DEFAULT_PHYS_BASE | MSR_IA32_APICBASE_ENABLE;
if (kvm_vcpu_is_reset_bsp(vcpu))
@ -2981,6 +2983,8 @@ int kvm_apic_set_state(struct kvm_vcpu *vcpu, struct kvm_lapic_state *s)
struct kvm_lapic *apic = vcpu->arch.apic;
int r;
static_call_cond(kvm_x86_apicv_pre_state_restore)(vcpu);
kvm_lapic_set_base(vcpu, vcpu->arch.apic_base);
/* set SPIV separately to get count of SW disabled APICs right */
apic_set_spiv(apic, *((u32 *)(s->regs + APIC_SPIV)));

7
arch/x86/kvm/mmu.h
View File

@ -237,6 +237,13 @@ static inline u8 permission_fault(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,
return -(u32)fault & errcode;
}
bool __kvm_mmu_honors_guest_mtrrs(bool vm_has_noncoherent_dma);
static inline bool kvm_mmu_honors_guest_mtrrs(struct kvm *kvm)
{
return __kvm_mmu_honors_guest_mtrrs(kvm_arch_has_noncoherent_dma(kvm));
}
void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end);
int kvm_arch_write_log_dirty(struct kvm_vcpu *vcpu);

37
arch/x86/kvm/mmu/mmu.c
View File

@ -3425,8 +3425,8 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
{
struct kvm_mmu_page *sp;
int ret = RET_PF_INVALID;
u64 spte = 0ull;
u64 *sptep = NULL;
u64 spte;
u64 *sptep;
uint retry_count = 0;
if (!page_fault_can_be_fast(fault))
@ -3442,6 +3442,14 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
else
sptep = fast_pf_get_last_sptep(vcpu, fault->addr, &spte);
/*
* It's entirely possible for the mapping to have been zapped
* by a different task, but the root page should always be
* available as the vCPU holds a reference to its root(s).
*/
if (WARN_ON_ONCE(!sptep))
spte = REMOVED_SPTE;
if (!is_shadow_present_pte(spte))
break;
@ -4479,21 +4487,28 @@ out_unlock:
}
#endif
bool __kvm_mmu_honors_guest_mtrrs(bool vm_has_noncoherent_dma)
{
/*
* If host MTRRs are ignored (shadow_memtype_mask is non-zero), and the
* VM has non-coherent DMA (DMA doesn't snoop CPU caches), KVM's ABI is
* to honor the memtype from the guest's MTRRs so that guest accesses
* to memory that is DMA'd aren't cached against the guest's wishes.
*
* Note, KVM may still ultimately ignore guest MTRRs for certain PFNs,
* e.g. KVM will force UC memtype for host MMIO.
*/
return vm_has_noncoherent_dma && shadow_memtype_mask;
}
int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
{
/*
* If the guest's MTRRs may be used to compute the "real" memtype,
* restrict the mapping level to ensure KVM uses a consistent memtype
* across the entire mapping. If the host MTRRs are ignored by TDP
* (shadow_memtype_mask is non-zero), and the VM has non-coherent DMA
* (DMA doesn't snoop CPU caches), KVM's ABI is to honor the memtype
* from the guest's MTRRs so that guest accesses to memory that is
* DMA'd aren't cached against the guest's wishes.
*
* Note, KVM may still ultimately ignore guest MTRRs for certain PFNs,
* e.g. KVM will force UC memtype for host MMIO.
* across the entire mapping.
*/
if (shadow_memtype_mask && kvm_arch_has_noncoherent_dma(vcpu->kvm)) {
if (kvm_mmu_honors_guest_mtrrs(vcpu->kvm)) {
for ( ; fault->max_level > PG_LEVEL_4K; --fault->max_level) {
int page_num = KVM_PAGES_PER_HPAGE(fault->max_level);
gfn_t base = gfn_round_for_level(fault->gfn,

2
arch/x86/kvm/mtrr.c
View File

@ -320,7 +320,7 @@ static void update_mtrr(struct kvm_vcpu *vcpu, u32 msr)
struct kvm_mtrr *mtrr_state = &vcpu->arch.mtrr_state;
gfn_t start, end;
if (!tdp_enabled || !kvm_arch_has_noncoherent_dma(vcpu->kvm))
if (!kvm_mmu_honors_guest_mtrrs(vcpu->kvm))
return;
if (!mtrr_is_enabled(mtrr_state) && msr != MSR_MTRRdefType)

1
arch/x86/kvm/smm.c
View File

@ -324,7 +324,6 @@ void enter_smm(struct kvm_vcpu *vcpu)
cr0 = vcpu->arch.cr0 & ~(X86_CR0_PE | X86_CR0_EM | X86_CR0_TS | X86_CR0_PG);
static_call(kvm_x86_set_cr0)(vcpu, cr0);
vcpu->arch.cr0 = cr0;
static_call(kvm_x86_set_cr4)(vcpu, 0);

52
arch/x86/kvm/svm/svm.c
View File

@ -199,7 +199,7 @@ module_param_named(npt, npt_enabled, bool, 0444);
/* allow nested virtualization in KVM/SVM */
static int nested = true;
module_param(nested, int, S_IRUGO);
module_param(nested, int, 0444);
/* enable/disable Next RIP Save */
int nrips = true;
@ -364,8 +364,6 @@ static void svm_set_interrupt_shadow(struct kvm_vcpu *vcpu, int mask)
svm->vmcb->control.int_state |= SVM_INTERRUPT_SHADOW_MASK;
}
static bool svm_can_emulate_instruction(struct kvm_vcpu *vcpu, int emul_type,
void *insn, int insn_len);
static int __svm_skip_emulated_instruction(struct kvm_vcpu *vcpu,
bool commit_side_effects)
@ -386,14 +384,6 @@ static int __svm_skip_emulated_instruction(struct kvm_vcpu *vcpu,
}
if (!svm->next_rip) {
/*
* FIXME: Drop this when kvm_emulate_instruction() does the
* right thing and treats "can't emulate" as outright failure
* for EMULTYPE_SKIP.
*/
if (!svm_can_emulate_instruction(vcpu, EMULTYPE_SKIP, NULL, 0))
return 0;
if (unlikely(!commit_side_effects))
old_rflags = svm->vmcb->save.rflags;
@ -2194,12 +2184,6 @@ static int shutdown_interception(struct kvm_vcpu *vcpu)
struct kvm_run *kvm_run = vcpu->run;
struct vcpu_svm *svm = to_svm(vcpu);
/*
* The VM save area has already been encrypted so it
* cannot be reinitialized - just terminate.
*/
if (sev_es_guest(vcpu->kvm))
return -EINVAL;
/*
* VMCB is undefined after a SHUTDOWN intercept. INIT the vCPU to put
@ -2208,9 +2192,14 @@ static int shutdown_interception(struct kvm_vcpu *vcpu)
* userspace. At a platform view, INIT is acceptable behavior as
* there exist bare metal platforms that automatically INIT the CPU
* in response to shutdown.
*
* The VM save area for SEV-ES guests has already been encrypted so it
* cannot be reinitialized, i.e. synthesizing INIT is futile.
*/
clear_page(svm->vmcb);
kvm_vcpu_reset(vcpu, true);
if (!sev_es_guest(vcpu->kvm)) {
clear_page(svm->vmcb);
kvm_vcpu_reset(vcpu, true);
}
kvm_run->exit_reason = KVM_EXIT_SHUTDOWN;
return 0;
@ -4719,15 +4708,15 @@ static void svm_enable_smi_window(struct kvm_vcpu *vcpu)
}
#endif
static bool svm_can_emulate_instruction(struct kvm_vcpu *vcpu, int emul_type,
void *insn, int insn_len)
static int svm_check_emulate_instruction(struct kvm_vcpu *vcpu, int emul_type,
void *insn, int insn_len)
{
bool smep, smap, is_user;
u64 error_code;
/* Emulation is always possible when KVM has access to all guest state. */
if (!sev_guest(vcpu->kvm))
return true;
return X86EMUL_CONTINUE;
/* #UD and #GP should never be intercepted for SEV guests. */
WARN_ON_ONCE(emul_type & (EMULTYPE_TRAP_UD |
@ -4739,14 +4728,14 @@ static bool svm_can_emulate_instruction(struct kvm_vcpu *vcpu, int emul_type,
* to guest register state.
*/
if (sev_es_guest(vcpu->kvm))
return false;
return X86EMUL_RETRY_INSTR;
/*
* Emulation is possible if the instruction is already decoded, e.g.
* when completing I/O after returning from userspace.
*/
if (emul_type & EMULTYPE_NO_DECODE)
return true;
return X86EMUL_CONTINUE;
/*
* Emulation is possible for SEV guests if and only if a prefilled
@ -4772,9 +4761,11 @@ static bool svm_can_emulate_instruction(struct kvm_vcpu *vcpu, int emul_type,
* success (and in practice it will work the vast majority of the time).
*/
if (unlikely(!insn)) {
if (!(emul_type & EMULTYPE_SKIP))
kvm_queue_exception(vcpu, UD_VECTOR);
return false;
if (emul_type & EMULTYPE_SKIP)
return X86EMUL_UNHANDLEABLE;
kvm_queue_exception(vcpu, UD_VECTOR);
return X86EMUL_PROPAGATE_FAULT;
}
/*
@ -4785,7 +4776,7 @@ static bool svm_can_emulate_instruction(struct kvm_vcpu *vcpu, int emul_type,
* table used to translate CS:RIP resides in emulated MMIO.
*/
if (likely(insn_len))
return true;
return X86EMUL_CONTINUE;
/*
* Detect and workaround Errata 1096 Fam_17h_00_0Fh.
@ -4843,6 +4834,7 @@ static bool svm_can_emulate_instruction(struct kvm_vcpu *vcpu, int emul_type,
kvm_inject_gp(vcpu, 0);
else
kvm_make_request(KVM_REQ_TRIPLE_FAULT, vcpu);
return X86EMUL_PROPAGATE_FAULT;
}
resume_guest:
@ -4860,7 +4852,7 @@ resume_guest:
* doesn't explicitly define "ignored", i.e. doing nothing and letting
* the guest spin is technically "ignoring" the access.
*/
return false;
return X86EMUL_RETRY_INSTR;
}
static bool svm_apic_init_signal_blocked(struct kvm_vcpu *vcpu)
@ -5020,7 +5012,7 @@ static struct kvm_x86_ops svm_x86_ops __initdata = {
.vm_copy_enc_context_from = sev_vm_copy_enc_context_from,
.vm_move_enc_context_from = sev_vm_move_enc_context_from,
.can_emulate_instruction = svm_can_emulate_instruction,
.check_emulate_instruction = svm_check_emulate_instruction,
.apic_init_signal_blocked = svm_apic_init_signal_blocked,

Some files were not shown because too many files have changed in this diff Show More