2019-06-03 05:44:50 +00:00
|
|
|
/* SPDX-License-Identifier: GPL-2.0-only */
|
2012-03-05 11:49:27 +00:00
|
|
|
/*
|
|
|
|
|
* Copyright (C) 2012 ARM Ltd.
|
|
|
|
|
*/
|
|
|
|
|
#ifndef __ASM_MMU_H
|
|
|
|
|
#define __ASM_MMU_H
|
|
|
|
|
|
2019-01-08 16:19:01 +00:00
|
|
|
#include <asm/cputype.h>
|
|
|
|
|
|
2017-08-20 10:20:47 +00:00
|
|
|
#define MMCF_AARCH32 0x1 /* mm context flag for AArch32 executables */
|
2018-01-08 15:38:18 +00:00
|
|
|
#define USER_ASID_BIT 48
|
|
|
|
|
#define USER_ASID_FLAG (UL(1) << USER_ASID_BIT)
|
2017-12-01 17:33:48 +00:00
|
|
|
#define TTBR_ASID_MASK (UL(0xffff) << 48)
|
2017-08-20 10:20:47 +00:00
|
|
|
|
2017-11-14 13:58:08 +00:00
|
|
|
#ifndef __ASSEMBLY__
|
|
|
|
|
|
arm64: mm: Pin down ASIDs for sharing mm with devices
To enable address space sharing with the IOMMU, introduce
arm64_mm_context_get() and arm64_mm_context_put(), that pin down a
context and ensure that it will keep its ASID after a rollover. Export
the symbols to let the modular SMMUv3 driver use them.
Pinning is necessary because a device constantly needs a valid ASID,
unlike tasks that only require one when running. Without pinning, we would
need to notify the IOMMU when we're about to use a new ASID for a task,
and it would get complicated when a new task is assigned a shared ASID.
Consider the following scenario with no ASID pinned:
1. Task t1 is running on CPUx with shared ASID (gen=1, asid=1)
2. Task t2 is scheduled on CPUx, gets ASID (1, 2)
3. Task tn is scheduled on CPUy, a rollover occurs, tn gets ASID (2, 1)
We would now have to immediately generate a new ASID for t1, notify
the IOMMU, and finally enable task tn. We are holding the lock during
all that time, since we can't afford having another CPU trigger a
rollover. The IOMMU issues invalidation commands that can take tens of
milliseconds.
It gets needlessly complicated. All we wanted to do was schedule task tn,
that has no business with the IOMMU. By letting the IOMMU pin tasks when
needed, we avoid stalling the slow path, and let the pinning fail when
we're out of shareable ASIDs.
After a rollover, the allocator expects at least one ASID to be available
in addition to the reserved ones (one per CPU). So (NR_ASIDS - NR_CPUS -
1) is the maximum number of ASIDs that can be shared with the IOMMU.
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Link: https://lore.kernel.org/r/20200918101852.582559-5-jean-philippe@linaro.org
Signed-off-by: Will Deacon <will@kernel.org>
2020-09-18 10:18:44 +00:00
|
|
|
#include <linux/refcount.h>
|
2021-12-02 17:10:46 +00:00
|
|
|
#include <asm/cpufeature.h>
|
arm64: mm: Pin down ASIDs for sharing mm with devices
To enable address space sharing with the IOMMU, introduce
arm64_mm_context_get() and arm64_mm_context_put(), that pin down a
context and ensure that it will keep its ASID after a rollover. Export
the symbols to let the modular SMMUv3 driver use them.
Pinning is necessary because a device constantly needs a valid ASID,
unlike tasks that only require one when running. Without pinning, we would
need to notify the IOMMU when we're about to use a new ASID for a task,
and it would get complicated when a new task is assigned a shared ASID.
Consider the following scenario with no ASID pinned:
1. Task t1 is running on CPUx with shared ASID (gen=1, asid=1)
2. Task t2 is scheduled on CPUx, gets ASID (1, 2)
3. Task tn is scheduled on CPUy, a rollover occurs, tn gets ASID (2, 1)
We would now have to immediately generate a new ASID for t1, notify
the IOMMU, and finally enable task tn. We are holding the lock during
all that time, since we can't afford having another CPU trigger a
rollover. The IOMMU issues invalidation commands that can take tens of
milliseconds.
It gets needlessly complicated. All we wanted to do was schedule task tn,
that has no business with the IOMMU. By letting the IOMMU pin tasks when
needed, we avoid stalling the slow path, and let the pinning fail when
we're out of shareable ASIDs.
After a rollover, the allocator expects at least one ASID to be available
in addition to the reserved ones (one per CPU). So (NR_ASIDS - NR_CPUS -
1) is the maximum number of ASIDs that can be shared with the IOMMU.
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Link: https://lore.kernel.org/r/20200918101852.582559-5-jean-philippe@linaro.org
Signed-off-by: Will Deacon <will@kernel.org>
2020-09-18 10:18:44 +00:00
|
|
|
|
2012-03-05 11:49:27 +00:00
|
|
|
typedef struct {
|
2015-10-06 17:46:24 +00:00
|
|
|
atomic64_t id;
|
2020-06-22 11:35:41 +00:00
|
|
|
#ifdef CONFIG_COMPAT
|
|
|
|
|
void *sigpage;
|
|
|
|
|
#endif
|
arm64: mm: Pin down ASIDs for sharing mm with devices
To enable address space sharing with the IOMMU, introduce
arm64_mm_context_get() and arm64_mm_context_put(), that pin down a
context and ensure that it will keep its ASID after a rollover. Export
the symbols to let the modular SMMUv3 driver use them.
Pinning is necessary because a device constantly needs a valid ASID,
unlike tasks that only require one when running. Without pinning, we would
need to notify the IOMMU when we're about to use a new ASID for a task,
and it would get complicated when a new task is assigned a shared ASID.
Consider the following scenario with no ASID pinned:
1. Task t1 is running on CPUx with shared ASID (gen=1, asid=1)
2. Task t2 is scheduled on CPUx, gets ASID (1, 2)
3. Task tn is scheduled on CPUy, a rollover occurs, tn gets ASID (2, 1)
We would now have to immediately generate a new ASID for t1, notify
the IOMMU, and finally enable task tn. We are holding the lock during
all that time, since we can't afford having another CPU trigger a
rollover. The IOMMU issues invalidation commands that can take tens of
milliseconds.
It gets needlessly complicated. All we wanted to do was schedule task tn,
that has no business with the IOMMU. By letting the IOMMU pin tasks when
needed, we avoid stalling the slow path, and let the pinning fail when
we're out of shareable ASIDs.
After a rollover, the allocator expects at least one ASID to be available
in addition to the reserved ones (one per CPU). So (NR_ASIDS - NR_CPUS -
1) is the maximum number of ASIDs that can be shared with the IOMMU.
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Link: https://lore.kernel.org/r/20200918101852.582559-5-jean-philippe@linaro.org
Signed-off-by: Will Deacon <will@kernel.org>
2020-09-18 10:18:44 +00:00
|
|
|
refcount_t pinned;
|
2015-10-06 17:46:24 +00:00
|
|
|
void *vdso;
|
2016-11-02 09:10:45 +00:00
|
|
|
unsigned long flags;
|
2012-03-05 11:49:27 +00:00
|
|
|
} mm_context_t;
|
|
|
|
|
|
2015-10-06 17:46:24 +00:00
|
|
|
/*
|
2021-08-06 11:31:04 +00:00
|
|
|
* We use atomic64_read() here because the ASID for an 'mm_struct' can
|
|
|
|
|
* be reallocated when scheduling one of its threads following a
|
|
|
|
|
* rollover event (see new_context() and flush_context()). In this case,
|
|
|
|
|
* a concurrent TLBI (e.g. via try_to_unmap_one() and ptep_clear_flush())
|
|
|
|
|
* may use a stale ASID. This is fine in principle as the new ASID is
|
|
|
|
|
* guaranteed to be clean in the TLB, but the TLBI routines have to take
|
|
|
|
|
* care to handle the following race:
|
|
|
|
|
*
|
|
|
|
|
* CPU 0 CPU 1 CPU 2
|
|
|
|
|
*
|
|
|
|
|
* // ptep_clear_flush(mm)
|
|
|
|
|
* xchg_relaxed(pte, 0)
|
|
|
|
|
* DSB ISHST
|
|
|
|
|
* old = ASID(mm)
|
|
|
|
|
* | <rollover>
|
|
|
|
|
* | new = new_context(mm)
|
|
|
|
|
* \-----------------> atomic_set(mm->context.id, new)
|
|
|
|
|
* cpu_switch_mm(mm)
|
|
|
|
|
* // Hardware walk of pte using new ASID
|
|
|
|
|
* TLBI(old)
|
|
|
|
|
*
|
|
|
|
|
* In this scenario, the barrier on CPU 0 and the dependency on CPU 1
|
|
|
|
|
* ensure that the page-table walker on CPU 1 *must* see the invalid PTE
|
|
|
|
|
* written by CPU 0.
|
2015-10-06 17:46:24 +00:00
|
|
|
*/
|
2021-08-06 11:31:04 +00:00
|
|
|
#define ASID(mm) (atomic64_read(&(mm)->context.id) & 0xffff)
|
2012-03-05 11:49:27 +00:00
|
|
|
|
2019-12-09 18:12:17 +00:00
|
|
|
static inline bool arm64_kernel_unmapped_at_el0(void)
|
2019-12-09 18:12:15 +00:00
|
|
|
{
|
arm64: Avoid cpus_have_const_cap() for ARM64_UNMAP_KERNEL_AT_EL0
In arm64_kernel_unmapped_at_el0() we use cpus_have_const_cap() to check
for ARM64_UNMAP_KERNEL_AT_EL0, but this is only necessary so that
arm64_get_bp_hardening_vector() and this_cpu_set_vectors() can run prior
to alternatives being patched. Otherwise this is not necessary and
alternative_has_cap_*() would be preferable.
For historical reasons, cpus_have_const_cap() is more complicated than
it needs to be. Before cpucaps are finalized, it will perform a bitmap
test of the system_cpucaps bitmap, and once cpucaps are finalized it
will use an alternative branch. This used to be necessary to handle some
race conditions in the window between cpucap detection and the
subsequent patching of alternatives and static branches, where different
branches could be out-of-sync with one another (or w.r.t. alternative
sequences). Now that we use alternative branches instead of static
branches, these are all patched atomically w.r.t. one another, and there
are only a handful of cases that need special care in the window between
cpucap detection and alternative patching.
Due to the above, it would be nice to remove cpus_have_const_cap(), and
migrate callers over to alternative_has_cap_*(), cpus_have_final_cap(),
or cpus_have_cap() depending on when their requirements. This will
remove redundant instructions and improve code generation, and will make
it easier to determine how each callsite will behave before, during, and
after alternative patching.
The ARM64_UNMAP_KERNEL_AT_EL0 cpucap is a system-wide feature that is
detected and patched before any translation tables are created for
userspace. In the window between detecting the ARM64_UNMAP_KERNEL_AT_EL0
cpucap and patching alternatives, most users of
arm64_kernel_unmapped_at_el0() do not need to know that the cpucap has
been detected:
* As KVM is initialized after cpucaps are finalized, no usaef of
arm64_kernel_unmapped_at_el0() in the KVM code is reachable during
this window.
* The arm64_mm_context_get() function in arch/arm64/mm/context.c is only
called after the SMMU driver is brought up after alternatives have
been patched. Thus this can safely use cpus_have_final_cap() or
alternative_has_cap_*().
Similarly the asids_update_limit() function is called after
alternatives have been patched as an arch_initcall, and this can
safely use cpus_have_final_cap() or alternative_has_cap_*().
Similarly we do not expect an ASID rollover to occur between cpucaps
being detected and patching alternatives. Thus
set_reserved_asid_bits() can safely use cpus_have_final_cap() or
alternative_has_cap_*().
* The __tlbi_user() and __tlbi_user_level() macros are not used during
this window, and only need to invalidate additional entries once
userspace translation tables have been active on a CPU. Thus these can
safely use alternative_has_cap_*().
* The xen_kernel_unmapped_at_usr() function is not used during this
window as it is only used in a late_initcall. Thus this can safely use
cpus_have_final_cap() or alternative_has_cap_*().
* The arm64_get_meltdown_state() function is not used during this
window. It only used by arm64_get_meltdown_state() and KVM code, both
of which are only used after cpucaps have been finalized. Thus this
can safely use cpus_have_final_cap() or alternative_has_cap_*().
* The tls_thread_switch() uses arm64_kernel_unmapped_at_el0() as an
optimization to avoid zeroing tpidrro_el0 when KPTI is enabled
and this will be trampled by the KPTI trampoline. It doesn't matter if
this continues to zero the register during the window between
detecting the cpucap and patching alternatives, so this can safely use
alternative_has_cap_*().
* The sdei_arch_get_entry_point() and do_sdei_event() functions aren't
reachable at this time as the SDEI driver is registered later by
acpi_init() -> acpi_ghes_init() -> sdei_init(), where acpi_init is a
subsys_initcall. Thus these can safely use cpus_have_final_cap() or
alternative_has_cap_*().
* The uses under drivers/ aren't reachable at this time as the drivers
are registered later:
- TRBE is registered via module_init()
- SMMUv3 is registred via module_driver()
- SPE is registred via module_init()
* The arm64_get_bp_hardening_vector() and this_cpu_set_vectors()
functions need to run on boot CPUs prior to patching alternatives.
As these are only called during the onlining of a CPU, it's fine to
perform a system_cpucaps bitmap test using cpus_have_cap().
This patch modifies this_cpu_set_vectors() to use cpus_have_cap(), and
replaced all other use of cpus_have_const_cap() with
alternative_has_cap_unlikely(), which will avoid generating code to test
the system_cpucaps bitmap and should be better for all subsequent calls
at runtime. The ARM64_UNMAP_KERNEL_AT_EL0 cpucap is added to
cpucap_is_possible() so that code can be elided entirely when this is
not possible.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: James Morse <james.morse@arm.com>
Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2023-10-16 10:24:53 +00:00
|
|
|
return alternative_has_cap_unlikely(ARM64_UNMAP_KERNEL_AT_EL0);
|
2019-01-08 16:19:01 +00:00
|
|
|
}
|
|
|
|
|
|
2019-01-14 14:22:24 +00:00
|
|
|
extern void arm64_memblock_init(void);
|
2012-03-05 11:49:27 +00:00
|
|
|
extern void paging_init(void);
|
2016-04-08 22:50:26 +00:00
|
|
|
extern void bootmem_init(void);
|
2012-10-23 13:55:08 +00:00
|
|
|
extern void __iomem *early_io_map(phys_addr_t phys, unsigned long virt);
|
2023-04-06 15:27:58 +00:00
|
|
|
extern void create_mapping_noalloc(phys_addr_t phys, unsigned long virt,
|
|
|
|
|
phys_addr_t size, pgprot_t prot);
|
2014-10-20 13:42:07 +00:00
|
|
|
extern void create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys,
|
|
|
|
|
unsigned long virt, phys_addr_t size,
|
2016-10-21 11:22:57 +00:00
|
|
|
pgprot_t prot, bool page_mappings_only);
|
2019-08-23 06:24:50 +00:00
|
|
|
extern void *fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot);
|
2017-03-09 20:52:01 +00:00
|
|
|
extern void mark_linear_text_alias_ro(void);
|
2024-02-14 12:29:02 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* This check is triggered during the early boot before the cpufeature
|
|
|
|
|
* is initialised. Checking the status on the local CPU allows the boot
|
|
|
|
|
* CPU to detect the need for non-global mappings and thus avoiding a
|
|
|
|
|
* pagetable re-write after all the CPUs are booted. This check will be
|
|
|
|
|
* anyway run on individual CPUs, allowing us to get the consistent
|
|
|
|
|
* state once the SMP CPUs are up and thus make the switch to non-global
|
|
|
|
|
* mappings if required.
|
|
|
|
|
*/
|
|
|
|
|
static inline bool kaslr_requires_kpti(void)
|
|
|
|
|
{
|
|
|
|
|
/*
|
|
|
|
|
* E0PD does a similar job to KPTI so can be used instead
|
|
|
|
|
* where available.
|
|
|
|
|
*/
|
|
|
|
|
if (IS_ENABLED(CONFIG_ARM64_E0PD)) {
|
|
|
|
|
u64 mmfr2 = read_sysreg_s(SYS_ID_AA64MMFR2_EL1);
|
|
|
|
|
if (cpuid_feature_extract_unsigned_field(mmfr2,
|
|
|
|
|
ID_AA64MMFR2_EL1_E0PD_SHIFT))
|
|
|
|
|
return false;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Systems affected by Cavium erratum 24756 are incompatible
|
|
|
|
|
* with KPTI.
|
|
|
|
|
*/
|
|
|
|
|
if (IS_ENABLED(CONFIG_CAVIUM_ERRATUM_27456)) {
|
|
|
|
|
extern const struct midr_range cavium_erratum_27456_cpus[];
|
|
|
|
|
|
|
|
|
|
if (is_midr_in_range_list(read_cpuid_id(),
|
|
|
|
|
cavium_erratum_27456_cpus))
|
|
|
|
|
return false;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return true;
|
|
|
|
|
}
|
2012-03-05 11:49:27 +00:00
|
|
|
|
arm64/mm: Separate boot-time page tables from swapper_pg_dir
Since the address of swapper_pg_dir is fixed for a given kernel image,
it is an attractive target for manipulation via an arbitrary write. To
mitigate this we'd like to make it read-only by moving it into the
rodata section.
We require that swapper_pg_dir is at a fixed offset from tramp_pg_dir
and reserved_ttbr0, so these will also need to move into rodata.
However, swapper_pg_dir is allocated along with some transient page
tables used for boot which we do not want to move into rodata.
As a step towards this, this patch separates the boot-time page tables
into a new init_pg_dir, and reduces swapper_pg_dir to the single page it
needs to be. This allows us to retain the relationship between
swapper_pg_dir, tramp_pg_dir, and swapper_pg_dir, while cleanly
separating these from the boot-time page tables.
The init_pg_dir holds all of the pgd/pud/pmd/pte levels needed during
boot, and all of these levels will be freed when we switch to the
swapper_pg_dir, which is initialized by the existing code in
paging_init(). Since we start off on the init_pg_dir, we no longer need
to allocate a transient page table in paging_init() in order to ensure
that swapper_pg_dir isn't live while we initialize it.
There should be no functional change as a result of this patch.
Signed-off-by: Jun Yao <yaojun8558363@gmail.com>
Reviewed-by: James Morse <james.morse@arm.com>
[Mark: place init_pg_dir after BSS, fold mm changes, commit message]
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2018-09-24 14:47:49 +00:00
|
|
|
#define INIT_MM_CONTEXT(name) \
|
2024-02-14 12:29:09 +00:00
|
|
|
.pgd = swapper_pg_dir,
|
arm64/mm: Separate boot-time page tables from swapper_pg_dir
Since the address of swapper_pg_dir is fixed for a given kernel image,
it is an attractive target for manipulation via an arbitrary write. To
mitigate this we'd like to make it read-only by moving it into the
rodata section.
We require that swapper_pg_dir is at a fixed offset from tramp_pg_dir
and reserved_ttbr0, so these will also need to move into rodata.
However, swapper_pg_dir is allocated along with some transient page
tables used for boot which we do not want to move into rodata.
As a step towards this, this patch separates the boot-time page tables
into a new init_pg_dir, and reduces swapper_pg_dir to the single page it
needs to be. This allows us to retain the relationship between
swapper_pg_dir, tramp_pg_dir, and swapper_pg_dir, while cleanly
separating these from the boot-time page tables.
The init_pg_dir holds all of the pgd/pud/pmd/pte levels needed during
boot, and all of these levels will be freed when we switch to the
swapper_pg_dir, which is initialized by the existing code in
paging_init(). Since we start off on the init_pg_dir, we no longer need
to allocate a transient page table in paging_init() in order to ensure
that swapper_pg_dir isn't live while we initialize it.
There should be no functional change as a result of this patch.
Signed-off-by: Jun Yao <yaojun8558363@gmail.com>
Reviewed-by: James Morse <james.morse@arm.com>
[Mark: place init_pg_dir after BSS, fold mm changes, commit message]
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2018-09-24 14:47:49 +00:00
|
|
|
|
2017-11-14 13:58:08 +00:00
|
|
|
#endif /* !__ASSEMBLY__ */
|
2012-03-05 11:49:27 +00:00
|
|
|
#endif
|