2020-06-25 13:14:08 +00:00
|
|
|
# SPDX-License-Identifier: GPL-2.0
|
|
|
|
|
#
|
|
|
|
|
# Makefile for Kernel-based Virtual Machine module, HYP/nVHE part
|
|
|
|
|
#
|
|
|
|
|
|
2021-02-03 14:19:31 +00:00
|
|
|
asflags-y := -D__KVM_NVHE_HYPERVISOR__ -D__DISABLE_EXPORTS
|
2022-05-18 16:44:15 +00:00
|
|
|
|
|
|
|
|
# Tracepoint and MMIO logging symbols should not be visible at nVHE KVM as
|
|
|
|
|
# there is no way to execute them and any such MMIO access from nVHE KVM
|
|
|
|
|
# will explode instantly (Words of Marc Zyngier). So introduce a generic flag
|
|
|
|
|
# __DISABLE_TRACE_MMIO__ to disable MMIO tracing for nVHE KVM.
|
|
|
|
|
ccflags-y := -D__KVM_NVHE_HYPERVISOR__ -D__DISABLE_EXPORTS -D__DISABLE_TRACE_MMIO__
|
2022-10-04 15:42:16 +00:00
|
|
|
ccflags-y += -fno-stack-protector \
|
|
|
|
|
-DDISABLE_BRANCH_PROFILING \
|
|
|
|
|
$(DISABLE_STACKLEAK_PLUGIN)
|
2020-06-25 13:14:08 +00:00
|
|
|
|
2021-01-05 18:05:37 +00:00
|
|
|
hostprogs := gen-hyprel
|
2021-01-30 13:07:51 +00:00
|
|
|
HOST_EXTRACFLAGS += -I$(objtree)/include
|
2021-01-05 18:05:37 +00:00
|
|
|
|
2021-03-19 10:01:10 +00:00
|
|
|
lib-objs := clear_page.o copy_page.o memcpy.o memset.o
|
|
|
|
|
lib-objs := $(addprefix ../../../lib/, $(lib-objs))
|
|
|
|
|
|
2022-06-13 09:20:25 +00:00
|
|
|
hyp-obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \
|
2022-01-31 12:40:53 +00:00
|
|
|
hyp-main.o hyp-smp.o psci-relay.o early_alloc.o page_alloc.o \
|
2023-05-23 10:18:18 +00:00
|
|
|
cache.o setup.o mm.o mem_protect.o sys_regs.o pkvm.o stacktrace.o ffa.o
|
2022-06-13 09:20:25 +00:00
|
|
|
hyp-obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \
|
KVM: arm64: Prepare the creation of s1 mappings at EL2
When memory protection is enabled, the EL2 code needs the ability to
create and manage its own page-table. To do so, introduce a new set of
hypercalls to bootstrap a memory management system at EL2.
This leads to the following boot flow in nVHE Protected mode:
1. the host allocates memory for the hypervisor very early on, using
the memblock API;
2. the host creates a set of stage 1 page-table for EL2, installs the
EL2 vectors, and issues the __pkvm_init hypercall;
3. during __pkvm_init, the hypervisor re-creates its stage 1 page-table
and stores it in the memory pool provided by the host;
4. the hypervisor then extends its stage 1 mappings to include a
vmemmap in the EL2 VA space, hence allowing to use the buddy
allocator introduced in a previous patch;
5. the hypervisor jumps back in the idmap page, switches from the
host-provided page-table to the new one, and wraps up its
initialization by enabling the new allocator, before returning to
the host.
6. the host can free the now unused page-table created for EL2, and
will now need to issue hypercalls to make changes to the EL2 stage 1
mappings instead of modifying them directly.
Note that for the sake of simplifying the review, this patch focuses on
the hypervisor side of things. In other words, this only implements the
new hypercalls, but does not make use of them from the host yet. The
host-side changes will follow in a subsequent patch.
Credits to Will for __pkvm_init_switch_pgd.
Acked-by: Will Deacon <will@kernel.org>
Co-authored-by: Will Deacon <will@kernel.org>
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20210319100146.1149909-18-qperret@google.com
2021-03-19 10:01:25 +00:00
|
|
|
../fpsimd.o ../hyp-entry.o ../exception.o ../pgtable.o
|
list: Introduce CONFIG_LIST_HARDENED
Numerous production kernel configs (see [1, 2]) are choosing to enable
CONFIG_DEBUG_LIST, which is also being recommended by KSPP for hardened
configs [3]. The motivation behind this is that the option can be used
as a security hardening feature (e.g. CVE-2019-2215 and CVE-2019-2025
are mitigated by the option [4]).
The feature has never been designed with performance in mind, yet common
list manipulation is happening across hot paths all over the kernel.
Introduce CONFIG_LIST_HARDENED, which performs list pointer checking
inline, and only upon list corruption calls the reporting slow path.
To generate optimal machine code with CONFIG_LIST_HARDENED:
1. Elide checking for pointer values which upon dereference would
result in an immediate access fault (i.e. minimal hardening
checks). The trade-off is lower-quality error reports.
2. Use the __preserve_most function attribute (available with Clang,
but not yet with GCC) to minimize the code footprint for calling
the reporting slow path. As a result, function size of callers is
reduced by avoiding saving registers before calling the rarely
called reporting slow path.
Note that all TUs in lib/Makefile already disable function tracing,
including list_debug.c, and __preserve_most's implied notrace has
no effect in this case.
3. Because the inline checks are a subset of the full set of checks in
__list_*_valid_or_report(), always return false if the inline
checks failed. This avoids redundant compare and conditional
branch right after return from the slow path.
As a side-effect of the checks being inline, if the compiler can prove
some condition to always be true, it can completely elide some checks.
Since DEBUG_LIST is functionally a superset of LIST_HARDENED, the
Kconfig variables are changed to reflect that: DEBUG_LIST selects
LIST_HARDENED, whereas LIST_HARDENED itself has no dependency on
DEBUG_LIST.
Running netperf with CONFIG_LIST_HARDENED (using a Clang compiler with
"preserve_most") shows throughput improvements, in my case of ~7% on
average (up to 20-30% on some test cases).
Link: https://r.android.com/1266735 [1]
Link: https://gitlab.archlinux.org/archlinux/packaging/packages/linux/-/blob/main/config [2]
Link: https://kernsec.org/wiki/index.php/Kernel_Self_Protection_Project/Recommended_Settings [3]
Link: https://googleprojectzero.blogspot.com/2019/11/bad-binder-android-in-wild-exploit.html [4]
Signed-off-by: Marco Elver <elver@google.com>
Link: https://lore.kernel.org/r/20230811151847.1594958-3-elver@google.com
Signed-off-by: Kees Cook <keescook@chromium.org>
2023-08-11 15:18:40 +00:00
|
|
|
hyp-obj-$(CONFIG_LIST_HARDENED) += list_debug.o
|
2022-06-13 09:20:25 +00:00
|
|
|
hyp-obj-y += $(lib-objs)
|
2020-06-25 13:14:08 +00:00
|
|
|
|
2020-09-22 20:49:01 +00:00
|
|
|
##
|
|
|
|
|
## Build rules for compiling nVHE hyp code
|
|
|
|
|
## Output of this folder is `kvm_nvhe.o`, a partially linked object
|
|
|
|
|
## file containing all nVHE hyp code and data.
|
|
|
|
|
##
|
2020-06-25 13:14:08 +00:00
|
|
|
|
2022-06-13 09:20:25 +00:00
|
|
|
hyp-obj := $(patsubst %.o,%.nvhe.o,$(hyp-obj-y))
|
2020-09-22 20:49:01 +00:00
|
|
|
obj-y := kvm_nvhe.o
|
2022-06-13 09:20:26 +00:00
|
|
|
targets += $(hyp-obj) kvm_nvhe.tmp.o kvm_nvhe.rel.o hyp.lds hyp-reloc.S hyp-reloc.o
|
2020-09-22 20:49:01 +00:00
|
|
|
|
|
|
|
|
# 1) Compile all source files to `.nvhe.o` object files. The file extension
|
|
|
|
|
# avoids file name clashes for files shared with VHE.
|
|
|
|
|
$(obj)/%.nvhe.o: $(src)/%.c FORCE
|
2020-06-25 13:14:08 +00:00
|
|
|
$(call if_changed_rule,cc_o_c)
|
2020-09-22 20:49:01 +00:00
|
|
|
$(obj)/%.nvhe.o: $(src)/%.S FORCE
|
2020-06-25 13:14:08 +00:00
|
|
|
$(call if_changed_rule,as_o_S)
|
|
|
|
|
|
2020-09-22 20:49:01 +00:00
|
|
|
# 2) Compile linker script.
|
|
|
|
|
$(obj)/hyp.lds: $(src)/hyp.lds.S FORCE
|
|
|
|
|
$(call if_changed_dep,cpp_lds_S)
|
|
|
|
|
|
|
|
|
|
# 3) Partially link all '.nvhe.o' files and apply the linker script.
|
|
|
|
|
# Prefixes names of ELF sections with '.hyp', eg. '.hyp.text'.
|
|
|
|
|
# Note: The following rule assumes that the 'ld' rule puts LDFLAGS before
|
|
|
|
|
# the list of dependencies to form '-T $(obj)/hyp.lds'. This is to
|
|
|
|
|
# keep the dependency on the target while avoiding an error from
|
|
|
|
|
# GNU ld if the linker script is passed to it twice.
|
|
|
|
|
LDFLAGS_kvm_nvhe.tmp.o := -r -T
|
|
|
|
|
$(obj)/kvm_nvhe.tmp.o: $(obj)/hyp.lds $(addprefix $(obj)/,$(hyp-obj)) FORCE
|
|
|
|
|
$(call if_changed,ld)
|
|
|
|
|
|
2021-01-05 18:05:37 +00:00
|
|
|
# 4) Generate list of hyp code/data positions that need to be relocated at
|
|
|
|
|
# runtime. Because the hypervisor is part of the kernel binary, relocations
|
|
|
|
|
# produce a kernel VA. We enumerate relocations targeting hyp at build time
|
|
|
|
|
# and convert the kernel VAs at those positions to hyp VAs.
|
2021-09-07 05:21:37 +00:00
|
|
|
$(obj)/hyp-reloc.S: $(obj)/kvm_nvhe.tmp.o $(obj)/gen-hyprel FORCE
|
2021-01-05 18:05:37 +00:00
|
|
|
$(call if_changed,hyprel)
|
|
|
|
|
|
|
|
|
|
# 5) Compile hyp-reloc.S and link it into the existing partially linked object.
|
|
|
|
|
# The object file now contains a section with pointers to hyp positions that
|
|
|
|
|
# will contain kernel VAs at runtime. These pointers have relocations on them
|
|
|
|
|
# so that they get updated as the hyp object is linked into `vmlinux`.
|
|
|
|
|
LDFLAGS_kvm_nvhe.rel.o := -r
|
|
|
|
|
$(obj)/kvm_nvhe.rel.o: $(obj)/kvm_nvhe.tmp.o $(obj)/hyp-reloc.o FORCE
|
|
|
|
|
$(call if_changed,ld)
|
|
|
|
|
|
|
|
|
|
# 6) Produce the final 'kvm_nvhe.o', ready to be linked into 'vmlinux'.
|
2020-09-22 20:49:01 +00:00
|
|
|
# Prefixes names of ELF symbols with '__kvm_nvhe_'.
|
2021-01-05 18:05:37 +00:00
|
|
|
$(obj)/kvm_nvhe.o: $(obj)/kvm_nvhe.rel.o FORCE
|
2020-09-22 20:49:01 +00:00
|
|
|
$(call if_changed,hypcopy)
|
2020-07-30 13:25:19 +00:00
|
|
|
|
2021-01-05 18:05:37 +00:00
|
|
|
# The HYPREL command calls `gen-hyprel` to generate an assembly file with
|
|
|
|
|
# a list of relocations targeting hyp code/data.
|
|
|
|
|
quiet_cmd_hyprel = HYPREL $@
|
|
|
|
|
cmd_hyprel = $(obj)/gen-hyprel $< > $@
|
|
|
|
|
|
2020-07-30 13:25:19 +00:00
|
|
|
# The HYPCOPY command uses `objcopy` to prefix all ELF symbol names
|
2020-09-22 20:49:01 +00:00
|
|
|
# to avoid clashes with VHE code/data.
|
2020-06-25 13:14:08 +00:00
|
|
|
quiet_cmd_hypcopy = HYPCOPY $@
|
2020-09-22 20:49:01 +00:00
|
|
|
cmd_hypcopy = $(OBJCOPY) --prefix-symbols=__kvm_nvhe_ $< $@
|
2020-06-25 13:14:19 +00:00
|
|
|
|
2021-04-08 18:28:42 +00:00
|
|
|
# Remove ftrace, Shadow Call Stack, and CFI CFLAGS.
|
|
|
|
|
# This is equivalent to the 'notrace', '__noscs', and '__nocfi' annotations.
|
|
|
|
|
KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS) $(CC_FLAGS_CFI), $(KBUILD_CFLAGS))
|
2022-10-14 18:45:32 +00:00
|
|
|
# Starting from 13.0.0 llvm emits SHT_REL section '.llvm.call-graph-profile'
|
|
|
|
|
# when profile optimization is applied. gen-hyprel does not support SHT_REL and
|
|
|
|
|
# causes a build failure. Remove profile optimization flags.
|
|
|
|
|
KBUILD_CFLAGS := $(filter-out -fprofile-sample-use=% -fprofile-use=%, $(KBUILD_CFLAGS))
|
2022-10-27 15:59:06 +00:00
|
|
|
KBUILD_CFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables
|
2020-06-25 13:14:08 +00:00
|
|
|
|
|
|
|
|
# KVM nVHE code is run at a different exception code with a different map, so
|
|
|
|
|
# compiler instrumentation that inserts callbacks or checks into the code may
|
|
|
|
|
# cause crashes. Just disable it.
|
|
|
|
|
GCOV_PROFILE := n
|
|
|
|
|
KASAN_SANITIZE := n
|
2021-12-11 13:17:34 +00:00
|
|
|
KCSAN_SANITIZE := n
|
2020-06-25 13:14:08 +00:00
|
|
|
UBSAN_SANITIZE := n
|
|
|
|
|
KCOV_INSTRUMENT := n
|
|
|
|
|
|
|
|
|
|
# Skip objtool checking for this directory because nVHE code is compiled with
|
|
|
|
|
# non-standard build rules.
|
|
|
|
|
OBJECT_FILES_NON_STANDARD := y
|