mirrors
/
linux-stable
mirror of https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
1
0
Fork 0
Code Issues Releases Wiki Activity
linux-stable/arch/riscv/kernel/alternative.c

237 lines
6.0 KiB
C
Raw Normal View History

riscv: Introduce alternative mechanism to apply errata solution Introduce the "alternative" mechanism from ARM64 and x86 to apply the CPU vendors' errata solution at runtime. The main purpose of this patch is to provide a framework. Therefore, the implementation is quite basic for now so that some scenarios could not use this schemei, such as patching code to a module, relocating the patching code and heterogeneous CPU topology. Users could use the macro ALTERNATIVE to apply an errata to the existing code flow. In the macro ALTERNATIVE, users need to specify the manufacturer information(vendorid, archid, and impid) for this errata. Therefore, kernel will know this errata is suitable for which CPU core. During the booting procedure, kernel will select the errata required by the CPU core and then patch it. It means that the kernel only applies the errata to the specified CPU core. In this case, the vendor's errata does not affect each other at runtime. The above patching procedure only occurs during the booting phase, so we only take the overhead of the "alternative" mechanism once. This "alternative" mechanism is enabled by default to ensure that all required errata will be applied. However, users can disable this feature by the Kconfig "CONFIG_RISCV_ERRATA_ALTERNATIVE". Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Reviewed-by: Anup Patel <anup@brainfault.org> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-03-22 14:26:03 +00:00
// SPDX-License-Identifier: GPL-2.0-only
/*
* alternative runtime patching
* inspired by the ARM64 and x86 version
*
* Copyright (C) 2021 Sifive.
*/
#include <linux/init.h>
riscv: implement module alternatives This allows alternatives to also be applied when loading modules and follows the implementation of other architectures (e.g. arm64). Signed-off-by: Heiko Stuebner <heiko@sntech.de> Reviewed-by: Philipp Tomsich <philipp.tomsich@vrull.eu> Link: https://lore.kernel.org/r/20220511192921.2223629-4-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-05-11 19:29:12 +00:00
#include <linux/module.h>
riscv: Introduce alternative mechanism to apply errata solution Introduce the "alternative" mechanism from ARM64 and x86 to apply the CPU vendors' errata solution at runtime. The main purpose of this patch is to provide a framework. Therefore, the implementation is quite basic for now so that some scenarios could not use this schemei, such as patching code to a module, relocating the patching code and heterogeneous CPU topology. Users could use the macro ALTERNATIVE to apply an errata to the existing code flow. In the macro ALTERNATIVE, users need to specify the manufacturer information(vendorid, archid, and impid) for this errata. Therefore, kernel will know this errata is suitable for which CPU core. During the booting procedure, kernel will select the errata required by the CPU core and then patch it. It means that the kernel only applies the errata to the specified CPU core. In this case, the vendor's errata does not affect each other at runtime. The above patching procedure only occurs during the booting phase, so we only take the overhead of the "alternative" mechanism once. This "alternative" mechanism is enabled by default to ensure that all required errata will be applied. However, users can disable this feature by the Kconfig "CONFIG_RISCV_ERRATA_ALTERNATIVE". Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Reviewed-by: Anup Patel <anup@brainfault.org> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-03-22 14:26:03 +00:00
#include <linux/cpu.h>
#include <linux/uaccess.h>
#include <asm/alternative.h>
riscv: alternative: patch alternatives in the vDSO Make it possible to use alternatives in the vDSO, so that better implementations can be used if possible. Signed-off-by: Jisheng Zhang <jszhang@kernel.org> Reviewed-by: Guo Ren <guoren@kernel.org> Reviewed-by: Andrew Jones <ajones@ventanamicro.com> Link: https://lore.kernel.org/r/20230128172856.3814-11-jszhang@kernel.org Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2023-01-28 17:28:53 +00:00
#include <asm/module.h>
riscv: Introduce alternative mechanism to apply errata solution Introduce the "alternative" mechanism from ARM64 and x86 to apply the CPU vendors' errata solution at runtime. The main purpose of this patch is to provide a framework. Therefore, the implementation is quite basic for now so that some scenarios could not use this schemei, such as patching code to a module, relocating the patching code and heterogeneous CPU topology. Users could use the macro ALTERNATIVE to apply an errata to the existing code flow. In the macro ALTERNATIVE, users need to specify the manufacturer information(vendorid, archid, and impid) for this errata. Therefore, kernel will know this errata is suitable for which CPU core. During the booting procedure, kernel will select the errata required by the CPU core and then patch it. It means that the kernel only applies the errata to the specified CPU core. In this case, the vendor's errata does not affect each other at runtime. The above patching procedure only occurs during the booting phase, so we only take the overhead of the "alternative" mechanism once. This "alternative" mechanism is enabled by default to ensure that all required errata will be applied. However, users can disable this feature by the Kconfig "CONFIG_RISCV_ERRATA_ALTERNATIVE". Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Reviewed-by: Anup Patel <anup@brainfault.org> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-03-22 14:26:03 +00:00
#include <asm/sections.h>
riscv: alternative: patch alternatives in the vDSO Make it possible to use alternatives in the vDSO, so that better implementations can be used if possible. Signed-off-by: Jisheng Zhang <jszhang@kernel.org> Reviewed-by: Guo Ren <guoren@kernel.org> Reviewed-by: Andrew Jones <ajones@ventanamicro.com> Link: https://lore.kernel.org/r/20230128172856.3814-11-jszhang@kernel.org Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2023-01-28 17:28:53 +00:00
#include <asm/vdso.h>
riscv: Introduce alternative mechanism to apply errata solution Introduce the "alternative" mechanism from ARM64 and x86 to apply the CPU vendors' errata solution at runtime. The main purpose of this patch is to provide a framework. Therefore, the implementation is quite basic for now so that some scenarios could not use this schemei, such as patching code to a module, relocating the patching code and heterogeneous CPU topology. Users could use the macro ALTERNATIVE to apply an errata to the existing code flow. In the macro ALTERNATIVE, users need to specify the manufacturer information(vendorid, archid, and impid) for this errata. Therefore, kernel will know this errata is suitable for which CPU core. During the booting procedure, kernel will select the errata required by the CPU core and then patch it. It means that the kernel only applies the errata to the specified CPU core. In this case, the vendor's errata does not affect each other at runtime. The above patching procedure only occurs during the booting phase, so we only take the overhead of the "alternative" mechanism once. This "alternative" mechanism is enabled by default to ensure that all required errata will be applied. However, users can disable this feature by the Kconfig "CONFIG_RISCV_ERRATA_ALTERNATIVE". Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Reviewed-by: Anup Patel <anup@brainfault.org> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-03-22 14:26:03 +00:00
#include <asm/vendorid_list.h>
#include <asm/sbi.h>
#include <asm/csr.h>
RISC-V: fix auipc-jalr addresses in patched alternatives Alternatives live in a different section, so addresses used by call functions will point to wrong locations after the patch got applied. Similar to arm64, adjust the location to consider that offset. Reviewed-by: Conor Dooley <conor.dooley@microchip.com> Reviewed-by: Andrew Jones <ajones@ventanamicro.com> Reviewed-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Tested-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Signed-off-by: Heiko Stuebner <heiko.stuebner@vrull.eu> Link: https://lore.kernel.org/r/20221223221332.4127602-13-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-12-23 22:13:32 +00:00
#include <asm/insn.h>
#include <asm/patch.h>
riscv: Introduce alternative mechanism to apply errata solution Introduce the "alternative" mechanism from ARM64 and x86 to apply the CPU vendors' errata solution at runtime. The main purpose of this patch is to provide a framework. Therefore, the implementation is quite basic for now so that some scenarios could not use this schemei, such as patching code to a module, relocating the patching code and heterogeneous CPU topology. Users could use the macro ALTERNATIVE to apply an errata to the existing code flow. In the macro ALTERNATIVE, users need to specify the manufacturer information(vendorid, archid, and impid) for this errata. Therefore, kernel will know this errata is suitable for which CPU core. During the booting procedure, kernel will select the errata required by the CPU core and then patch it. It means that the kernel only applies the errata to the specified CPU core. In this case, the vendor's errata does not affect each other at runtime. The above patching procedure only occurs during the booting phase, so we only take the overhead of the "alternative" mechanism once. This "alternative" mechanism is enabled by default to ensure that all required errata will be applied. However, users can disable this feature by the Kconfig "CONFIG_RISCV_ERRATA_ALTERNATIVE". Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Reviewed-by: Anup Patel <anup@brainfault.org> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-03-22 14:26:03 +00:00
riscv: don't use global static vars to store alternative data Right now the code uses a global struct to store vendor-ids and another global variable to store the vendor-patch-function. There exist specific cases where we'll need to patch the kernel at an even earlier stage, where trying to write to a static variable might actually result in hangs. Also collecting the vendor-information consists of 3 sbi-ecalls (or csr-reads) which is pretty negligible in the context of booting a kernel. So rework the code to not rely on static variables and instead collect the vendor-information when a round of alternatives is to be applied. Signed-off-by: Heiko Stuebner <heiko@sntech.de> Reviewed-by: Guo Ren <guoren@kernel.org> Reviewed-by: Philipp Tomsich <philipp.tomsich@vrull.eu> Link: https://lore.kernel.org/r/20220511192921.2223629-12-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-05-11 19:29:20 +00:00
struct cpu_manufacturer_info_t {
riscv: Introduce alternative mechanism to apply errata solution Introduce the "alternative" mechanism from ARM64 and x86 to apply the CPU vendors' errata solution at runtime. The main purpose of this patch is to provide a framework. Therefore, the implementation is quite basic for now so that some scenarios could not use this schemei, such as patching code to a module, relocating the patching code and heterogeneous CPU topology. Users could use the macro ALTERNATIVE to apply an errata to the existing code flow. In the macro ALTERNATIVE, users need to specify the manufacturer information(vendorid, archid, and impid) for this errata. Therefore, kernel will know this errata is suitable for which CPU core. During the booting procedure, kernel will select the errata required by the CPU core and then patch it. It means that the kernel only applies the errata to the specified CPU core. In this case, the vendor's errata does not affect each other at runtime. The above patching procedure only occurs during the booting phase, so we only take the overhead of the "alternative" mechanism once. This "alternative" mechanism is enabled by default to ensure that all required errata will be applied. However, users can disable this feature by the Kconfig "CONFIG_RISCV_ERRATA_ALTERNATIVE". Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Reviewed-by: Anup Patel <anup@brainfault.org> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-03-22 14:26:03 +00:00
unsigned long vendor_id;
unsigned long arch_id;
unsigned long imp_id;
riscv: make patch-function pointer more generic in cpu_manufacturer_info struct During review the naming of the function-pointer was called confusing as the vendor id is just one of three inputs for the patching and indeed it serves no real purpose, as with recent changes the function pointer is not a static global element anymore, so drop the "vendor_" prefix. Suggested-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Heiko Stuebner <heiko@sntech.de> Link: https://lore.kernel.org/r/20220526205646.258337-4-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-05-26 20:56:44 +00:00
void (*patch_func)(struct alt_entry *begin, struct alt_entry *end,
riscv: don't use global static vars to store alternative data Right now the code uses a global struct to store vendor-ids and another global variable to store the vendor-patch-function. There exist specific cases where we'll need to patch the kernel at an even earlier stage, where trying to write to a static variable might actually result in hangs. Also collecting the vendor-information consists of 3 sbi-ecalls (or csr-reads) which is pretty negligible in the context of booting a kernel. So rework the code to not rely on static variables and instead collect the vendor-information when a round of alternatives is to be applied. Signed-off-by: Heiko Stuebner <heiko@sntech.de> Reviewed-by: Guo Ren <guoren@kernel.org> Reviewed-by: Philipp Tomsich <philipp.tomsich@vrull.eu> Link: https://lore.kernel.org/r/20220511192921.2223629-12-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-05-11 19:29:20 +00:00
unsigned long archid, unsigned long impid,
unsigned int stage);
};
riscv: Introduce alternative mechanism to apply errata solution Introduce the "alternative" mechanism from ARM64 and x86 to apply the CPU vendors' errata solution at runtime. The main purpose of this patch is to provide a framework. Therefore, the implementation is quite basic for now so that some scenarios could not use this schemei, such as patching code to a module, relocating the patching code and heterogeneous CPU topology. Users could use the macro ALTERNATIVE to apply an errata to the existing code flow. In the macro ALTERNATIVE, users need to specify the manufacturer information(vendorid, archid, and impid) for this errata. Therefore, kernel will know this errata is suitable for which CPU core. During the booting procedure, kernel will select the errata required by the CPU core and then patch it. It means that the kernel only applies the errata to the specified CPU core. In this case, the vendor's errata does not affect each other at runtime. The above patching procedure only occurs during the booting phase, so we only take the overhead of the "alternative" mechanism once. This "alternative" mechanism is enabled by default to ensure that all required errata will be applied. However, users can disable this feature by the Kconfig "CONFIG_RISCV_ERRATA_ALTERNATIVE". Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Reviewed-by: Anup Patel <anup@brainfault.org> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-03-22 14:26:03 +00:00
RISC-V: hwprobe: Remove __init on probe_vendor_features() probe_vendor_features() is now called from smp_callin(), which is not __init code and runs during cpu hotplug events. Remove the __init_or_module decoration from it and the functions it calls to avoid walking into outer space. Fixes: 62a31d6e38bd ("RISC-V: hwprobe: Support probing of misaligned access performance") Signed-off-by: Evan Green <evan@rivosinc.com> Reviewed-by: Conor Dooley <conor.dooley@microchip.com> Link: https://lore.kernel.org/r/20230420194934.1871356-1-evan@rivosinc.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2023-04-20 19:49:34 +00:00
static void riscv_fill_cpu_mfr_info(struct cpu_manufacturer_info_t *cpu_mfr_info)
riscv: Introduce alternative mechanism to apply errata solution Introduce the "alternative" mechanism from ARM64 and x86 to apply the CPU vendors' errata solution at runtime. The main purpose of this patch is to provide a framework. Therefore, the implementation is quite basic for now so that some scenarios could not use this schemei, such as patching code to a module, relocating the patching code and heterogeneous CPU topology. Users could use the macro ALTERNATIVE to apply an errata to the existing code flow. In the macro ALTERNATIVE, users need to specify the manufacturer information(vendorid, archid, and impid) for this errata. Therefore, kernel will know this errata is suitable for which CPU core. During the booting procedure, kernel will select the errata required by the CPU core and then patch it. It means that the kernel only applies the errata to the specified CPU core. In this case, the vendor's errata does not affect each other at runtime. The above patching procedure only occurs during the booting phase, so we only take the overhead of the "alternative" mechanism once. This "alternative" mechanism is enabled by default to ensure that all required errata will be applied. However, users can disable this feature by the Kconfig "CONFIG_RISCV_ERRATA_ALTERNATIVE". Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Reviewed-by: Anup Patel <anup@brainfault.org> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-03-22 14:26:03 +00:00
{
#ifdef CONFIG_RISCV_M_MODE
riscv: don't use global static vars to store alternative data Right now the code uses a global struct to store vendor-ids and another global variable to store the vendor-patch-function. There exist specific cases where we'll need to patch the kernel at an even earlier stage, where trying to write to a static variable might actually result in hangs. Also collecting the vendor-information consists of 3 sbi-ecalls (or csr-reads) which is pretty negligible in the context of booting a kernel. So rework the code to not rely on static variables and instead collect the vendor-information when a round of alternatives is to be applied. Signed-off-by: Heiko Stuebner <heiko@sntech.de> Reviewed-by: Guo Ren <guoren@kernel.org> Reviewed-by: Philipp Tomsich <philipp.tomsich@vrull.eu> Link: https://lore.kernel.org/r/20220511192921.2223629-12-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-05-11 19:29:20 +00:00
cpu_mfr_info->vendor_id = csr_read(CSR_MVENDORID);
cpu_mfr_info->arch_id = csr_read(CSR_MARCHID);
cpu_mfr_info->imp_id = csr_read(CSR_MIMPID);
riscv: Introduce alternative mechanism to apply errata solution Introduce the "alternative" mechanism from ARM64 and x86 to apply the CPU vendors' errata solution at runtime. The main purpose of this patch is to provide a framework. Therefore, the implementation is quite basic for now so that some scenarios could not use this schemei, such as patching code to a module, relocating the patching code and heterogeneous CPU topology. Users could use the macro ALTERNATIVE to apply an errata to the existing code flow. In the macro ALTERNATIVE, users need to specify the manufacturer information(vendorid, archid, and impid) for this errata. Therefore, kernel will know this errata is suitable for which CPU core. During the booting procedure, kernel will select the errata required by the CPU core and then patch it. It means that the kernel only applies the errata to the specified CPU core. In this case, the vendor's errata does not affect each other at runtime. The above patching procedure only occurs during the booting phase, so we only take the overhead of the "alternative" mechanism once. This "alternative" mechanism is enabled by default to ensure that all required errata will be applied. However, users can disable this feature by the Kconfig "CONFIG_RISCV_ERRATA_ALTERNATIVE". Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Reviewed-by: Anup Patel <anup@brainfault.org> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-03-22 14:26:03 +00:00
#else
riscv: don't use global static vars to store alternative data Right now the code uses a global struct to store vendor-ids and another global variable to store the vendor-patch-function. There exist specific cases where we'll need to patch the kernel at an even earlier stage, where trying to write to a static variable might actually result in hangs. Also collecting the vendor-information consists of 3 sbi-ecalls (or csr-reads) which is pretty negligible in the context of booting a kernel. So rework the code to not rely on static variables and instead collect the vendor-information when a round of alternatives is to be applied. Signed-off-by: Heiko Stuebner <heiko@sntech.de> Reviewed-by: Guo Ren <guoren@kernel.org> Reviewed-by: Philipp Tomsich <philipp.tomsich@vrull.eu> Link: https://lore.kernel.org/r/20220511192921.2223629-12-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-05-11 19:29:20 +00:00
cpu_mfr_info->vendor_id = sbi_get_mvendorid();
cpu_mfr_info->arch_id = sbi_get_marchid();
cpu_mfr_info->imp_id = sbi_get_mimpid();
riscv: Introduce alternative mechanism to apply errata solution Introduce the "alternative" mechanism from ARM64 and x86 to apply the CPU vendors' errata solution at runtime. The main purpose of this patch is to provide a framework. Therefore, the implementation is quite basic for now so that some scenarios could not use this schemei, such as patching code to a module, relocating the patching code and heterogeneous CPU topology. Users could use the macro ALTERNATIVE to apply an errata to the existing code flow. In the macro ALTERNATIVE, users need to specify the manufacturer information(vendorid, archid, and impid) for this errata. Therefore, kernel will know this errata is suitable for which CPU core. During the booting procedure, kernel will select the errata required by the CPU core and then patch it. It means that the kernel only applies the errata to the specified CPU core. In this case, the vendor's errata does not affect each other at runtime. The above patching procedure only occurs during the booting phase, so we only take the overhead of the "alternative" mechanism once. This "alternative" mechanism is enabled by default to ensure that all required errata will be applied. However, users can disable this feature by the Kconfig "CONFIG_RISCV_ERRATA_ALTERNATIVE". Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Reviewed-by: Anup Patel <anup@brainfault.org> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-03-22 14:26:03 +00:00
#endif
riscv: don't use global static vars to store alternative data Right now the code uses a global struct to store vendor-ids and another global variable to store the vendor-patch-function. There exist specific cases where we'll need to patch the kernel at an even earlier stage, where trying to write to a static variable might actually result in hangs. Also collecting the vendor-information consists of 3 sbi-ecalls (or csr-reads) which is pretty negligible in the context of booting a kernel. So rework the code to not rely on static variables and instead collect the vendor-information when a round of alternatives is to be applied. Signed-off-by: Heiko Stuebner <heiko@sntech.de> Reviewed-by: Guo Ren <guoren@kernel.org> Reviewed-by: Philipp Tomsich <philipp.tomsich@vrull.eu> Link: https://lore.kernel.org/r/20220511192921.2223629-12-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-05-11 19:29:20 +00:00
switch (cpu_mfr_info->vendor_id) {
riscv: errata: Add Andes alternative ports Add required ports of the Alternative scheme for Andes CPU cores. I/O Coherence Port (IOCP) provides an AXI interface for connecting external non-caching masters, such as DMA controllers. IOCP is a specification option and is disabled on the Renesas RZ/Five SoC due to this reason cache management needs a software workaround. Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Reviewed-by: Conor Dooley <conor.dooley@microchip.com> Tested-by: Conor Dooley <conor.dooley@microchip.com> # tyre-kicking on a d1 Link: https://lore.kernel.org/r/20230818135723.80612-3-prabhakar.mahadev-lad.rj@bp.renesas.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2023-08-18 13:57:19 +00:00
#ifdef CONFIG_ERRATA_ANDES
riscv: errata: Rename defines for Andes Use "ANDES" rather than "ANDESTECH" to unify the naming convention with directory, file names, Kconfig options and other definitions. Signed-off-by: Yu Chien Peter Lin <peterlin@andestech.com> Reviewed-by: Charles Ci-Jyun Wu <dminus@andestech.com> Reviewed-by: Leo Yu-Chi Liang <ycliang@andestech.com> Acked-by: Conor Dooley <conor.dooley@microchip.com> Reviewed-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Link: https://lore.kernel.org/r/20240222083946.3977135-2-peterlin@andestech.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2024-02-22 08:39:37 +00:00
case ANDES_VENDOR_ID:
riscv: errata: Add Andes alternative ports Add required ports of the Alternative scheme for Andes CPU cores. I/O Coherence Port (IOCP) provides an AXI interface for connecting external non-caching masters, such as DMA controllers. IOCP is a specification option and is disabled on the Renesas RZ/Five SoC due to this reason cache management needs a software workaround. Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Reviewed-by: Conor Dooley <conor.dooley@microchip.com> Tested-by: Conor Dooley <conor.dooley@microchip.com> # tyre-kicking on a d1 Link: https://lore.kernel.org/r/20230818135723.80612-3-prabhakar.mahadev-lad.rj@bp.renesas.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2023-08-18 13:57:19 +00:00
cpu_mfr_info->patch_func = andes_errata_patch_func;
break;
#endif
riscv: sifive: Add SiFive alternative ports Add required ports of the Alternative scheme for SiFive. Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Reviewed-by: Anup Patel <anup@brainfault.org> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-03-22 14:26:04 +00:00
#ifdef CONFIG_ERRATA_SIFIVE
case SIFIVE_VENDOR_ID:
riscv: make patch-function pointer more generic in cpu_manufacturer_info struct During review the naming of the function-pointer was called confusing as the vendor id is just one of three inputs for the patching and indeed it serves no real purpose, as with recent changes the function pointer is not a static global element anymore, so drop the "vendor_" prefix. Suggested-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Heiko Stuebner <heiko@sntech.de> Link: https://lore.kernel.org/r/20220526205646.258337-4-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-05-26 20:56:44 +00:00
cpu_mfr_info->patch_func = sifive_errata_patch_func;
riscv: sifive: Add SiFive alternative ports Add required ports of the Alternative scheme for SiFive. Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Reviewed-by: Anup Patel <anup@brainfault.org> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-03-22 14:26:04 +00:00
break;
riscv: add memory-type errata for T-Head Some current cpus based on T-Head cores implement memory-types way different than described in the svpbmt spec even going so far as using PTE bits marked as reserved. Add the T-Head vendor-id and necessary errata code to replace the affected instructions. Signed-off-by: Heiko Stuebner <heiko@sntech.de> Tested-by: Samuel Holland <samuel@sholland.org> Link: https://lore.kernel.org/r/20220511192921.2223629-13-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-05-11 19:29:21 +00:00
#endif
#ifdef CONFIG_ERRATA_THEAD
case THEAD_VENDOR_ID:
riscv: make patch-function pointer more generic in cpu_manufacturer_info struct During review the naming of the function-pointer was called confusing as the vendor id is just one of three inputs for the patching and indeed it serves no real purpose, as with recent changes the function pointer is not a static global element anymore, so drop the "vendor_" prefix. Suggested-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Heiko Stuebner <heiko@sntech.de> Link: https://lore.kernel.org/r/20220526205646.258337-4-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-05-26 20:56:44 +00:00
cpu_mfr_info->patch_func = thead_errata_patch_func;
riscv: add memory-type errata for T-Head Some current cpus based on T-Head cores implement memory-types way different than described in the svpbmt spec even going so far as using PTE bits marked as reserved. Add the T-Head vendor-id and necessary errata code to replace the affected instructions. Signed-off-by: Heiko Stuebner <heiko@sntech.de> Tested-by: Samuel Holland <samuel@sholland.org> Link: https://lore.kernel.org/r/20220511192921.2223629-13-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-05-11 19:29:21 +00:00
break;
riscv: sifive: Add SiFive alternative ports Add required ports of the Alternative scheme for SiFive. Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Reviewed-by: Anup Patel <anup@brainfault.org> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-03-22 14:26:04 +00:00
#endif
riscv: Introduce alternative mechanism to apply errata solution Introduce the "alternative" mechanism from ARM64 and x86 to apply the CPU vendors' errata solution at runtime. The main purpose of this patch is to provide a framework. Therefore, the implementation is quite basic for now so that some scenarios could not use this schemei, such as patching code to a module, relocating the patching code and heterogeneous CPU topology. Users could use the macro ALTERNATIVE to apply an errata to the existing code flow. In the macro ALTERNATIVE, users need to specify the manufacturer information(vendorid, archid, and impid) for this errata. Therefore, kernel will know this errata is suitable for which CPU core. During the booting procedure, kernel will select the errata required by the CPU core and then patch it. It means that the kernel only applies the errata to the specified CPU core. In this case, the vendor's errata does not affect each other at runtime. The above patching procedure only occurs during the booting phase, so we only take the overhead of the "alternative" mechanism once. This "alternative" mechanism is enabled by default to ensure that all required errata will be applied. However, users can disable this feature by the Kconfig "CONFIG_RISCV_ERRATA_ALTERNATIVE". Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Reviewed-by: Anup Patel <anup@brainfault.org> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-03-22 14:26:03 +00:00
default:
riscv: make patch-function pointer more generic in cpu_manufacturer_info struct During review the naming of the function-pointer was called confusing as the vendor id is just one of three inputs for the patching and indeed it serves no real purpose, as with recent changes the function pointer is not a static global element anymore, so drop the "vendor_" prefix. Suggested-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Heiko Stuebner <heiko@sntech.de> Link: https://lore.kernel.org/r/20220526205646.258337-4-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-05-26 20:56:44 +00:00
cpu_mfr_info->patch_func = NULL;
riscv: Introduce alternative mechanism to apply errata solution Introduce the "alternative" mechanism from ARM64 and x86 to apply the CPU vendors' errata solution at runtime. The main purpose of this patch is to provide a framework. Therefore, the implementation is quite basic for now so that some scenarios could not use this schemei, such as patching code to a module, relocating the patching code and heterogeneous CPU topology. Users could use the macro ALTERNATIVE to apply an errata to the existing code flow. In the macro ALTERNATIVE, users need to specify the manufacturer information(vendorid, archid, and impid) for this errata. Therefore, kernel will know this errata is suitable for which CPU core. During the booting procedure, kernel will select the errata required by the CPU core and then patch it. It means that the kernel only applies the errata to the specified CPU core. In this case, the vendor's errata does not affect each other at runtime. The above patching procedure only occurs during the booting phase, so we only take the overhead of the "alternative" mechanism once. This "alternative" mechanism is enabled by default to ensure that all required errata will be applied. However, users can disable this feature by the Kconfig "CONFIG_RISCV_ERRATA_ALTERNATIVE". Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Reviewed-by: Anup Patel <anup@brainfault.org> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-03-22 14:26:03 +00:00
}
}
RISC-V: fix auipc-jalr addresses in patched alternatives Alternatives live in a different section, so addresses used by call functions will point to wrong locations after the patch got applied. Similar to arm64, adjust the location to consider that offset. Reviewed-by: Conor Dooley <conor.dooley@microchip.com> Reviewed-by: Andrew Jones <ajones@ventanamicro.com> Reviewed-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Tested-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Signed-off-by: Heiko Stuebner <heiko.stuebner@vrull.eu> Link: https://lore.kernel.org/r/20221223221332.4127602-13-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-12-23 22:13:32 +00:00
static u32 riscv_instruction_at(void *p)
{
u16 *parcel = p;
return (u32)parcel[0] | (u32)parcel[1] << 16;
}
static void riscv_alternative_fix_auipc_jalr(void *ptr, u32 auipc_insn,
u32 jalr_insn, int patch_offset)
{
u32 call[2] = { auipc_insn, jalr_insn };
s32 imm;
/* get and adjust new target address */
imm = riscv_insn_extract_utype_itype_imm(auipc_insn, jalr_insn);
imm -= patch_offset;
/* update instructions */
riscv_insn_insert_utype_itype_imm(&call[0], &call[1], imm);
/* patch the call place again */
patch_text_nosync(ptr, call, sizeof(u32) * 2);
}
riscv: fix jal offsets in patched alternatives Alternatives live in a different section, so offsets used by jal instruction will point to wrong locations after the patch got applied. Similar to arm64, adjust the location to consider that offset. Co-developed-by: Heiko Stuebner <heiko.stuebner@vrull.eu> Signed-off-by: Jisheng Zhang <jszhang@kernel.org> Reviewed-by: Conor Dooley <conor.dooley@microchip.com> Reviewed-by: Andrew Jones <ajones@ventanamicro.com> Signed-off-by: Heiko Stuebner <heiko.stuebner@vrull.eu> Link: https://lore.kernel.org/r/20230113212205.3534622-1-heiko@sntech.de Fixes: 27c653c06505 ("RISC-V: fix auipc-jalr addresses in patched alternatives") Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2023-01-13 21:22:05 +00:00
static void riscv_alternative_fix_jal(void *ptr, u32 jal_insn, int patch_offset)
{
s32 imm;
/* get and adjust new target address */
imm = riscv_insn_extract_jtype_imm(jal_insn);
imm -= patch_offset;
/* update instruction */
riscv_insn_insert_jtype_imm(&jal_insn, imm);
/* patch the call place again */
patch_text_nosync(ptr, &jal_insn, sizeof(u32));
}
RISC-V: fix auipc-jalr addresses in patched alternatives Alternatives live in a different section, so addresses used by call functions will point to wrong locations after the patch got applied. Similar to arm64, adjust the location to consider that offset. Reviewed-by: Conor Dooley <conor.dooley@microchip.com> Reviewed-by: Andrew Jones <ajones@ventanamicro.com> Reviewed-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Tested-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Signed-off-by: Heiko Stuebner <heiko.stuebner@vrull.eu> Link: https://lore.kernel.org/r/20221223221332.4127602-13-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-12-23 22:13:32 +00:00
void riscv_alternative_fix_offsets(void *alt_ptr, unsigned int len,
int patch_offset)
{
int num_insn = len / sizeof(u32);
int i;
for (i = 0; i < num_insn; i++) {
u32 insn = riscv_instruction_at(alt_ptr + i * sizeof(u32));
/*
* May be the start of an auipc + jalr pair
* Needs to check that at least one more instruction
* is in the list.
*/
if (riscv_insn_is_auipc(insn) && i < num_insn - 1) {
u32 insn2 = riscv_instruction_at(alt_ptr + (i + 1) * sizeof(u32));
if (!riscv_insn_is_jalr(insn2))
continue;
/* if instruction pair is a call, it will use the ra register */
if (RV_EXTRACT_RD_REG(insn) != 1)
continue;
riscv_alternative_fix_auipc_jalr(alt_ptr + i * sizeof(u32),
insn, insn2, patch_offset);
riscv: alternative: proceed one more instruction for auipc/jalr pair If we patched auipc + jalr pair, we'd better proceed one more instruction. Andrew pointed out "There's not a problem now, since we're only adding a fixup for jal, not jalr, but we should future-proof this and there's no reason to revisit an already fixed-up instruction anyway." Signed-off-by: Jisheng Zhang <jszhang@kernel.org> Suggested-by: Andrew Jones <ajones@ventanamicro.com> Reviewed-by: Andrew Jones <ajones@ventanamicro.com> Reviewed-by: Heiko Stuebner <heiko@sntech.de> Link: https://lore.kernel.org/r/20230115162811.3146-1-jszhang@kernel.org Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2023-01-15 16:28:11 +00:00
i++;
RISC-V: fix auipc-jalr addresses in patched alternatives Alternatives live in a different section, so addresses used by call functions will point to wrong locations after the patch got applied. Similar to arm64, adjust the location to consider that offset. Reviewed-by: Conor Dooley <conor.dooley@microchip.com> Reviewed-by: Andrew Jones <ajones@ventanamicro.com> Reviewed-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Tested-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Signed-off-by: Heiko Stuebner <heiko.stuebner@vrull.eu> Link: https://lore.kernel.org/r/20221223221332.4127602-13-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-12-23 22:13:32 +00:00
}
riscv: fix jal offsets in patched alternatives Alternatives live in a different section, so offsets used by jal instruction will point to wrong locations after the patch got applied. Similar to arm64, adjust the location to consider that offset. Co-developed-by: Heiko Stuebner <heiko.stuebner@vrull.eu> Signed-off-by: Jisheng Zhang <jszhang@kernel.org> Reviewed-by: Conor Dooley <conor.dooley@microchip.com> Reviewed-by: Andrew Jones <ajones@ventanamicro.com> Signed-off-by: Heiko Stuebner <heiko.stuebner@vrull.eu> Link: https://lore.kernel.org/r/20230113212205.3534622-1-heiko@sntech.de Fixes: 27c653c06505 ("RISC-V: fix auipc-jalr addresses in patched alternatives") Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2023-01-13 21:22:05 +00:00
if (riscv_insn_is_jal(insn)) {
s32 imm = riscv_insn_extract_jtype_imm(insn);
/* Don't modify jumps inside the alternative block */
if ((alt_ptr + i * sizeof(u32) + imm) >= alt_ptr &&
(alt_ptr + i * sizeof(u32) + imm) < (alt_ptr + len))
continue;
riscv_alternative_fix_jal(alt_ptr + i * sizeof(u32),
insn, patch_offset);
}
RISC-V: fix auipc-jalr addresses in patched alternatives Alternatives live in a different section, so addresses used by call functions will point to wrong locations after the patch got applied. Similar to arm64, adjust the location to consider that offset. Reviewed-by: Conor Dooley <conor.dooley@microchip.com> Reviewed-by: Andrew Jones <ajones@ventanamicro.com> Reviewed-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Tested-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Signed-off-by: Heiko Stuebner <heiko.stuebner@vrull.eu> Link: https://lore.kernel.org/r/20221223221332.4127602-13-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-12-23 22:13:32 +00:00
}
}
riscv: Introduce alternative mechanism to apply errata solution Introduce the "alternative" mechanism from ARM64 and x86 to apply the CPU vendors' errata solution at runtime. The main purpose of this patch is to provide a framework. Therefore, the implementation is quite basic for now so that some scenarios could not use this schemei, such as patching code to a module, relocating the patching code and heterogeneous CPU topology. Users could use the macro ALTERNATIVE to apply an errata to the existing code flow. In the macro ALTERNATIVE, users need to specify the manufacturer information(vendorid, archid, and impid) for this errata. Therefore, kernel will know this errata is suitable for which CPU core. During the booting procedure, kernel will select the errata required by the CPU core and then patch it. It means that the kernel only applies the errata to the specified CPU core. In this case, the vendor's errata does not affect each other at runtime. The above patching procedure only occurs during the booting phase, so we only take the overhead of the "alternative" mechanism once. This "alternative" mechanism is enabled by default to ensure that all required errata will be applied. However, users can disable this feature by the Kconfig "CONFIG_RISCV_ERRATA_ALTERNATIVE". Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Reviewed-by: Anup Patel <anup@brainfault.org> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-03-22 14:26:03 +00:00
/*
* This is called very early in the boot process (directly after we run
* a feature detect on the boot CPU). No need to worry about other CPUs
* here.
*/
riscv: implement module alternatives This allows alternatives to also be applied when loading modules and follows the implementation of other architectures (e.g. arm64). Signed-off-by: Heiko Stuebner <heiko@sntech.de> Reviewed-by: Philipp Tomsich <philipp.tomsich@vrull.eu> Link: https://lore.kernel.org/r/20220511192921.2223629-4-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-05-11 19:29:12 +00:00
static void __init_or_module _apply_alternatives(struct alt_entry *begin,
struct alt_entry *end,
unsigned int stage)
riscv: allow different stages with alternatives Future features may need to be applied at a different time during boot, so allow defining stages for alternatives and handling them differently depending on the stage. Also make the alternatives-location more flexible so that future stages may provide their own location. Signed-off-by: Heiko Stuebner <heiko@sntech.de> Reviewed-by: Philipp Tomsich <philipp.tomsich@vrull.eu> Link: https://lore.kernel.org/r/20220511192921.2223629-3-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-05-11 19:29:11 +00:00
{
riscv: don't use global static vars to store alternative data Right now the code uses a global struct to store vendor-ids and another global variable to store the vendor-patch-function. There exist specific cases where we'll need to patch the kernel at an even earlier stage, where trying to write to a static variable might actually result in hangs. Also collecting the vendor-information consists of 3 sbi-ecalls (or csr-reads) which is pretty negligible in the context of booting a kernel. So rework the code to not rely on static variables and instead collect the vendor-information when a round of alternatives is to be applied. Signed-off-by: Heiko Stuebner <heiko@sntech.de> Reviewed-by: Guo Ren <guoren@kernel.org> Reviewed-by: Philipp Tomsich <philipp.tomsich@vrull.eu> Link: https://lore.kernel.org/r/20220511192921.2223629-12-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-05-11 19:29:20 +00:00
struct cpu_manufacturer_info_t cpu_mfr_info;
riscv_fill_cpu_mfr_info(&cpu_mfr_info);
riscv: add RISC-V Svpbmt extension support Svpbmt (the S should be capitalized) is the "Supervisor-mode: page-based memory types" extension that specifies attributes for cacheability, idempotency and ordering. The relevant settings are done in special bits in PTEs: Here is the svpbmt PTE format: | 63 | 62-61 | 60-8 | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 N MT RSW D A G U X W R V ^ Of the Reserved bits [63:54] in a leaf PTE, the high bit is already allocated (as the N bit), so bits [62:61] are used as the MT (aka MemType) field. This field specifies one of three memory types that are close equivalents (or equivalent in effect) to the three main x86 and ARMv8 memory types - as shown in the following table. RISC-V Encoding & MemType RISC-V Description ---------- ------------------------------------------------ 00 - PMA Normal Cacheable, No change to implied PMA memory type 01 - NC Non-cacheable, idempotent, weakly-ordered Main Memory 10 - IO Non-cacheable, non-idempotent, strongly-ordered I/O memory 11 - Rsvd Reserved for future standard use As the extension will not be present on all implementations, implement a method to handle cpufeatures via alternatives to not incur runtime penalties on cpu variants not supporting specific extensions and patch relevant code parts at runtime. Co-developed-by: Wei Fu <wefu@redhat.com> Signed-off-by: Wei Fu <wefu@redhat.com> Co-developed-by: Liu Shaohua <liush@allwinnertech.com> Signed-off-by: Liu Shaohua <liush@allwinnertech.com> Co-developed-by: Guo Ren <guoren@kernel.org> Signed-off-by: Guo Ren <guoren@kernel.org> [moved to use the alternatives mechanism] Signed-off-by: Heiko Stuebner <heiko@sntech.de> Reviewed-by: Philipp Tomsich <philipp.tomsich@vrull.eu> Link: https://lore.kernel.org/r/20220511192921.2223629-10-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-05-11 19:29:18 +00:00
riscv_cpufeature_patch_func(begin, end, stage);
riscv: make patch-function pointer more generic in cpu_manufacturer_info struct During review the naming of the function-pointer was called confusing as the vendor id is just one of three inputs for the patching and indeed it serves no real purpose, as with recent changes the function pointer is not a static global element anymore, so drop the "vendor_" prefix. Suggested-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Heiko Stuebner <heiko@sntech.de> Link: https://lore.kernel.org/r/20220526205646.258337-4-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-05-26 20:56:44 +00:00
if (!cpu_mfr_info.patch_func)
riscv: allow different stages with alternatives Future features may need to be applied at a different time during boot, so allow defining stages for alternatives and handling them differently depending on the stage. Also make the alternatives-location more flexible so that future stages may provide their own location. Signed-off-by: Heiko Stuebner <heiko@sntech.de> Reviewed-by: Philipp Tomsich <philipp.tomsich@vrull.eu> Link: https://lore.kernel.org/r/20220511192921.2223629-3-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-05-11 19:29:11 +00:00
return;
riscv: make patch-function pointer more generic in cpu_manufacturer_info struct During review the naming of the function-pointer was called confusing as the vendor id is just one of three inputs for the patching and indeed it serves no real purpose, as with recent changes the function pointer is not a static global element anymore, so drop the "vendor_" prefix. Suggested-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Heiko Stuebner <heiko@sntech.de> Link: https://lore.kernel.org/r/20220526205646.258337-4-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-05-26 20:56:44 +00:00
cpu_mfr_info.patch_func(begin, end,
cpu_mfr_info.arch_id,
cpu_mfr_info.imp_id,
stage);
riscv: allow different stages with alternatives Future features may need to be applied at a different time during boot, so allow defining stages for alternatives and handling them differently depending on the stage. Also make the alternatives-location more flexible so that future stages may provide their own location. Signed-off-by: Heiko Stuebner <heiko@sntech.de> Reviewed-by: Philipp Tomsich <philipp.tomsich@vrull.eu> Link: https://lore.kernel.org/r/20220511192921.2223629-3-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-05-11 19:29:11 +00:00
}
riscv: alternative: patch alternatives in the vDSO Make it possible to use alternatives in the vDSO, so that better implementations can be used if possible. Signed-off-by: Jisheng Zhang <jszhang@kernel.org> Reviewed-by: Guo Ren <guoren@kernel.org> Reviewed-by: Andrew Jones <ajones@ventanamicro.com> Link: https://lore.kernel.org/r/20230128172856.3814-11-jszhang@kernel.org Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2023-01-28 17:28:53 +00:00
#ifdef CONFIG_MMU
static void __init apply_vdso_alternatives(void)
{
const Elf_Ehdr *hdr;
const Elf_Shdr *shdr;
const Elf_Shdr *alt;
struct alt_entry *begin, *end;
hdr = (Elf_Ehdr *)vdso_start;
shdr = (void *)hdr + hdr->e_shoff;
alt = find_section(hdr, shdr, ".alternative");
if (!alt)
return;
begin = (void *)hdr + alt->sh_offset,
end = (void *)hdr + alt->sh_offset + alt->sh_size,
_apply_alternatives((struct alt_entry *)begin,
(struct alt_entry *)end,
RISCV_ALTERNATIVES_BOOT);
}
#else
static void __init apply_vdso_alternatives(void) { }
#endif
riscv: Introduce alternative mechanism to apply errata solution Introduce the "alternative" mechanism from ARM64 and x86 to apply the CPU vendors' errata solution at runtime. The main purpose of this patch is to provide a framework. Therefore, the implementation is quite basic for now so that some scenarios could not use this schemei, such as patching code to a module, relocating the patching code and heterogeneous CPU topology. Users could use the macro ALTERNATIVE to apply an errata to the existing code flow. In the macro ALTERNATIVE, users need to specify the manufacturer information(vendorid, archid, and impid) for this errata. Therefore, kernel will know this errata is suitable for which CPU core. During the booting procedure, kernel will select the errata required by the CPU core and then patch it. It means that the kernel only applies the errata to the specified CPU core. In this case, the vendor's errata does not affect each other at runtime. The above patching procedure only occurs during the booting phase, so we only take the overhead of the "alternative" mechanism once. This "alternative" mechanism is enabled by default to ensure that all required errata will be applied. However, users can disable this feature by the Kconfig "CONFIG_RISCV_ERRATA_ALTERNATIVE". Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Reviewed-by: Anup Patel <anup@brainfault.org> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-03-22 14:26:03 +00:00
void __init apply_boot_alternatives(void)
{
/* If called on non-boot cpu things could go wrong */
WARN_ON(smp_processor_id() != 0);
riscv: allow different stages with alternatives Future features may need to be applied at a different time during boot, so allow defining stages for alternatives and handling them differently depending on the stage. Also make the alternatives-location more flexible so that future stages may provide their own location. Signed-off-by: Heiko Stuebner <heiko@sntech.de> Reviewed-by: Philipp Tomsich <philipp.tomsich@vrull.eu> Link: https://lore.kernel.org/r/20220511192921.2223629-3-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-05-11 19:29:11 +00:00
_apply_alternatives((struct alt_entry *)__alt_start,
(struct alt_entry *)__alt_end,
RISCV_ALTERNATIVES_BOOT);
riscv: alternative: patch alternatives in the vDSO Make it possible to use alternatives in the vDSO, so that better implementations can be used if possible. Signed-off-by: Jisheng Zhang <jszhang@kernel.org> Reviewed-by: Guo Ren <guoren@kernel.org> Reviewed-by: Andrew Jones <ajones@ventanamicro.com> Link: https://lore.kernel.org/r/20230128172856.3814-11-jszhang@kernel.org Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2023-01-28 17:28:53 +00:00
apply_vdso_alternatives();
riscv: Introduce alternative mechanism to apply errata solution Introduce the "alternative" mechanism from ARM64 and x86 to apply the CPU vendors' errata solution at runtime. The main purpose of this patch is to provide a framework. Therefore, the implementation is quite basic for now so that some scenarios could not use this schemei, such as patching code to a module, relocating the patching code and heterogeneous CPU topology. Users could use the macro ALTERNATIVE to apply an errata to the existing code flow. In the macro ALTERNATIVE, users need to specify the manufacturer information(vendorid, archid, and impid) for this errata. Therefore, kernel will know this errata is suitable for which CPU core. During the booting procedure, kernel will select the errata required by the CPU core and then patch it. It means that the kernel only applies the errata to the specified CPU core. In this case, the vendor's errata does not affect each other at runtime. The above patching procedure only occurs during the booting phase, so we only take the overhead of the "alternative" mechanism once. This "alternative" mechanism is enabled by default to ensure that all required errata will be applied. However, users can disable this feature by the Kconfig "CONFIG_RISCV_ERRATA_ALTERNATIVE". Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Reviewed-by: Anup Patel <anup@brainfault.org> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-03-22 14:26:03 +00:00
}
riscv: implement module alternatives This allows alternatives to also be applied when loading modules and follows the implementation of other architectures (e.g. arm64). Signed-off-by: Heiko Stuebner <heiko@sntech.de> Reviewed-by: Philipp Tomsich <philipp.tomsich@vrull.eu> Link: https://lore.kernel.org/r/20220511192921.2223629-4-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-05-11 19:29:12 +00:00
riscv: add memory-type errata for T-Head Some current cpus based on T-Head cores implement memory-types way different than described in the svpbmt spec even going so far as using PTE bits marked as reserved. Add the T-Head vendor-id and necessary errata code to replace the affected instructions. Signed-off-by: Heiko Stuebner <heiko@sntech.de> Tested-by: Samuel Holland <samuel@sholland.org> Link: https://lore.kernel.org/r/20220511192921.2223629-13-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-05-11 19:29:21 +00:00
/*
* apply_early_boot_alternatives() is called from setup_vm() with MMU-off.
*
* Following requirements should be honoured for it to work correctly:
* 1) It should use PC-relative addressing for accessing kernel symbols.
* To achieve this we always use GCC cmodel=medany.
* 2) The compiler instrumentation for FTRACE will not work for setup_vm()
* so disable compiler instrumentation when FTRACE is enabled.
*
* Currently, the above requirements are honoured by using custom CFLAGS
* for alternative.o in kernel/Makefile.
*/
void __init apply_early_boot_alternatives(void)
{
#ifdef CONFIG_RISCV_ALTERNATIVE_EARLY
_apply_alternatives((struct alt_entry *)__alt_start,
(struct alt_entry *)__alt_end,
RISCV_ALTERNATIVES_EARLY_BOOT);
#endif
}
riscv: implement module alternatives This allows alternatives to also be applied when loading modules and follows the implementation of other architectures (e.g. arm64). Signed-off-by: Heiko Stuebner <heiko@sntech.de> Reviewed-by: Philipp Tomsich <philipp.tomsich@vrull.eu> Link: https://lore.kernel.org/r/20220511192921.2223629-4-heiko@sntech.de Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-05-11 19:29:12 +00:00
#ifdef CONFIG_MODULES
void apply_module_alternatives(void *start, size_t length)
{
_apply_alternatives((struct alt_entry *)start,
(struct alt_entry *)(start + length),
RISCV_ALTERNATIVES_MODULE);
}
#endif