License cleanup: add SPDX GPL-2.0 license identifier to files with no license
Many source files in the tree are missing licensing information, which
makes it harder for compliance tools to determine the correct license.
By default all files without license information are under the default
license of the kernel, which is GPL version 2.
Update the files which contain no license information with the 'GPL-2.0'
SPDX license identifier. The SPDX identifier is a legally binding
shorthand, which can be used instead of the full boiler plate text.
This patch is based on work done by Thomas Gleixner and Kate Stewart and
Philippe Ombredanne.
How this work was done:
Patches were generated and checked against linux-4.14-rc6 for a subset of
the use cases:
- file had no licensing information it it.
- file was a */uapi/* one with no licensing information in it,
- file was a */uapi/* one with existing licensing information,
Further patches will be generated in subsequent months to fix up cases
where non-standard license headers were used, and references to license
had to be inferred by heuristics based on keywords.
The analysis to determine which SPDX License Identifier to be applied to
a file was done in a spreadsheet of side by side results from of the
output of two independent scanners (ScanCode & Windriver) producing SPDX
tag:value files created by Philippe Ombredanne. Philippe prepared the
base worksheet, and did an initial spot review of a few 1000 files.
The 4.13 kernel was the starting point of the analysis with 60,537 files
assessed. Kate Stewart did a file by file comparison of the scanner
results in the spreadsheet to determine which SPDX license identifier(s)
to be applied to the file. She confirmed any determination that was not
immediately clear with lawyers working with the Linux Foundation.
Criteria used to select files for SPDX license identifier tagging was:
- Files considered eligible had to be source code files.
- Make and config files were included as candidates if they contained >5
lines of source
- File already had some variant of a license header in it (even if <5
lines).
All documentation files were explicitly excluded.
The following heuristics were used to determine which SPDX license
identifiers to apply.
- when both scanners couldn't find any license traces, file was
considered to have no license information in it, and the top level
COPYING file license applied.
For non */uapi/* files that summary was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 11139
and resulted in the first patch in this series.
If that file was a */uapi/* path one, it was "GPL-2.0 WITH
Linux-syscall-note" otherwise it was "GPL-2.0". Results of that was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 WITH Linux-syscall-note 930
and resulted in the second patch in this series.
- if a file had some form of licensing information in it, and was one
of the */uapi/* ones, it was denoted with the Linux-syscall-note if
any GPL family license was found in the file or had no licensing in
it (per prior point). Results summary:
SPDX license identifier # files
---------------------------------------------------|------
GPL-2.0 WITH Linux-syscall-note 270
GPL-2.0+ WITH Linux-syscall-note 169
((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) 21
((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) 17
LGPL-2.1+ WITH Linux-syscall-note 15
GPL-1.0+ WITH Linux-syscall-note 14
((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) 5
LGPL-2.0+ WITH Linux-syscall-note 4
LGPL-2.1 WITH Linux-syscall-note 3
((GPL-2.0 WITH Linux-syscall-note) OR MIT) 3
((GPL-2.0 WITH Linux-syscall-note) AND MIT) 1
and that resulted in the third patch in this series.
- when the two scanners agreed on the detected license(s), that became
the concluded license(s).
- when there was disagreement between the two scanners (one detected a
license but the other didn't, or they both detected different
licenses) a manual inspection of the file occurred.
- In most cases a manual inspection of the information in the file
resulted in a clear resolution of the license that should apply (and
which scanner probably needed to revisit its heuristics).
- When it was not immediately clear, the license identifier was
confirmed with lawyers working with the Linux Foundation.
- If there was any question as to the appropriate license identifier,
the file was flagged for further research and to be revisited later
in time.
In total, over 70 hours of logged manual review was done on the
spreadsheet to determine the SPDX license identifiers to apply to the
source files by Kate, Philippe, Thomas and, in some cases, confirmation
by lawyers working with the Linux Foundation.
Kate also obtained a third independent scan of the 4.13 code base from
FOSSology, and compared selected files where the other two scanners
disagreed against that SPDX file, to see if there was new insights. The
Windriver scanner is based on an older version of FOSSology in part, so
they are related.
Thomas did random spot checks in about 500 files from the spreadsheets
for the uapi headers and agreed with SPDX license identifier in the
files he inspected. For the non-uapi files Thomas did random spot checks
in about 15000 files.
In initial set of patches against 4.14-rc6, 3 files were found to have
copy/paste license identifier errors, and have been fixed to reflect the
correct identifier.
Additionally Philippe spent 10 hours this week doing a detailed manual
inspection and review of the 12,461 patched files from the initial patch
version early this week with:
- a full scancode scan run, collecting the matched texts, detected
license ids and scores
- reviewing anything where there was a license detected (about 500+
files) to ensure that the applied SPDX license was correct
- reviewing anything where there was no detection but the patch license
was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied
SPDX license was correct
This produced a worksheet with 20 files needing minor correction. This
worksheet was then exported into 3 different .csv files for the
different types of files to be modified.
These .csv files were then reviewed by Greg. Thomas wrote a script to
parse the csv files and add the proper SPDX tag to the file, in the
format that the file expected. This script was further refined by Greg
based on the output to detect more types of files automatically and to
distinguish between header and source .c files (which need different
comment types.) Finally Greg ran the script using the .csv files to
generate the patches.
Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Philippe Ombredanne <pombredanne@nexb.com>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2017-11-01 14:07:57 +00:00
|
|
|
/* SPDX-License-Identifier: GPL-2.0 */
|
2005-04-16 22:20:36 +00:00
|
|
|
/*
|
2015-01-27 18:16:28 +00:00
|
|
|
* linux/arch/x86/kernel/head_64.S -- start in 32bit and switch to 64bit
|
2005-04-16 22:20:36 +00:00
|
|
|
*
|
|
|
|
|
* Copyright (C) 2000 Andrea Arcangeli <andrea@suse.de> SuSE
|
|
|
|
|
* Copyright (C) 2000 Pavel Machek <pavel@suse.cz>
|
|
|
|
|
* Copyright (C) 2000 Karsten Keil <kkeil@suse.de>
|
|
|
|
|
* Copyright (C) 2001,2002 Andi Kleen <ak@suse.de>
|
2007-05-02 17:27:07 +00:00
|
|
|
* Copyright (C) 2005 Eric Biederman <ebiederm@xmission.com>
|
2005-04-16 22:20:36 +00:00
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
#include <linux/linkage.h>
|
|
|
|
|
#include <linux/threads.h>
|
2005-11-05 16:25:53 +00:00
|
|
|
#include <linux/init.h>
|
2020-06-09 04:32:38 +00:00
|
|
|
#include <linux/pgtable.h>
|
2020-06-09 04:32:42 +00:00
|
|
|
#include <asm/segment.h>
|
2005-04-16 22:20:36 +00:00
|
|
|
#include <asm/page.h>
|
|
|
|
|
#include <asm/msr.h>
|
|
|
|
|
#include <asm/cache.h>
|
2008-05-12 13:43:38 +00:00
|
|
|
#include <asm/processor-flags.h>
|
2009-01-13 11:41:35 +00:00
|
|
|
#include <asm/percpu.h>
|
2012-04-19 00:16:49 +00:00
|
|
|
#include <asm/nops.h>
|
2016-04-02 14:01:32 +00:00
|
|
|
#include "../entry/calling.h"
|
2016-01-11 16:04:34 +00:00
|
|
|
#include <asm/export.h>
|
2018-01-16 09:38:09 +00:00
|
|
|
#include <asm/nospec-branch.h>
|
2018-09-20 02:58:28 +00:00
|
|
|
#include <asm/fixmap.h>
|
2007-05-02 17:27:07 +00:00
|
|
|
|
2018-08-28 07:40:25 +00:00
|
|
|
#ifdef CONFIG_PARAVIRT_XXL
|
2008-01-30 12:31:10 +00:00
|
|
|
#include <asm/asm-offsets.h>
|
|
|
|
|
#include <asm/paravirt.h>
|
2020-05-21 20:05:46 +00:00
|
|
|
#define GET_CR2_INTO(reg) GET_CR2_INTO_AX ; _ASM_MOV %_ASM_AX, reg
|
2008-01-30 12:31:10 +00:00
|
|
|
#else
|
2012-04-19 00:16:49 +00:00
|
|
|
#define INTERRUPT_RETURN iretq
|
2020-05-21 20:05:46 +00:00
|
|
|
#define GET_CR2_INTO(reg) _ASM_MOV %cr2, reg
|
2008-01-30 12:31:10 +00:00
|
|
|
#endif
|
|
|
|
|
|
2020-05-21 20:05:46 +00:00
|
|
|
/*
|
|
|
|
|
* We are not able to switch in one step to the final KERNEL ADDRESS SPACE
|
2007-05-02 17:27:07 +00:00
|
|
|
* because we need identity-mapped pages.
|
2005-04-16 22:20:36 +00:00
|
|
|
*/
|
2018-02-16 11:49:46 +00:00
|
|
|
#define l4_index(x) (((x) >> 39) & 511)
|
2008-06-25 04:19:16 +00:00
|
|
|
#define pud_index(x) (((x) >> PUD_SHIFT) & (PTRS_PER_PUD-1))
|
|
|
|
|
|
2018-02-16 11:49:46 +00:00
|
|
|
L4_PAGE_OFFSET = l4_index(__PAGE_OFFSET_BASE_L4)
|
|
|
|
|
L4_START_KERNEL = l4_index(__START_KERNEL_map)
|
|
|
|
|
|
2008-06-25 04:19:16 +00:00
|
|
|
L3_START_KERNEL = pud_index(__START_KERNEL_map)
|
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
.text
|
2009-09-16 20:44:28 +00:00
|
|
|
__HEAD
|
2007-05-02 17:27:07 +00:00
|
|
|
.code64
|
2019-10-11 11:50:56 +00:00
|
|
|
SYM_CODE_START_NOALIGN(startup_64)
|
2017-09-19 02:43:37 +00:00
|
|
|
UNWIND_HINT_EMPTY
|
2005-04-16 22:20:36 +00:00
|
|
|
/*
|
2013-02-25 20:54:10 +00:00
|
|
|
* At this point the CPU runs in 64bit mode CS.L = 1 CS.D = 0,
|
2007-05-02 17:27:07 +00:00
|
|
|
* and someone has loaded an identity mapped page table
|
|
|
|
|
* for us. These identity mapped page tables map all of the
|
|
|
|
|
* kernel pages and possibly all of memory.
|
|
|
|
|
*
|
x86, 64bit: Use a #PF handler to materialize early mappings on demand
Linear mode (CR0.PG = 0) is mutually exclusive with 64-bit mode; all
64-bit code has to use page tables. This makes it awkward before we
have first set up properly all-covering page tables to access objects
that are outside the static kernel range.
So far we have dealt with that simply by mapping a fixed amount of
low memory, but that fails in at least two upcoming use cases:
1. We will support load and run kernel, struct boot_params, ramdisk,
command line, etc. above the 4 GiB mark.
2. need to access ramdisk early to get microcode to update that as
early possible.
We could use early_iomap to access them too, but it will make code to
messy and hard to be unified with 32 bit.
Hence, set up a #PF table and use a fixed number of buffers to set up
page tables on demand. If the buffers fill up then we simply flush
them and start over. These buffers are all in __initdata, so it does
not increase RAM usage at runtime.
Thus, with the help of the #PF handler, we can set the final kernel
mapping from blank, and switch to init_level4_pgt later.
During the switchover in head_64.S, before #PF handler is available,
we use three pages to handle kernel crossing 1G, 512G boundaries with
sharing page by playing games with page aliasing: the same page is
mapped twice in the higher-level tables with appropriate wraparound.
The kernel region itself will be properly mapped; other mappings may
be spurious.
early_make_pgtable is using kernel high mapping address to access pages
to set page table.
-v4: Add phys_base offset to make kexec happy, and add
init_mapping_kernel() - Yinghai
-v5: fix compiling with xen, and add back ident level3 and level2 for xen
also move back init_level4_pgt from BSS to DATA again.
because we have to clear it anyway. - Yinghai
-v6: switch to init_level4_pgt in init_mem_mapping. - Yinghai
-v7: remove not needed clear_page for init_level4_page
it is with fill 512,8,0 already in head_64.S - Yinghai
-v8: we need to keep that handler alive until init_mem_mapping and don't
let early_trap_init to trash that early #PF handler.
So split early_trap_pf_init out and move it down. - Yinghai
-v9: switchover only cover kernel space instead of 1G so could avoid
touch possible mem holes. - Yinghai
-v11: change far jmp back to far return to initial_code, that is needed
to fix failure that is reported by Konrad on AMD systems. - Yinghai
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Link: http://lkml.kernel.org/r/1359058816-7615-12-git-send-email-yinghai@kernel.org
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
2013-01-24 20:19:52 +00:00
|
|
|
* %rsi holds a physical pointer to real_mode_data.
|
2007-05-02 17:27:07 +00:00
|
|
|
*
|
|
|
|
|
* We come here either directly from a 64bit bootloader, or from
|
2015-01-27 18:16:28 +00:00
|
|
|
* arch/x86/boot/compressed/head_64.S.
|
2007-05-02 17:27:07 +00:00
|
|
|
*
|
|
|
|
|
* We only come here initially at boot nothing else comes here.
|
|
|
|
|
*
|
|
|
|
|
* Since we may be loaded at an address different from what we were
|
|
|
|
|
* compiled to run at we first fixup the physical addresses in our page
|
|
|
|
|
* tables and then reload them.
|
2005-04-16 22:20:36 +00:00
|
|
|
*/
|
|
|
|
|
|
2016-09-21 21:04:06 +00:00
|
|
|
/* Set up the stack for verify_cpu(), similar to initial_stack below */
|
|
|
|
|
leaq (__end_init_task - SIZEOF_PTREGS)(%rip), %rsp
|
2016-03-31 14:21:02 +00:00
|
|
|
|
2020-09-07 13:15:30 +00:00
|
|
|
leaq _text(%rip), %rdi
|
x86/head/64: Re-enable stack protection
commit 469693d8f62299709e8ba56d8fb3da9ea990213c upstream.
Due to
103a4908ad4d ("x86/head/64: Disable stack protection for head$(BITS).o")
kernel/head{32,64}.c are compiled with -fno-stack-protector to allow
a call to set_bringup_idt_handler(), which would otherwise have stack
protection enabled with CONFIG_STACKPROTECTOR_STRONG.
While sufficient for that case, there may still be issues with calls to
any external functions that were compiled with stack protection enabled
that in-turn make stack-protected calls, or if the exception handlers
set up by set_bringup_idt_handler() make calls to stack-protected
functions.
Subsequent patches for SEV-SNP CPUID validation support will introduce
both such cases. Attempting to disable stack protection for everything
in scope to address that is prohibitive since much of the code, like the
SEV-ES #VC handler, is shared code that remains in use after boot and
could benefit from having stack protection enabled. Attempting to inline
calls is brittle and can quickly balloon out to library/helper code
where that's not really an option.
Instead, re-enable stack protection for head32.c/head64.c, and make the
appropriate changes to ensure the segment used for the stack canary is
initialized in advance of any stack-protected C calls.
For head64.c:
- The BSP will enter from startup_64() and call into C code
(startup_64_setup_env()) shortly after setting up the stack, which
may result in calls to stack-protected code. Set up %gs early to allow
for this safely.
- APs will enter from secondary_startup_64*(), and %gs will be set up
soon after. There is one call to C code prior to %gs being setup
(__startup_secondary_64()), but it is only to fetch 'sme_me_mask'
global, so just load 'sme_me_mask' directly instead, and remove the
now-unused __startup_secondary_64() function.
For head32.c:
- BSPs/APs will set %fs to __BOOT_DS prior to any C calls. In recent
kernels, the compiler is configured to access the stack canary at
%fs:__stack_chk_guard [1], which overlaps with the initial per-cpu
'__stack_chk_guard' variable in the initial/"master" .data..percpu
area. This is sufficient to allow access to the canary for use
during initial startup, so no changes are needed there.
[1] 3fb0fdb3bbe7 ("x86/stackprotector/32: Make the canary into a regular percpu variable")
[ bp: Massage commit message. ]
Suggested-by: Joerg Roedel <jroedel@suse.de> #for 64-bit %gs set up
Signed-off-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Brijesh Singh <brijesh.singh@amd.com>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lore.kernel.org/r/20220307213356.2797205-24-brijesh.singh@amd.com
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2022-02-09 18:10:17 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* initial_gs points to initial fixed_percpu_data struct with storage for
|
|
|
|
|
* the stack protector canary. Global pointer fixups are needed at this
|
|
|
|
|
* stage, so apply them as is done in fixup_pointer(), and initialize %gs
|
|
|
|
|
* such that the canary can be accessed at %gs:40 for subsequent C calls.
|
|
|
|
|
*/
|
|
|
|
|
movl $MSR_GS_BASE, %ecx
|
|
|
|
|
movq initial_gs(%rip), %rax
|
|
|
|
|
movq $_text, %rdx
|
|
|
|
|
subq %rdx, %rax
|
|
|
|
|
addq %rdi, %rax
|
|
|
|
|
movq %rax, %rdx
|
|
|
|
|
shrq $32, %rdx
|
|
|
|
|
wrmsr
|
|
|
|
|
|
2020-09-07 13:15:30 +00:00
|
|
|
pushq %rsi
|
|
|
|
|
call startup_64_setup_env
|
|
|
|
|
popq %rsi
|
|
|
|
|
|
|
|
|
|
/* Now switch to __KERNEL_CS so IRET works reliably */
|
|
|
|
|
pushq $__KERNEL_CS
|
|
|
|
|
leaq .Lon_kernel_cs(%rip), %rax
|
|
|
|
|
pushq %rax
|
|
|
|
|
lretq
|
|
|
|
|
|
|
|
|
|
.Lon_kernel_cs:
|
|
|
|
|
UNWIND_HINT_EMPTY
|
|
|
|
|
|
2015-11-05 15:57:56 +00:00
|
|
|
/* Sanitize CPU configuration */
|
|
|
|
|
call verify_cpu
|
|
|
|
|
|
2017-07-17 21:10:05 +00:00
|
|
|
/*
|
|
|
|
|
* Perform pagetable fixups. Additionally, if SME is active, encrypt
|
|
|
|
|
* the kernel and retrieve the modifier (SME encryption mask if SME
|
|
|
|
|
* is active) to be added to the initial pgdir entry that will be
|
|
|
|
|
* programmed into CR3.
|
|
|
|
|
*/
|
2007-05-02 17:27:07 +00:00
|
|
|
leaq _text(%rip), %rdi
|
2017-06-06 11:31:26 +00:00
|
|
|
pushq %rsi
|
|
|
|
|
call __startup_64
|
|
|
|
|
popq %rsi
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2017-07-17 21:10:05 +00:00
|
|
|
/* Form the CR3 value being sure to include the CR3 modifier */
|
|
|
|
|
addq $(early_top_pgt - __START_KERNEL_map), %rax
|
x86, 64bit: Use a #PF handler to materialize early mappings on demand
Linear mode (CR0.PG = 0) is mutually exclusive with 64-bit mode; all
64-bit code has to use page tables. This makes it awkward before we
have first set up properly all-covering page tables to access objects
that are outside the static kernel range.
So far we have dealt with that simply by mapping a fixed amount of
low memory, but that fails in at least two upcoming use cases:
1. We will support load and run kernel, struct boot_params, ramdisk,
command line, etc. above the 4 GiB mark.
2. need to access ramdisk early to get microcode to update that as
early possible.
We could use early_iomap to access them too, but it will make code to
messy and hard to be unified with 32 bit.
Hence, set up a #PF table and use a fixed number of buffers to set up
page tables on demand. If the buffers fill up then we simply flush
them and start over. These buffers are all in __initdata, so it does
not increase RAM usage at runtime.
Thus, with the help of the #PF handler, we can set the final kernel
mapping from blank, and switch to init_level4_pgt later.
During the switchover in head_64.S, before #PF handler is available,
we use three pages to handle kernel crossing 1G, 512G boundaries with
sharing page by playing games with page aliasing: the same page is
mapped twice in the higher-level tables with appropriate wraparound.
The kernel region itself will be properly mapped; other mappings may
be spurious.
early_make_pgtable is using kernel high mapping address to access pages
to set page table.
-v4: Add phys_base offset to make kexec happy, and add
init_mapping_kernel() - Yinghai
-v5: fix compiling with xen, and add back ident level3 and level2 for xen
also move back init_level4_pgt from BSS to DATA again.
because we have to clear it anyway. - Yinghai
-v6: switch to init_level4_pgt in init_mem_mapping. - Yinghai
-v7: remove not needed clear_page for init_level4_page
it is with fill 512,8,0 already in head_64.S - Yinghai
-v8: we need to keep that handler alive until init_mem_mapping and don't
let early_trap_init to trash that early #PF handler.
So split early_trap_pf_init out and move it down. - Yinghai
-v9: switchover only cover kernel space instead of 1G so could avoid
touch possible mem holes. - Yinghai
-v11: change far jmp back to far return to initial_code, that is needed
to fix failure that is reported by Konrad on AMD systems. - Yinghai
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Link: http://lkml.kernel.org/r/1359058816-7615-12-git-send-email-yinghai@kernel.org
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
2013-01-24 20:19:52 +00:00
|
|
|
jmp 1f
|
2019-10-11 11:50:56 +00:00
|
|
|
SYM_CODE_END(startup_64)
|
|
|
|
|
|
2019-10-11 11:51:03 +00:00
|
|
|
SYM_CODE_START(secondary_startup_64)
|
2017-09-19 02:43:37 +00:00
|
|
|
UNWIND_HINT_EMPTY
|
2007-05-02 17:27:07 +00:00
|
|
|
/*
|
2013-02-25 20:54:10 +00:00
|
|
|
* At this point the CPU runs in 64bit mode CS.L = 1 CS.D = 0,
|
2007-05-02 17:27:07 +00:00
|
|
|
* and someone has loaded a mapped page table.
|
|
|
|
|
*
|
x86, 64bit: Use a #PF handler to materialize early mappings on demand
Linear mode (CR0.PG = 0) is mutually exclusive with 64-bit mode; all
64-bit code has to use page tables. This makes it awkward before we
have first set up properly all-covering page tables to access objects
that are outside the static kernel range.
So far we have dealt with that simply by mapping a fixed amount of
low memory, but that fails in at least two upcoming use cases:
1. We will support load and run kernel, struct boot_params, ramdisk,
command line, etc. above the 4 GiB mark.
2. need to access ramdisk early to get microcode to update that as
early possible.
We could use early_iomap to access them too, but it will make code to
messy and hard to be unified with 32 bit.
Hence, set up a #PF table and use a fixed number of buffers to set up
page tables on demand. If the buffers fill up then we simply flush
them and start over. These buffers are all in __initdata, so it does
not increase RAM usage at runtime.
Thus, with the help of the #PF handler, we can set the final kernel
mapping from blank, and switch to init_level4_pgt later.
During the switchover in head_64.S, before #PF handler is available,
we use three pages to handle kernel crossing 1G, 512G boundaries with
sharing page by playing games with page aliasing: the same page is
mapped twice in the higher-level tables with appropriate wraparound.
The kernel region itself will be properly mapped; other mappings may
be spurious.
early_make_pgtable is using kernel high mapping address to access pages
to set page table.
-v4: Add phys_base offset to make kexec happy, and add
init_mapping_kernel() - Yinghai
-v5: fix compiling with xen, and add back ident level3 and level2 for xen
also move back init_level4_pgt from BSS to DATA again.
because we have to clear it anyway. - Yinghai
-v6: switch to init_level4_pgt in init_mem_mapping. - Yinghai
-v7: remove not needed clear_page for init_level4_page
it is with fill 512,8,0 already in head_64.S - Yinghai
-v8: we need to keep that handler alive until init_mem_mapping and don't
let early_trap_init to trash that early #PF handler.
So split early_trap_pf_init out and move it down. - Yinghai
-v9: switchover only cover kernel space instead of 1G so could avoid
touch possible mem holes. - Yinghai
-v11: change far jmp back to far return to initial_code, that is needed
to fix failure that is reported by Konrad on AMD systems. - Yinghai
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Link: http://lkml.kernel.org/r/1359058816-7615-12-git-send-email-yinghai@kernel.org
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
2013-01-24 20:19:52 +00:00
|
|
|
* %rsi holds a physical pointer to real_mode_data.
|
2007-05-02 17:27:07 +00:00
|
|
|
*
|
|
|
|
|
* We come here either from startup_64 (using physical addresses)
|
|
|
|
|
* or from trampoline.S (using virtual addresses).
|
|
|
|
|
*
|
|
|
|
|
* Using virtual addresses from trampoline.S removes the need
|
|
|
|
|
* to have any identity mapped pages in the kernel page table
|
|
|
|
|
* after the boot processor executes this code.
|
2005-04-16 22:20:36 +00:00
|
|
|
*/
|
|
|
|
|
|
2015-11-05 15:57:56 +00:00
|
|
|
/* Sanitize CPU configuration */
|
|
|
|
|
call verify_cpu
|
|
|
|
|
|
2020-09-07 13:16:09 +00:00
|
|
|
/*
|
|
|
|
|
* The secondary_startup_64_no_verify entry point is only used by
|
|
|
|
|
* SEV-ES guests. In those guests the call to verify_cpu() would cause
|
|
|
|
|
* #VC exceptions which can not be handled at this stage of secondary
|
|
|
|
|
* CPU bringup.
|
|
|
|
|
*
|
|
|
|
|
* All non SEV-ES systems, especially Intel systems, need to execute
|
|
|
|
|
* verify_cpu() above to make sure NX is enabled.
|
|
|
|
|
*/
|
|
|
|
|
SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL)
|
|
|
|
|
UNWIND_HINT_EMPTY
|
|
|
|
|
|
2017-07-17 21:10:05 +00:00
|
|
|
/*
|
|
|
|
|
* Retrieve the modifier (SME encryption mask if SME is active) to be
|
|
|
|
|
* added to the initial pgdir entry that will be programmed into CR3.
|
|
|
|
|
*/
|
x86/head/64: Re-enable stack protection
commit 469693d8f62299709e8ba56d8fb3da9ea990213c upstream.
Due to
103a4908ad4d ("x86/head/64: Disable stack protection for head$(BITS).o")
kernel/head{32,64}.c are compiled with -fno-stack-protector to allow
a call to set_bringup_idt_handler(), which would otherwise have stack
protection enabled with CONFIG_STACKPROTECTOR_STRONG.
While sufficient for that case, there may still be issues with calls to
any external functions that were compiled with stack protection enabled
that in-turn make stack-protected calls, or if the exception handlers
set up by set_bringup_idt_handler() make calls to stack-protected
functions.
Subsequent patches for SEV-SNP CPUID validation support will introduce
both such cases. Attempting to disable stack protection for everything
in scope to address that is prohibitive since much of the code, like the
SEV-ES #VC handler, is shared code that remains in use after boot and
could benefit from having stack protection enabled. Attempting to inline
calls is brittle and can quickly balloon out to library/helper code
where that's not really an option.
Instead, re-enable stack protection for head32.c/head64.c, and make the
appropriate changes to ensure the segment used for the stack canary is
initialized in advance of any stack-protected C calls.
For head64.c:
- The BSP will enter from startup_64() and call into C code
(startup_64_setup_env()) shortly after setting up the stack, which
may result in calls to stack-protected code. Set up %gs early to allow
for this safely.
- APs will enter from secondary_startup_64*(), and %gs will be set up
soon after. There is one call to C code prior to %gs being setup
(__startup_secondary_64()), but it is only to fetch 'sme_me_mask'
global, so just load 'sme_me_mask' directly instead, and remove the
now-unused __startup_secondary_64() function.
For head32.c:
- BSPs/APs will set %fs to __BOOT_DS prior to any C calls. In recent
kernels, the compiler is configured to access the stack canary at
%fs:__stack_chk_guard [1], which overlaps with the initial per-cpu
'__stack_chk_guard' variable in the initial/"master" .data..percpu
area. This is sufficient to allow access to the canary for use
during initial startup, so no changes are needed there.
[1] 3fb0fdb3bbe7 ("x86/stackprotector/32: Make the canary into a regular percpu variable")
[ bp: Massage commit message. ]
Suggested-by: Joerg Roedel <jroedel@suse.de> #for 64-bit %gs set up
Signed-off-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Brijesh Singh <brijesh.singh@amd.com>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lore.kernel.org/r/20220307213356.2797205-24-brijesh.singh@amd.com
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2022-02-09 18:10:17 +00:00
|
|
|
#ifdef CONFIG_AMD_MEM_ENCRYPT
|
|
|
|
|
movq sme_me_mask, %rax
|
|
|
|
|
#else
|
|
|
|
|
xorq %rax, %rax
|
|
|
|
|
#endif
|
2017-07-17 21:10:05 +00:00
|
|
|
|
|
|
|
|
/* Form the CR3 value being sure to include the CR3 modifier */
|
|
|
|
|
addq $(init_top_pgt - __START_KERNEL_map), %rax
|
x86, 64bit: Use a #PF handler to materialize early mappings on demand
Linear mode (CR0.PG = 0) is mutually exclusive with 64-bit mode; all
64-bit code has to use page tables. This makes it awkward before we
have first set up properly all-covering page tables to access objects
that are outside the static kernel range.
So far we have dealt with that simply by mapping a fixed amount of
low memory, but that fails in at least two upcoming use cases:
1. We will support load and run kernel, struct boot_params, ramdisk,
command line, etc. above the 4 GiB mark.
2. need to access ramdisk early to get microcode to update that as
early possible.
We could use early_iomap to access them too, but it will make code to
messy and hard to be unified with 32 bit.
Hence, set up a #PF table and use a fixed number of buffers to set up
page tables on demand. If the buffers fill up then we simply flush
them and start over. These buffers are all in __initdata, so it does
not increase RAM usage at runtime.
Thus, with the help of the #PF handler, we can set the final kernel
mapping from blank, and switch to init_level4_pgt later.
During the switchover in head_64.S, before #PF handler is available,
we use three pages to handle kernel crossing 1G, 512G boundaries with
sharing page by playing games with page aliasing: the same page is
mapped twice in the higher-level tables with appropriate wraparound.
The kernel region itself will be properly mapped; other mappings may
be spurious.
early_make_pgtable is using kernel high mapping address to access pages
to set page table.
-v4: Add phys_base offset to make kexec happy, and add
init_mapping_kernel() - Yinghai
-v5: fix compiling with xen, and add back ident level3 and level2 for xen
also move back init_level4_pgt from BSS to DATA again.
because we have to clear it anyway. - Yinghai
-v6: switch to init_level4_pgt in init_mem_mapping. - Yinghai
-v7: remove not needed clear_page for init_level4_page
it is with fill 512,8,0 already in head_64.S - Yinghai
-v8: we need to keep that handler alive until init_mem_mapping and don't
let early_trap_init to trash that early #PF handler.
So split early_trap_pf_init out and move it down. - Yinghai
-v9: switchover only cover kernel space instead of 1G so could avoid
touch possible mem holes. - Yinghai
-v11: change far jmp back to far return to initial_code, that is needed
to fix failure that is reported by Konrad on AMD systems. - Yinghai
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Link: http://lkml.kernel.org/r/1359058816-7615-12-git-send-email-yinghai@kernel.org
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
2013-01-24 20:19:52 +00:00
|
|
|
1:
|
|
|
|
|
|
2017-06-06 11:31:28 +00:00
|
|
|
/* Enable PAE mode, PGE and LA57 */
|
x86, 64bit: Use a #PF handler to materialize early mappings on demand
Linear mode (CR0.PG = 0) is mutually exclusive with 64-bit mode; all
64-bit code has to use page tables. This makes it awkward before we
have first set up properly all-covering page tables to access objects
that are outside the static kernel range.
So far we have dealt with that simply by mapping a fixed amount of
low memory, but that fails in at least two upcoming use cases:
1. We will support load and run kernel, struct boot_params, ramdisk,
command line, etc. above the 4 GiB mark.
2. need to access ramdisk early to get microcode to update that as
early possible.
We could use early_iomap to access them too, but it will make code to
messy and hard to be unified with 32 bit.
Hence, set up a #PF table and use a fixed number of buffers to set up
page tables on demand. If the buffers fill up then we simply flush
them and start over. These buffers are all in __initdata, so it does
not increase RAM usage at runtime.
Thus, with the help of the #PF handler, we can set the final kernel
mapping from blank, and switch to init_level4_pgt later.
During the switchover in head_64.S, before #PF handler is available,
we use three pages to handle kernel crossing 1G, 512G boundaries with
sharing page by playing games with page aliasing: the same page is
mapped twice in the higher-level tables with appropriate wraparound.
The kernel region itself will be properly mapped; other mappings may
be spurious.
early_make_pgtable is using kernel high mapping address to access pages
to set page table.
-v4: Add phys_base offset to make kexec happy, and add
init_mapping_kernel() - Yinghai
-v5: fix compiling with xen, and add back ident level3 and level2 for xen
also move back init_level4_pgt from BSS to DATA again.
because we have to clear it anyway. - Yinghai
-v6: switch to init_level4_pgt in init_mem_mapping. - Yinghai
-v7: remove not needed clear_page for init_level4_page
it is with fill 512,8,0 already in head_64.S - Yinghai
-v8: we need to keep that handler alive until init_mem_mapping and don't
let early_trap_init to trash that early #PF handler.
So split early_trap_pf_init out and move it down. - Yinghai
-v9: switchover only cover kernel space instead of 1G so could avoid
touch possible mem holes. - Yinghai
-v11: change far jmp back to far return to initial_code, that is needed
to fix failure that is reported by Konrad on AMD systems. - Yinghai
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Link: http://lkml.kernel.org/r/1359058816-7615-12-git-send-email-yinghai@kernel.org
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
2013-01-24 20:19:52 +00:00
|
|
|
movl $(X86_CR4_PAE | X86_CR4_PGE), %ecx
|
2017-06-06 11:31:28 +00:00
|
|
|
#ifdef CONFIG_X86_5LEVEL
|
x86/mm: Optimize boot-time paging mode switching cost
By this point we have functioning boot-time switching between 4- and
5-level paging mode. But naive approach comes with cost.
Numbers below are for kernel build, allmodconfig, 5 times.
CONFIG_X86_5LEVEL=n:
Performance counter stats for 'sh -c make -j100 -B -k >/dev/null' (5 runs):
17308719.892691 task-clock:u (msec) # 26.772 CPUs utilized ( +- 0.11% )
0 context-switches:u # 0.000 K/sec
0 cpu-migrations:u # 0.000 K/sec
331,993,164 page-faults:u # 0.019 M/sec ( +- 0.01% )
43,614,978,867,455 cycles:u # 2.520 GHz ( +- 0.01% )
39,371,534,575,126 stalled-cycles-frontend:u # 90.27% frontend cycles idle ( +- 0.09% )
28,363,350,152,428 instructions:u # 0.65 insn per cycle
# 1.39 stalled cycles per insn ( +- 0.00% )
6,316,784,066,413 branches:u # 364.948 M/sec ( +- 0.00% )
250,808,144,781 branch-misses:u # 3.97% of all branches ( +- 0.01% )
646.531974142 seconds time elapsed ( +- 1.15% )
CONFIG_X86_5LEVEL=y:
Performance counter stats for 'sh -c make -j100 -B -k >/dev/null' (5 runs):
17411536.780625 task-clock:u (msec) # 26.426 CPUs utilized ( +- 0.10% )
0 context-switches:u # 0.000 K/sec
0 cpu-migrations:u # 0.000 K/sec
331,868,663 page-faults:u # 0.019 M/sec ( +- 0.01% )
43,865,909,056,301 cycles:u # 2.519 GHz ( +- 0.01% )
39,740,130,365,581 stalled-cycles-frontend:u # 90.59% frontend cycles idle ( +- 0.05% )
28,363,358,997,959 instructions:u # 0.65 insn per cycle
# 1.40 stalled cycles per insn ( +- 0.00% )
6,316,784,937,460 branches:u # 362.793 M/sec ( +- 0.00% )
251,531,919,485 branch-misses:u # 3.98% of all branches ( +- 0.00% )
658.886307752 seconds time elapsed ( +- 0.92% )
The patch tries to fix the performance regression by using
cpu_feature_enabled(X86_FEATURE_LA57) instead of pgtable_l5_enabled in
all hot code paths. These will statically patch the target code for
additional performance.
CONFIG_X86_5LEVEL=y + the patch:
Performance counter stats for 'sh -c make -j100 -B -k >/dev/null' (5 runs):
17381990.268506 task-clock:u (msec) # 26.907 CPUs utilized ( +- 0.19% )
0 context-switches:u # 0.000 K/sec
0 cpu-migrations:u # 0.000 K/sec
331,862,625 page-faults:u # 0.019 M/sec ( +- 0.01% )
43,697,726,320,051 cycles:u # 2.514 GHz ( +- 0.03% )
39,480,408,690,401 stalled-cycles-frontend:u # 90.35% frontend cycles idle ( +- 0.05% )
28,363,394,221,388 instructions:u # 0.65 insn per cycle
# 1.39 stalled cycles per insn ( +- 0.00% )
6,316,794,985,573 branches:u # 363.410 M/sec ( +- 0.00% )
251,013,232,547 branch-misses:u # 3.97% of all branches ( +- 0.01% )
645.991174661 seconds time elapsed ( +- 1.19% )
Unfortunately, this approach doesn't help with text size:
vmlinux.before .text size: 8190319
vmlinux.after .text size: 8200623
The .text section is increased by about 4k. Not sure if we can do anything
about this.
Signed-off-by: Kirill A. Shuemov <kirill.shutemov@linux.intel.com>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Arjan van de Ven <arjan@linux.intel.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Borislav Petkov <bp@suse.de>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: David Woodhouse <dwmw2@infradead.org>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-mm@kvack.org
Link: http://lkml.kernel.org/r/20180216114948.68868-4-kirill.shutemov@linux.intel.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-02-16 11:49:48 +00:00
|
|
|
testl $1, __pgtable_l5_enabled(%rip)
|
2018-02-14 18:25:39 +00:00
|
|
|
jz 1f
|
2017-06-06 11:31:28 +00:00
|
|
|
orl $X86_CR4_LA57, %ecx
|
2018-02-14 18:25:39 +00:00
|
|
|
1:
|
2017-06-06 11:31:28 +00:00
|
|
|
#endif
|
x86, 64bit: Use a #PF handler to materialize early mappings on demand
Linear mode (CR0.PG = 0) is mutually exclusive with 64-bit mode; all
64-bit code has to use page tables. This makes it awkward before we
have first set up properly all-covering page tables to access objects
that are outside the static kernel range.
So far we have dealt with that simply by mapping a fixed amount of
low memory, but that fails in at least two upcoming use cases:
1. We will support load and run kernel, struct boot_params, ramdisk,
command line, etc. above the 4 GiB mark.
2. need to access ramdisk early to get microcode to update that as
early possible.
We could use early_iomap to access them too, but it will make code to
messy and hard to be unified with 32 bit.
Hence, set up a #PF table and use a fixed number of buffers to set up
page tables on demand. If the buffers fill up then we simply flush
them and start over. These buffers are all in __initdata, so it does
not increase RAM usage at runtime.
Thus, with the help of the #PF handler, we can set the final kernel
mapping from blank, and switch to init_level4_pgt later.
During the switchover in head_64.S, before #PF handler is available,
we use three pages to handle kernel crossing 1G, 512G boundaries with
sharing page by playing games with page aliasing: the same page is
mapped twice in the higher-level tables with appropriate wraparound.
The kernel region itself will be properly mapped; other mappings may
be spurious.
early_make_pgtable is using kernel high mapping address to access pages
to set page table.
-v4: Add phys_base offset to make kexec happy, and add
init_mapping_kernel() - Yinghai
-v5: fix compiling with xen, and add back ident level3 and level2 for xen
also move back init_level4_pgt from BSS to DATA again.
because we have to clear it anyway. - Yinghai
-v6: switch to init_level4_pgt in init_mem_mapping. - Yinghai
-v7: remove not needed clear_page for init_level4_page
it is with fill 512,8,0 already in head_64.S - Yinghai
-v8: we need to keep that handler alive until init_mem_mapping and don't
let early_trap_init to trash that early #PF handler.
So split early_trap_pf_init out and move it down. - Yinghai
-v9: switchover only cover kernel space instead of 1G so could avoid
touch possible mem holes. - Yinghai
-v11: change far jmp back to far return to initial_code, that is needed
to fix failure that is reported by Konrad on AMD systems. - Yinghai
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Link: http://lkml.kernel.org/r/1359058816-7615-12-git-send-email-yinghai@kernel.org
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
2013-01-24 20:19:52 +00:00
|
|
|
movq %rcx, %cr4
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2017-06-06 11:31:28 +00:00
|
|
|
/* Setup early boot stage 4-/5-level pagetables. */
|
2007-05-02 17:27:07 +00:00
|
|
|
addq phys_base(%rip), %rax
|
2020-10-28 16:46:58 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* For SEV guests: Verify that the C-bit is correct. A malicious
|
|
|
|
|
* hypervisor could lie about the C-bit position to perform a ROP
|
|
|
|
|
* attack on the guest by writing to the unencrypted stack and wait for
|
|
|
|
|
* the next RET instruction.
|
|
|
|
|
* %rsi carries pointer to realmode data and is callee-clobbered. Save
|
|
|
|
|
* and restore it.
|
|
|
|
|
*/
|
|
|
|
|
pushq %rsi
|
|
|
|
|
movq %rax, %rdi
|
|
|
|
|
call sev_verify_cbit
|
|
|
|
|
popq %rsi
|
|
|
|
|
|
|
|
|
|
/* Switch to new page-table */
|
2005-04-16 22:20:36 +00:00
|
|
|
movq %rax, %cr3
|
|
|
|
|
|
2007-05-02 17:27:07 +00:00
|
|
|
/* Ensure I am executing from virtual addresses */
|
|
|
|
|
movq $1f, %rax
|
2018-01-16 09:38:09 +00:00
|
|
|
ANNOTATE_RETPOLINE_SAFE
|
2007-05-02 17:27:07 +00:00
|
|
|
jmp *%rax
|
|
|
|
|
1:
|
2017-09-19 02:43:37 +00:00
|
|
|
UNWIND_HINT_EMPTY
|
2007-05-02 17:27:07 +00:00
|
|
|
|
2020-09-07 13:15:31 +00:00
|
|
|
/*
|
|
|
|
|
* We must switch to a new descriptor in kernel space for the GDT
|
|
|
|
|
* because soon the kernel won't have access anymore to the userspace
|
|
|
|
|
* addresses where we're currently running on. We have to do that here
|
|
|
|
|
* because in 32bit we couldn't load a 64bit linear address.
|
|
|
|
|
*/
|
|
|
|
|
lgdt early_gdt_descr(%rip)
|
|
|
|
|
|
2020-09-07 13:15:32 +00:00
|
|
|
/* set up data segments */
|
|
|
|
|
xorl %eax,%eax
|
|
|
|
|
movl %eax,%ds
|
|
|
|
|
movl %eax,%ss
|
|
|
|
|
movl %eax,%es
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* We don't really need to load %fs or %gs, but load them anyway
|
|
|
|
|
* to kill any stale realmode selectors. This allows execution
|
|
|
|
|
* under VT hardware.
|
|
|
|
|
*/
|
|
|
|
|
movl %eax,%fs
|
|
|
|
|
movl %eax,%gs
|
|
|
|
|
|
|
|
|
|
/* Set up %gs.
|
|
|
|
|
*
|
|
|
|
|
* The base of %gs always points to fixed_percpu_data. If the
|
|
|
|
|
* stack protector canary is enabled, it is located at %gs:40.
|
|
|
|
|
* Note that, on SMP, the boot cpu uses init data section until
|
|
|
|
|
* the per cpu areas are set up.
|
|
|
|
|
*/
|
|
|
|
|
movl $MSR_GS_BASE,%ecx
|
|
|
|
|
movl initial_gs(%rip),%eax
|
|
|
|
|
movl initial_gs+4(%rip),%edx
|
|
|
|
|
wrmsr
|
|
|
|
|
|
2020-09-07 13:15:33 +00:00
|
|
|
/*
|
|
|
|
|
* Setup a boot time stack - Any secondary CPU will have lost its stack
|
|
|
|
|
* by now because the cr3-switch above unmaps the real-mode stack
|
|
|
|
|
*/
|
|
|
|
|
movq initial_stack(%rip), %rsp
|
|
|
|
|
|
2020-09-07 13:15:34 +00:00
|
|
|
/* Setup and Load IDT */
|
|
|
|
|
pushq %rsi
|
|
|
|
|
call early_setup_idt
|
|
|
|
|
popq %rsi
|
|
|
|
|
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Check if nx is implemented */
|
|
|
|
|
movl $0x80000001, %eax
|
|
|
|
|
cpuid
|
|
|
|
|
movl %edx,%edi
|
|
|
|
|
|
|
|
|
|
/* Setup EFER (Extended Feature Enable Register) */
|
|
|
|
|
movl $MSR_EFER, %ecx
|
|
|
|
|
rdmsr
|
2007-05-02 17:27:07 +00:00
|
|
|
btsl $_EFER_SCE, %eax /* Enable System Call */
|
|
|
|
|
btl $20,%edi /* No Execute supported? */
|
2005-04-16 22:20:36 +00:00
|
|
|
jnc 1f
|
|
|
|
|
btsl $_EFER_NX, %eax
|
2013-05-02 17:33:46 +00:00
|
|
|
btsq $_PAGE_BIT_NX,early_pmd_flags(%rip)
|
2007-05-02 17:27:07 +00:00
|
|
|
1: wrmsr /* Make changes effective */
|
2005-04-16 22:20:36 +00:00
|
|
|
|
|
|
|
|
/* Setup cr0 */
|
2008-05-12 13:43:38 +00:00
|
|
|
movl $CR0_STATE, %eax
|
2005-04-16 22:20:36 +00:00
|
|
|
/* Make changes effective */
|
|
|
|
|
movq %rax, %cr0
|
|
|
|
|
|
|
|
|
|
/* zero EFLAGS after setting rsp */
|
|
|
|
|
pushq $0
|
|
|
|
|
popfq
|
|
|
|
|
|
x86, 64bit: Use a #PF handler to materialize early mappings on demand
Linear mode (CR0.PG = 0) is mutually exclusive with 64-bit mode; all
64-bit code has to use page tables. This makes it awkward before we
have first set up properly all-covering page tables to access objects
that are outside the static kernel range.
So far we have dealt with that simply by mapping a fixed amount of
low memory, but that fails in at least two upcoming use cases:
1. We will support load and run kernel, struct boot_params, ramdisk,
command line, etc. above the 4 GiB mark.
2. need to access ramdisk early to get microcode to update that as
early possible.
We could use early_iomap to access them too, but it will make code to
messy and hard to be unified with 32 bit.
Hence, set up a #PF table and use a fixed number of buffers to set up
page tables on demand. If the buffers fill up then we simply flush
them and start over. These buffers are all in __initdata, so it does
not increase RAM usage at runtime.
Thus, with the help of the #PF handler, we can set the final kernel
mapping from blank, and switch to init_level4_pgt later.
During the switchover in head_64.S, before #PF handler is available,
we use three pages to handle kernel crossing 1G, 512G boundaries with
sharing page by playing games with page aliasing: the same page is
mapped twice in the higher-level tables with appropriate wraparound.
The kernel region itself will be properly mapped; other mappings may
be spurious.
early_make_pgtable is using kernel high mapping address to access pages
to set page table.
-v4: Add phys_base offset to make kexec happy, and add
init_mapping_kernel() - Yinghai
-v5: fix compiling with xen, and add back ident level3 and level2 for xen
also move back init_level4_pgt from BSS to DATA again.
because we have to clear it anyway. - Yinghai
-v6: switch to init_level4_pgt in init_mem_mapping. - Yinghai
-v7: remove not needed clear_page for init_level4_page
it is with fill 512,8,0 already in head_64.S - Yinghai
-v8: we need to keep that handler alive until init_mem_mapping and don't
let early_trap_init to trash that early #PF handler.
So split early_trap_pf_init out and move it down. - Yinghai
-v9: switchover only cover kernel space instead of 1G so could avoid
touch possible mem holes. - Yinghai
-v11: change far jmp back to far return to initial_code, that is needed
to fix failure that is reported by Konrad on AMD systems. - Yinghai
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Link: http://lkml.kernel.org/r/1359058816-7615-12-git-send-email-yinghai@kernel.org
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
2013-01-24 20:19:52 +00:00
|
|
|
/* rsi is pointer to real mode structure with interesting info.
|
2005-04-16 22:20:36 +00:00
|
|
|
pass it to C */
|
x86, 64bit: Use a #PF handler to materialize early mappings on demand
Linear mode (CR0.PG = 0) is mutually exclusive with 64-bit mode; all
64-bit code has to use page tables. This makes it awkward before we
have first set up properly all-covering page tables to access objects
that are outside the static kernel range.
So far we have dealt with that simply by mapping a fixed amount of
low memory, but that fails in at least two upcoming use cases:
1. We will support load and run kernel, struct boot_params, ramdisk,
command line, etc. above the 4 GiB mark.
2. need to access ramdisk early to get microcode to update that as
early possible.
We could use early_iomap to access them too, but it will make code to
messy and hard to be unified with 32 bit.
Hence, set up a #PF table and use a fixed number of buffers to set up
page tables on demand. If the buffers fill up then we simply flush
them and start over. These buffers are all in __initdata, so it does
not increase RAM usage at runtime.
Thus, with the help of the #PF handler, we can set the final kernel
mapping from blank, and switch to init_level4_pgt later.
During the switchover in head_64.S, before #PF handler is available,
we use three pages to handle kernel crossing 1G, 512G boundaries with
sharing page by playing games with page aliasing: the same page is
mapped twice in the higher-level tables with appropriate wraparound.
The kernel region itself will be properly mapped; other mappings may
be spurious.
early_make_pgtable is using kernel high mapping address to access pages
to set page table.
-v4: Add phys_base offset to make kexec happy, and add
init_mapping_kernel() - Yinghai
-v5: fix compiling with xen, and add back ident level3 and level2 for xen
also move back init_level4_pgt from BSS to DATA again.
because we have to clear it anyway. - Yinghai
-v6: switch to init_level4_pgt in init_mem_mapping. - Yinghai
-v7: remove not needed clear_page for init_level4_page
it is with fill 512,8,0 already in head_64.S - Yinghai
-v8: we need to keep that handler alive until init_mem_mapping and don't
let early_trap_init to trash that early #PF handler.
So split early_trap_pf_init out and move it down. - Yinghai
-v9: switchover only cover kernel space instead of 1G so could avoid
touch possible mem holes. - Yinghai
-v11: change far jmp back to far return to initial_code, that is needed
to fix failure that is reported by Konrad on AMD systems. - Yinghai
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Link: http://lkml.kernel.org/r/1359058816-7615-12-git-send-email-yinghai@kernel.org
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
2013-01-24 20:19:52 +00:00
|
|
|
movq %rsi, %rdi
|
2016-09-21 21:04:04 +00:00
|
|
|
|
2017-03-04 09:56:10 +00:00
|
|
|
.Ljump_to_C_code:
|
2016-09-21 21:04:04 +00:00
|
|
|
/*
|
|
|
|
|
* Jump to run C code and to be on a real kernel address.
|
2005-04-16 22:20:36 +00:00
|
|
|
* Since we are running on identity-mapped space we have to jump
|
2006-09-26 08:52:38 +00:00
|
|
|
* to the full 64bit address, this is only possible as indirect
|
|
|
|
|
* jump. In addition we need to ensure %cs is set so we make this
|
|
|
|
|
* a far return.
|
x86, 64bit: Use a #PF handler to materialize early mappings on demand
Linear mode (CR0.PG = 0) is mutually exclusive with 64-bit mode; all
64-bit code has to use page tables. This makes it awkward before we
have first set up properly all-covering page tables to access objects
that are outside the static kernel range.
So far we have dealt with that simply by mapping a fixed amount of
low memory, but that fails in at least two upcoming use cases:
1. We will support load and run kernel, struct boot_params, ramdisk,
command line, etc. above the 4 GiB mark.
2. need to access ramdisk early to get microcode to update that as
early possible.
We could use early_iomap to access them too, but it will make code to
messy and hard to be unified with 32 bit.
Hence, set up a #PF table and use a fixed number of buffers to set up
page tables on demand. If the buffers fill up then we simply flush
them and start over. These buffers are all in __initdata, so it does
not increase RAM usage at runtime.
Thus, with the help of the #PF handler, we can set the final kernel
mapping from blank, and switch to init_level4_pgt later.
During the switchover in head_64.S, before #PF handler is available,
we use three pages to handle kernel crossing 1G, 512G boundaries with
sharing page by playing games with page aliasing: the same page is
mapped twice in the higher-level tables with appropriate wraparound.
The kernel region itself will be properly mapped; other mappings may
be spurious.
early_make_pgtable is using kernel high mapping address to access pages
to set page table.
-v4: Add phys_base offset to make kexec happy, and add
init_mapping_kernel() - Yinghai
-v5: fix compiling with xen, and add back ident level3 and level2 for xen
also move back init_level4_pgt from BSS to DATA again.
because we have to clear it anyway. - Yinghai
-v6: switch to init_level4_pgt in init_mem_mapping. - Yinghai
-v7: remove not needed clear_page for init_level4_page
it is with fill 512,8,0 already in head_64.S - Yinghai
-v8: we need to keep that handler alive until init_mem_mapping and don't
let early_trap_init to trash that early #PF handler.
So split early_trap_pf_init out and move it down. - Yinghai
-v9: switchover only cover kernel space instead of 1G so could avoid
touch possible mem holes. - Yinghai
-v11: change far jmp back to far return to initial_code, that is needed
to fix failure that is reported by Konrad on AMD systems. - Yinghai
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Link: http://lkml.kernel.org/r/1359058816-7615-12-git-send-email-yinghai@kernel.org
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
2013-01-24 20:19:52 +00:00
|
|
|
*
|
|
|
|
|
* Note: do not change to far jump indirect with 64bit offset.
|
|
|
|
|
*
|
|
|
|
|
* AMD does not support far jump indirect with 64bit offset.
|
|
|
|
|
* AMD64 Architecture Programmer's Manual, Volume 3: states only
|
|
|
|
|
* JMP FAR mem16:16 FF /5 Far jump indirect,
|
|
|
|
|
* with the target specified by a far pointer in memory.
|
|
|
|
|
* JMP FAR mem16:32 FF /5 Far jump indirect,
|
|
|
|
|
* with the target specified by a far pointer in memory.
|
|
|
|
|
*
|
|
|
|
|
* Intel64 does support 64bit offset.
|
|
|
|
|
* Software Developer Manual Vol 2: states:
|
|
|
|
|
* FF /5 JMP m16:16 Jump far, absolute indirect,
|
|
|
|
|
* address given in m16:16
|
|
|
|
|
* FF /5 JMP m16:32 Jump far, absolute indirect,
|
|
|
|
|
* address given in m16:32.
|
|
|
|
|
* REX.W + FF /5 JMP m16:64 Jump far, absolute indirect,
|
|
|
|
|
* address given in m16:64.
|
2005-04-16 22:20:36 +00:00
|
|
|
*/
|
2016-12-14 03:25:36 +00:00
|
|
|
pushq $.Lafter_lret # put return address on stack for unwinder
|
2018-07-02 10:31:54 +00:00
|
|
|
xorl %ebp, %ebp # clear frame pointer
|
2016-09-21 21:04:05 +00:00
|
|
|
movq initial_code(%rip), %rax
|
2006-09-26 08:52:38 +00:00
|
|
|
pushq $__KERNEL_CS # set correct cs
|
|
|
|
|
pushq %rax # target address in negative space
|
|
|
|
|
lretq
|
2016-12-14 03:25:36 +00:00
|
|
|
.Lafter_lret:
|
2019-10-11 11:51:03 +00:00
|
|
|
SYM_CODE_END(secondary_startup_64)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2015-11-05 15:57:56 +00:00
|
|
|
#include "verify_cpu.S"
|
2020-10-28 16:46:58 +00:00
|
|
|
#include "sev_verify_cbit.S"
|
2015-11-05 15:57:56 +00:00
|
|
|
|
2012-11-13 19:32:44 +00:00
|
|
|
#ifdef CONFIG_HOTPLUG_CPU
|
|
|
|
|
/*
|
|
|
|
|
* Boot CPU0 entry point. It's called from play_dead(). Everything has been set
|
|
|
|
|
* up already except stack. We just set up stack here. Then call
|
2017-03-04 09:56:10 +00:00
|
|
|
* start_secondary() via .Ljump_to_C_code.
|
2012-11-13 19:32:44 +00:00
|
|
|
*/
|
2019-10-11 11:51:03 +00:00
|
|
|
SYM_CODE_START(start_cpu0)
|
2017-09-19 02:43:37 +00:00
|
|
|
UNWIND_HINT_EMPTY
|
2019-07-18 01:36:41 +00:00
|
|
|
movq initial_stack(%rip), %rsp
|
2017-03-04 09:56:10 +00:00
|
|
|
jmp .Ljump_to_C_code
|
2019-10-11 11:51:03 +00:00
|
|
|
SYM_CODE_END(start_cpu0)
|
2020-09-08 12:38:16 +00:00
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#ifdef CONFIG_AMD_MEM_ENCRYPT
|
|
|
|
|
/*
|
|
|
|
|
* VC Exception handler used during early boot when running on kernel
|
|
|
|
|
* addresses, but before the switch to the idt_table can be made.
|
|
|
|
|
* The early_idt_handler_array can't be used here because it calls into a lot
|
|
|
|
|
* of __init code and this handler is also used during CPU offlining/onlining.
|
|
|
|
|
* Therefore this handler ends up in the .text section so that it stays around
|
|
|
|
|
* when .init.text is freed.
|
|
|
|
|
*/
|
|
|
|
|
SYM_CODE_START_NOALIGN(vc_boot_ghcb)
|
|
|
|
|
UNWIND_HINT_IRET_REGS offset=8
|
|
|
|
|
|
2022-06-14 21:16:03 +00:00
|
|
|
ANNOTATE_UNRET_END
|
|
|
|
|
|
2020-09-08 12:38:16 +00:00
|
|
|
/* Build pt_regs */
|
|
|
|
|
PUSH_AND_CLEAR_REGS
|
|
|
|
|
|
|
|
|
|
/* Call C handler */
|
|
|
|
|
movq %rsp, %rdi
|
|
|
|
|
movq ORIG_RAX(%rsp), %rsi
|
|
|
|
|
movq initial_vc_handler(%rip), %rax
|
|
|
|
|
ANNOTATE_RETPOLINE_SAFE
|
|
|
|
|
call *%rax
|
|
|
|
|
|
|
|
|
|
/* Unwind pt_regs */
|
|
|
|
|
POP_REGS
|
|
|
|
|
|
|
|
|
|
/* Remove Error Code */
|
|
|
|
|
addq $8, %rsp
|
|
|
|
|
|
|
|
|
|
/* Pure iret required here - don't use INTERRUPT_RETURN */
|
|
|
|
|
iretq
|
|
|
|
|
SYM_CODE_END(vc_boot_ghcb)
|
2012-11-13 19:32:44 +00:00
|
|
|
#endif
|
|
|
|
|
|
2016-08-18 15:59:03 +00:00
|
|
|
/* Both SMP bootup and ACPI suspend change these variables */
|
2008-02-17 12:22:59 +00:00
|
|
|
__REFDATA
|
x86, 64bit: Use a #PF handler to materialize early mappings on demand
Linear mode (CR0.PG = 0) is mutually exclusive with 64-bit mode; all
64-bit code has to use page tables. This makes it awkward before we
have first set up properly all-covering page tables to access objects
that are outside the static kernel range.
So far we have dealt with that simply by mapping a fixed amount of
low memory, but that fails in at least two upcoming use cases:
1. We will support load and run kernel, struct boot_params, ramdisk,
command line, etc. above the 4 GiB mark.
2. need to access ramdisk early to get microcode to update that as
early possible.
We could use early_iomap to access them too, but it will make code to
messy and hard to be unified with 32 bit.
Hence, set up a #PF table and use a fixed number of buffers to set up
page tables on demand. If the buffers fill up then we simply flush
them and start over. These buffers are all in __initdata, so it does
not increase RAM usage at runtime.
Thus, with the help of the #PF handler, we can set the final kernel
mapping from blank, and switch to init_level4_pgt later.
During the switchover in head_64.S, before #PF handler is available,
we use three pages to handle kernel crossing 1G, 512G boundaries with
sharing page by playing games with page aliasing: the same page is
mapped twice in the higher-level tables with appropriate wraparound.
The kernel region itself will be properly mapped; other mappings may
be spurious.
early_make_pgtable is using kernel high mapping address to access pages
to set page table.
-v4: Add phys_base offset to make kexec happy, and add
init_mapping_kernel() - Yinghai
-v5: fix compiling with xen, and add back ident level3 and level2 for xen
also move back init_level4_pgt from BSS to DATA again.
because we have to clear it anyway. - Yinghai
-v6: switch to init_level4_pgt in init_mem_mapping. - Yinghai
-v7: remove not needed clear_page for init_level4_page
it is with fill 512,8,0 already in head_64.S - Yinghai
-v8: we need to keep that handler alive until init_mem_mapping and don't
let early_trap_init to trash that early #PF handler.
So split early_trap_pf_init out and move it down. - Yinghai
-v9: switchover only cover kernel space instead of 1G so could avoid
touch possible mem holes. - Yinghai
-v11: change far jmp back to far return to initial_code, that is needed
to fix failure that is reported by Konrad on AMD systems. - Yinghai
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Link: http://lkml.kernel.org/r/1359058816-7615-12-git-send-email-yinghai@kernel.org
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
2013-01-24 20:19:52 +00:00
|
|
|
.balign 8
|
2019-10-11 11:50:51 +00:00
|
|
|
SYM_DATA(initial_code, .quad x86_64_start_kernel)
|
|
|
|
|
SYM_DATA(initial_gs, .quad INIT_PER_CPU_VAR(fixed_percpu_data))
|
2020-09-08 12:38:16 +00:00
|
|
|
#ifdef CONFIG_AMD_MEM_ENCRYPT
|
|
|
|
|
SYM_DATA(initial_vc_handler, .quad handle_vc_boot_ghcb)
|
|
|
|
|
#endif
|
2019-10-11 11:50:51 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* The SIZEOF_PTREGS gap is a convention which helps the in-kernel unwinder
|
|
|
|
|
* reliably detect the end of the stack.
|
|
|
|
|
*/
|
|
|
|
|
SYM_DATA(initial_stack, .quad init_thread_union + THREAD_SIZE - SIZEOF_PTREGS)
|
2009-10-14 21:46:55 +00:00
|
|
|
__FINITDATA
|
2005-04-16 22:20:36 +00:00
|
|
|
|
x86, 64bit: Use a #PF handler to materialize early mappings on demand
Linear mode (CR0.PG = 0) is mutually exclusive with 64-bit mode; all
64-bit code has to use page tables. This makes it awkward before we
have first set up properly all-covering page tables to access objects
that are outside the static kernel range.
So far we have dealt with that simply by mapping a fixed amount of
low memory, but that fails in at least two upcoming use cases:
1. We will support load and run kernel, struct boot_params, ramdisk,
command line, etc. above the 4 GiB mark.
2. need to access ramdisk early to get microcode to update that as
early possible.
We could use early_iomap to access them too, but it will make code to
messy and hard to be unified with 32 bit.
Hence, set up a #PF table and use a fixed number of buffers to set up
page tables on demand. If the buffers fill up then we simply flush
them and start over. These buffers are all in __initdata, so it does
not increase RAM usage at runtime.
Thus, with the help of the #PF handler, we can set the final kernel
mapping from blank, and switch to init_level4_pgt later.
During the switchover in head_64.S, before #PF handler is available,
we use three pages to handle kernel crossing 1G, 512G boundaries with
sharing page by playing games with page aliasing: the same page is
mapped twice in the higher-level tables with appropriate wraparound.
The kernel region itself will be properly mapped; other mappings may
be spurious.
early_make_pgtable is using kernel high mapping address to access pages
to set page table.
-v4: Add phys_base offset to make kexec happy, and add
init_mapping_kernel() - Yinghai
-v5: fix compiling with xen, and add back ident level3 and level2 for xen
also move back init_level4_pgt from BSS to DATA again.
because we have to clear it anyway. - Yinghai
-v6: switch to init_level4_pgt in init_mem_mapping. - Yinghai
-v7: remove not needed clear_page for init_level4_page
it is with fill 512,8,0 already in head_64.S - Yinghai
-v8: we need to keep that handler alive until init_mem_mapping and don't
let early_trap_init to trash that early #PF handler.
So split early_trap_pf_init out and move it down. - Yinghai
-v9: switchover only cover kernel space instead of 1G so could avoid
touch possible mem holes. - Yinghai
-v11: change far jmp back to far return to initial_code, that is needed
to fix failure that is reported by Konrad on AMD systems. - Yinghai
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Link: http://lkml.kernel.org/r/1359058816-7615-12-git-send-email-yinghai@kernel.org
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
2013-01-24 20:19:52 +00:00
|
|
|
__INIT
|
2019-10-11 11:51:03 +00:00
|
|
|
SYM_CODE_START(early_idt_handler_array)
|
2008-03-11 01:23:22 +00:00
|
|
|
i = 0
|
|
|
|
|
.rept NUM_EXCEPTION_VECTORS
|
2017-10-20 16:21:35 +00:00
|
|
|
.if ((EXCEPTION_ERRCODE_MASK >> i) & 1) == 0
|
2017-09-19 02:43:37 +00:00
|
|
|
UNWIND_HINT_IRET_REGS
|
|
|
|
|
pushq $0 # Dummy error code, to make stack frame uniform
|
|
|
|
|
.else
|
|
|
|
|
UNWIND_HINT_IRET_REGS offset=8
|
2012-04-19 00:16:49 +00:00
|
|
|
.endif
|
|
|
|
|
pushq $i # 72(%rsp) Vector number
|
2015-05-22 23:15:47 +00:00
|
|
|
jmp early_idt_handler_common
|
2017-09-19 02:43:37 +00:00
|
|
|
UNWIND_HINT_IRET_REGS
|
2008-03-11 01:23:22 +00:00
|
|
|
i = i + 1
|
2015-05-22 23:15:47 +00:00
|
|
|
.fill early_idt_handler_array + i*EARLY_IDT_HANDLER_SIZE - ., 1, 0xcc
|
2008-03-11 01:23:22 +00:00
|
|
|
.endr
|
2017-09-19 02:43:37 +00:00
|
|
|
UNWIND_HINT_IRET_REGS offset=16
|
2019-10-11 11:51:03 +00:00
|
|
|
SYM_CODE_END(early_idt_handler_array)
|
2008-01-30 12:33:06 +00:00
|
|
|
|
2019-10-11 11:50:45 +00:00
|
|
|
SYM_CODE_START_LOCAL(early_idt_handler_common)
|
2022-06-14 21:16:03 +00:00
|
|
|
ANNOTATE_UNRET_END
|
2015-05-22 23:15:47 +00:00
|
|
|
/*
|
|
|
|
|
* The stack is the hardware frame, an error code or zero, and the
|
|
|
|
|
* vector number.
|
|
|
|
|
*/
|
2012-04-19 00:16:49 +00:00
|
|
|
cld
|
|
|
|
|
|
2005-04-16 22:25:00 +00:00
|
|
|
incl early_recursion_flag(%rip)
|
2012-04-19 00:16:49 +00:00
|
|
|
|
2016-04-02 14:01:32 +00:00
|
|
|
/* The vector number is currently in the pt_regs->di slot. */
|
|
|
|
|
pushq %rsi /* pt_regs->si */
|
|
|
|
|
movq 8(%rsp), %rsi /* RSI = vector number */
|
|
|
|
|
movq %rdi, 8(%rsp) /* pt_regs->di = RDI */
|
|
|
|
|
pushq %rdx /* pt_regs->dx */
|
|
|
|
|
pushq %rcx /* pt_regs->cx */
|
|
|
|
|
pushq %rax /* pt_regs->ax */
|
|
|
|
|
pushq %r8 /* pt_regs->r8 */
|
|
|
|
|
pushq %r9 /* pt_regs->r9 */
|
|
|
|
|
pushq %r10 /* pt_regs->r10 */
|
|
|
|
|
pushq %r11 /* pt_regs->r11 */
|
|
|
|
|
pushq %rbx /* pt_regs->bx */
|
|
|
|
|
pushq %rbp /* pt_regs->bp */
|
|
|
|
|
pushq %r12 /* pt_regs->r12 */
|
|
|
|
|
pushq %r13 /* pt_regs->r13 */
|
|
|
|
|
pushq %r14 /* pt_regs->r14 */
|
|
|
|
|
pushq %r15 /* pt_regs->r15 */
|
2017-09-19 02:43:37 +00:00
|
|
|
UNWIND_HINT_REGS
|
2016-04-02 14:01:32 +00:00
|
|
|
|
|
|
|
|
movq %rsp,%rdi /* RDI = pt_regs; RSI is already trapnr */
|
2020-09-07 13:15:36 +00:00
|
|
|
call do_early_exception
|
2008-01-30 12:33:06 +00:00
|
|
|
|
2012-04-19 00:16:49 +00:00
|
|
|
decl early_recursion_flag(%rip)
|
2017-11-02 07:58:59 +00:00
|
|
|
jmp restore_regs_and_return_to_kernel
|
2019-10-11 11:50:45 +00:00
|
|
|
SYM_CODE_END(early_idt_handler_common)
|
2012-04-19 00:16:49 +00:00
|
|
|
|
2020-09-08 12:35:17 +00:00
|
|
|
#ifdef CONFIG_AMD_MEM_ENCRYPT
|
|
|
|
|
/*
|
|
|
|
|
* VC Exception handler used during very early boot. The
|
|
|
|
|
* early_idt_handler_array can't be used because it returns via the
|
|
|
|
|
* paravirtualized INTERRUPT_RETURN and pv-ops don't work that early.
|
|
|
|
|
*
|
|
|
|
|
* This handler will end up in the .init.text section and not be
|
|
|
|
|
* available to boot secondary CPUs.
|
|
|
|
|
*/
|
|
|
|
|
SYM_CODE_START_NOALIGN(vc_no_ghcb)
|
|
|
|
|
UNWIND_HINT_IRET_REGS offset=8
|
|
|
|
|
|
2022-06-14 21:16:03 +00:00
|
|
|
ANNOTATE_UNRET_END
|
|
|
|
|
|
2020-09-08 12:35:17 +00:00
|
|
|
/* Build pt_regs */
|
|
|
|
|
PUSH_AND_CLEAR_REGS
|
|
|
|
|
|
|
|
|
|
/* Call C handler */
|
|
|
|
|
movq %rsp, %rdi
|
|
|
|
|
movq ORIG_RAX(%rsp), %rsi
|
|
|
|
|
call do_vc_no_ghcb
|
|
|
|
|
|
|
|
|
|
/* Unwind pt_regs */
|
|
|
|
|
POP_REGS
|
|
|
|
|
|
|
|
|
|
/* Remove Error Code */
|
|
|
|
|
addq $8, %rsp
|
|
|
|
|
|
|
|
|
|
/* Pure iret required here - don't use INTERRUPT_RETURN */
|
|
|
|
|
iretq
|
|
|
|
|
SYM_CODE_END(vc_no_ghcb)
|
|
|
|
|
#endif
|
2019-10-11 11:50:51 +00:00
|
|
|
|
|
|
|
|
#define SYM_DATA_START_PAGE_ALIGNED(name) \
|
|
|
|
|
SYM_START(name, SYM_L_GLOBAL, .balign PAGE_SIZE)
|
2006-01-17 06:03:32 +00:00
|
|
|
|
2017-12-04 14:07:39 +00:00
|
|
|
#ifdef CONFIG_PAGE_TABLE_ISOLATION
|
|
|
|
|
/*
|
|
|
|
|
* Each PGD needs to be 8k long and 8k aligned. We do not
|
|
|
|
|
* ever go out to userspace with these, so we do not
|
|
|
|
|
* strictly *need* the second page, but this allows us to
|
|
|
|
|
* have a single set_pgd() implementation that does not
|
|
|
|
|
* need to worry about whether it has 4k or 8k to work
|
|
|
|
|
* with.
|
|
|
|
|
*
|
|
|
|
|
* This ensures PGDs are 8k long:
|
|
|
|
|
*/
|
|
|
|
|
#define PTI_USER_PGD_FILL 512
|
|
|
|
|
/* This ensures they are 8k-aligned: */
|
2019-10-11 11:50:51 +00:00
|
|
|
#define SYM_DATA_START_PTI_ALIGNED(name) \
|
|
|
|
|
SYM_START(name, SYM_L_GLOBAL, .balign 2 * PAGE_SIZE)
|
2017-12-04 14:07:39 +00:00
|
|
|
#else
|
2019-10-11 11:50:51 +00:00
|
|
|
#define SYM_DATA_START_PTI_ALIGNED(name) \
|
|
|
|
|
SYM_DATA_START_PAGE_ALIGNED(name)
|
2017-12-04 14:07:39 +00:00
|
|
|
#define PTI_USER_PGD_FILL 0
|
|
|
|
|
#endif
|
|
|
|
|
|
2007-05-02 17:27:06 +00:00
|
|
|
/* Automate the creation of 1 to 1 mapping pmd entries */
|
2008-05-13 16:55:40 +00:00
|
|
|
#define PMDS(START, PERM, COUNT) \
|
|
|
|
|
i = 0 ; \
|
|
|
|
|
.rept (COUNT) ; \
|
|
|
|
|
.quad (START) + (i << PMD_SHIFT) + (PERM) ; \
|
|
|
|
|
i = i + 1 ; \
|
2007-05-02 17:27:06 +00:00
|
|
|
.endr
|
|
|
|
|
|
x86, 64bit: Use a #PF handler to materialize early mappings on demand
Linear mode (CR0.PG = 0) is mutually exclusive with 64-bit mode; all
64-bit code has to use page tables. This makes it awkward before we
have first set up properly all-covering page tables to access objects
that are outside the static kernel range.
So far we have dealt with that simply by mapping a fixed amount of
low memory, but that fails in at least two upcoming use cases:
1. We will support load and run kernel, struct boot_params, ramdisk,
command line, etc. above the 4 GiB mark.
2. need to access ramdisk early to get microcode to update that as
early possible.
We could use early_iomap to access them too, but it will make code to
messy and hard to be unified with 32 bit.
Hence, set up a #PF table and use a fixed number of buffers to set up
page tables on demand. If the buffers fill up then we simply flush
them and start over. These buffers are all in __initdata, so it does
not increase RAM usage at runtime.
Thus, with the help of the #PF handler, we can set the final kernel
mapping from blank, and switch to init_level4_pgt later.
During the switchover in head_64.S, before #PF handler is available,
we use three pages to handle kernel crossing 1G, 512G boundaries with
sharing page by playing games with page aliasing: the same page is
mapped twice in the higher-level tables with appropriate wraparound.
The kernel region itself will be properly mapped; other mappings may
be spurious.
early_make_pgtable is using kernel high mapping address to access pages
to set page table.
-v4: Add phys_base offset to make kexec happy, and add
init_mapping_kernel() - Yinghai
-v5: fix compiling with xen, and add back ident level3 and level2 for xen
also move back init_level4_pgt from BSS to DATA again.
because we have to clear it anyway. - Yinghai
-v6: switch to init_level4_pgt in init_mem_mapping. - Yinghai
-v7: remove not needed clear_page for init_level4_page
it is with fill 512,8,0 already in head_64.S - Yinghai
-v8: we need to keep that handler alive until init_mem_mapping and don't
let early_trap_init to trash that early #PF handler.
So split early_trap_pf_init out and move it down. - Yinghai
-v9: switchover only cover kernel space instead of 1G so could avoid
touch possible mem holes. - Yinghai
-v11: change far jmp back to far return to initial_code, that is needed
to fix failure that is reported by Konrad on AMD systems. - Yinghai
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Link: http://lkml.kernel.org/r/1359058816-7615-12-git-send-email-yinghai@kernel.org
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
2013-01-24 20:19:52 +00:00
|
|
|
__INITDATA
|
2019-10-03 09:52:37 +00:00
|
|
|
.balign 4
|
|
|
|
|
|
2019-10-11 11:50:51 +00:00
|
|
|
SYM_DATA_START_PTI_ALIGNED(early_top_pgt)
|
2018-02-14 18:25:39 +00:00
|
|
|
.fill 512,8,0
|
2017-12-04 14:07:39 +00:00
|
|
|
.fill PTI_USER_PGD_FILL,8,0
|
2019-10-11 11:50:51 +00:00
|
|
|
SYM_DATA_END(early_top_pgt)
|
x86, 64bit: Use a #PF handler to materialize early mappings on demand
Linear mode (CR0.PG = 0) is mutually exclusive with 64-bit mode; all
64-bit code has to use page tables. This makes it awkward before we
have first set up properly all-covering page tables to access objects
that are outside the static kernel range.
So far we have dealt with that simply by mapping a fixed amount of
low memory, but that fails in at least two upcoming use cases:
1. We will support load and run kernel, struct boot_params, ramdisk,
command line, etc. above the 4 GiB mark.
2. need to access ramdisk early to get microcode to update that as
early possible.
We could use early_iomap to access them too, but it will make code to
messy and hard to be unified with 32 bit.
Hence, set up a #PF table and use a fixed number of buffers to set up
page tables on demand. If the buffers fill up then we simply flush
them and start over. These buffers are all in __initdata, so it does
not increase RAM usage at runtime.
Thus, with the help of the #PF handler, we can set the final kernel
mapping from blank, and switch to init_level4_pgt later.
During the switchover in head_64.S, before #PF handler is available,
we use three pages to handle kernel crossing 1G, 512G boundaries with
sharing page by playing games with page aliasing: the same page is
mapped twice in the higher-level tables with appropriate wraparound.
The kernel region itself will be properly mapped; other mappings may
be spurious.
early_make_pgtable is using kernel high mapping address to access pages
to set page table.
-v4: Add phys_base offset to make kexec happy, and add
init_mapping_kernel() - Yinghai
-v5: fix compiling with xen, and add back ident level3 and level2 for xen
also move back init_level4_pgt from BSS to DATA again.
because we have to clear it anyway. - Yinghai
-v6: switch to init_level4_pgt in init_mem_mapping. - Yinghai
-v7: remove not needed clear_page for init_level4_page
it is with fill 512,8,0 already in head_64.S - Yinghai
-v8: we need to keep that handler alive until init_mem_mapping and don't
let early_trap_init to trash that early #PF handler.
So split early_trap_pf_init out and move it down. - Yinghai
-v9: switchover only cover kernel space instead of 1G so could avoid
touch possible mem holes. - Yinghai
-v11: change far jmp back to far return to initial_code, that is needed
to fix failure that is reported by Konrad on AMD systems. - Yinghai
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Link: http://lkml.kernel.org/r/1359058816-7615-12-git-send-email-yinghai@kernel.org
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
2013-01-24 20:19:52 +00:00
|
|
|
|
2019-10-11 11:50:51 +00:00
|
|
|
SYM_DATA_START_PAGE_ALIGNED(early_dynamic_pgts)
|
x86, 64bit: Use a #PF handler to materialize early mappings on demand
Linear mode (CR0.PG = 0) is mutually exclusive with 64-bit mode; all
64-bit code has to use page tables. This makes it awkward before we
have first set up properly all-covering page tables to access objects
that are outside the static kernel range.
So far we have dealt with that simply by mapping a fixed amount of
low memory, but that fails in at least two upcoming use cases:
1. We will support load and run kernel, struct boot_params, ramdisk,
command line, etc. above the 4 GiB mark.
2. need to access ramdisk early to get microcode to update that as
early possible.
We could use early_iomap to access them too, but it will make code to
messy and hard to be unified with 32 bit.
Hence, set up a #PF table and use a fixed number of buffers to set up
page tables on demand. If the buffers fill up then we simply flush
them and start over. These buffers are all in __initdata, so it does
not increase RAM usage at runtime.
Thus, with the help of the #PF handler, we can set the final kernel
mapping from blank, and switch to init_level4_pgt later.
During the switchover in head_64.S, before #PF handler is available,
we use three pages to handle kernel crossing 1G, 512G boundaries with
sharing page by playing games with page aliasing: the same page is
mapped twice in the higher-level tables with appropriate wraparound.
The kernel region itself will be properly mapped; other mappings may
be spurious.
early_make_pgtable is using kernel high mapping address to access pages
to set page table.
-v4: Add phys_base offset to make kexec happy, and add
init_mapping_kernel() - Yinghai
-v5: fix compiling with xen, and add back ident level3 and level2 for xen
also move back init_level4_pgt from BSS to DATA again.
because we have to clear it anyway. - Yinghai
-v6: switch to init_level4_pgt in init_mem_mapping. - Yinghai
-v7: remove not needed clear_page for init_level4_page
it is with fill 512,8,0 already in head_64.S - Yinghai
-v8: we need to keep that handler alive until init_mem_mapping and don't
let early_trap_init to trash that early #PF handler.
So split early_trap_pf_init out and move it down. - Yinghai
-v9: switchover only cover kernel space instead of 1G so could avoid
touch possible mem holes. - Yinghai
-v11: change far jmp back to far return to initial_code, that is needed
to fix failure that is reported by Konrad on AMD systems. - Yinghai
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Link: http://lkml.kernel.org/r/1359058816-7615-12-git-send-email-yinghai@kernel.org
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
2013-01-24 20:19:52 +00:00
|
|
|
.fill 512*EARLY_DYNAMIC_PAGE_TABLES,8,0
|
2019-10-11 11:50:51 +00:00
|
|
|
SYM_DATA_END(early_dynamic_pgts)
|
x86, 64bit: Use a #PF handler to materialize early mappings on demand
Linear mode (CR0.PG = 0) is mutually exclusive with 64-bit mode; all
64-bit code has to use page tables. This makes it awkward before we
have first set up properly all-covering page tables to access objects
that are outside the static kernel range.
So far we have dealt with that simply by mapping a fixed amount of
low memory, but that fails in at least two upcoming use cases:
1. We will support load and run kernel, struct boot_params, ramdisk,
command line, etc. above the 4 GiB mark.
2. need to access ramdisk early to get microcode to update that as
early possible.
We could use early_iomap to access them too, but it will make code to
messy and hard to be unified with 32 bit.
Hence, set up a #PF table and use a fixed number of buffers to set up
page tables on demand. If the buffers fill up then we simply flush
them and start over. These buffers are all in __initdata, so it does
not increase RAM usage at runtime.
Thus, with the help of the #PF handler, we can set the final kernel
mapping from blank, and switch to init_level4_pgt later.
During the switchover in head_64.S, before #PF handler is available,
we use three pages to handle kernel crossing 1G, 512G boundaries with
sharing page by playing games with page aliasing: the same page is
mapped twice in the higher-level tables with appropriate wraparound.
The kernel region itself will be properly mapped; other mappings may
be spurious.
early_make_pgtable is using kernel high mapping address to access pages
to set page table.
-v4: Add phys_base offset to make kexec happy, and add
init_mapping_kernel() - Yinghai
-v5: fix compiling with xen, and add back ident level3 and level2 for xen
also move back init_level4_pgt from BSS to DATA again.
because we have to clear it anyway. - Yinghai
-v6: switch to init_level4_pgt in init_mem_mapping. - Yinghai
-v7: remove not needed clear_page for init_level4_page
it is with fill 512,8,0 already in head_64.S - Yinghai
-v8: we need to keep that handler alive until init_mem_mapping and don't
let early_trap_init to trash that early #PF handler.
So split early_trap_pf_init out and move it down. - Yinghai
-v9: switchover only cover kernel space instead of 1G so could avoid
touch possible mem holes. - Yinghai
-v11: change far jmp back to far return to initial_code, that is needed
to fix failure that is reported by Konrad on AMD systems. - Yinghai
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Link: http://lkml.kernel.org/r/1359058816-7615-12-git-send-email-yinghai@kernel.org
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
2013-01-24 20:19:52 +00:00
|
|
|
|
2019-10-11 11:50:51 +00:00
|
|
|
SYM_DATA(early_recursion_flag, .long 0)
|
2019-10-03 09:52:37 +00:00
|
|
|
|
2009-10-14 21:46:55 +00:00
|
|
|
.data
|
x86, 64bit: Use a #PF handler to materialize early mappings on demand
Linear mode (CR0.PG = 0) is mutually exclusive with 64-bit mode; all
64-bit code has to use page tables. This makes it awkward before we
have first set up properly all-covering page tables to access objects
that are outside the static kernel range.
So far we have dealt with that simply by mapping a fixed amount of
low memory, but that fails in at least two upcoming use cases:
1. We will support load and run kernel, struct boot_params, ramdisk,
command line, etc. above the 4 GiB mark.
2. need to access ramdisk early to get microcode to update that as
early possible.
We could use early_iomap to access them too, but it will make code to
messy and hard to be unified with 32 bit.
Hence, set up a #PF table and use a fixed number of buffers to set up
page tables on demand. If the buffers fill up then we simply flush
them and start over. These buffers are all in __initdata, so it does
not increase RAM usage at runtime.
Thus, with the help of the #PF handler, we can set the final kernel
mapping from blank, and switch to init_level4_pgt later.
During the switchover in head_64.S, before #PF handler is available,
we use three pages to handle kernel crossing 1G, 512G boundaries with
sharing page by playing games with page aliasing: the same page is
mapped twice in the higher-level tables with appropriate wraparound.
The kernel region itself will be properly mapped; other mappings may
be spurious.
early_make_pgtable is using kernel high mapping address to access pages
to set page table.
-v4: Add phys_base offset to make kexec happy, and add
init_mapping_kernel() - Yinghai
-v5: fix compiling with xen, and add back ident level3 and level2 for xen
also move back init_level4_pgt from BSS to DATA again.
because we have to clear it anyway. - Yinghai
-v6: switch to init_level4_pgt in init_mem_mapping. - Yinghai
-v7: remove not needed clear_page for init_level4_page
it is with fill 512,8,0 already in head_64.S - Yinghai
-v8: we need to keep that handler alive until init_mem_mapping and don't
let early_trap_init to trash that early #PF handler.
So split early_trap_pf_init out and move it down. - Yinghai
-v9: switchover only cover kernel space instead of 1G so could avoid
touch possible mem holes. - Yinghai
-v11: change far jmp back to far return to initial_code, that is needed
to fix failure that is reported by Konrad on AMD systems. - Yinghai
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Link: http://lkml.kernel.org/r/1359058816-7615-12-git-send-email-yinghai@kernel.org
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
2013-01-24 20:19:52 +00:00
|
|
|
|
2018-12-10 19:07:28 +00:00
|
|
|
#if defined(CONFIG_XEN_PV) || defined(CONFIG_PVH)
|
2019-10-11 11:50:51 +00:00
|
|
|
SYM_DATA_START_PTI_ALIGNED(init_top_pgt)
|
2017-07-17 21:10:07 +00:00
|
|
|
.quad level3_ident_pgt - __START_KERNEL_map + _KERNPG_TABLE_NOENC
|
2018-02-16 11:49:46 +00:00
|
|
|
.org init_top_pgt + L4_PAGE_OFFSET*8, 0
|
2017-07-17 21:10:07 +00:00
|
|
|
.quad level3_ident_pgt - __START_KERNEL_map + _KERNPG_TABLE_NOENC
|
2018-02-16 11:49:46 +00:00
|
|
|
.org init_top_pgt + L4_START_KERNEL*8, 0
|
2007-05-02 17:27:07 +00:00
|
|
|
/* (2^48-(2*1024*1024*1024))/(2^39) = 511 */
|
2017-07-17 21:10:07 +00:00
|
|
|
.quad level3_kernel_pgt - __START_KERNEL_map + _PAGE_TABLE_NOENC
|
2017-12-04 14:07:39 +00:00
|
|
|
.fill PTI_USER_PGD_FILL,8,0
|
2019-10-11 11:50:51 +00:00
|
|
|
SYM_DATA_END(init_top_pgt)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2019-10-11 11:50:51 +00:00
|
|
|
SYM_DATA_START_PAGE_ALIGNED(level3_ident_pgt)
|
2017-07-17 21:10:07 +00:00
|
|
|
.quad level2_ident_pgt - __START_KERNEL_map + _KERNPG_TABLE_NOENC
|
x86, 64bit: Use a #PF handler to materialize early mappings on demand
Linear mode (CR0.PG = 0) is mutually exclusive with 64-bit mode; all
64-bit code has to use page tables. This makes it awkward before we
have first set up properly all-covering page tables to access objects
that are outside the static kernel range.
So far we have dealt with that simply by mapping a fixed amount of
low memory, but that fails in at least two upcoming use cases:
1. We will support load and run kernel, struct boot_params, ramdisk,
command line, etc. above the 4 GiB mark.
2. need to access ramdisk early to get microcode to update that as
early possible.
We could use early_iomap to access them too, but it will make code to
messy and hard to be unified with 32 bit.
Hence, set up a #PF table and use a fixed number of buffers to set up
page tables on demand. If the buffers fill up then we simply flush
them and start over. These buffers are all in __initdata, so it does
not increase RAM usage at runtime.
Thus, with the help of the #PF handler, we can set the final kernel
mapping from blank, and switch to init_level4_pgt later.
During the switchover in head_64.S, before #PF handler is available,
we use three pages to handle kernel crossing 1G, 512G boundaries with
sharing page by playing games with page aliasing: the same page is
mapped twice in the higher-level tables with appropriate wraparound.
The kernel region itself will be properly mapped; other mappings may
be spurious.
early_make_pgtable is using kernel high mapping address to access pages
to set page table.
-v4: Add phys_base offset to make kexec happy, and add
init_mapping_kernel() - Yinghai
-v5: fix compiling with xen, and add back ident level3 and level2 for xen
also move back init_level4_pgt from BSS to DATA again.
because we have to clear it anyway. - Yinghai
-v6: switch to init_level4_pgt in init_mem_mapping. - Yinghai
-v7: remove not needed clear_page for init_level4_page
it is with fill 512,8,0 already in head_64.S - Yinghai
-v8: we need to keep that handler alive until init_mem_mapping and don't
let early_trap_init to trash that early #PF handler.
So split early_trap_pf_init out and move it down. - Yinghai
-v9: switchover only cover kernel space instead of 1G so could avoid
touch possible mem holes. - Yinghai
-v11: change far jmp back to far return to initial_code, that is needed
to fix failure that is reported by Konrad on AMD systems. - Yinghai
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Link: http://lkml.kernel.org/r/1359058816-7615-12-git-send-email-yinghai@kernel.org
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
2013-01-24 20:19:52 +00:00
|
|
|
.fill 511, 8, 0
|
2019-10-11 11:50:51 +00:00
|
|
|
SYM_DATA_END(level3_ident_pgt)
|
|
|
|
|
SYM_DATA_START_PAGE_ALIGNED(level2_ident_pgt)
|
2018-04-06 20:55:13 +00:00
|
|
|
/*
|
|
|
|
|
* Since I easily can, map the first 1G.
|
x86, 64bit: Use a #PF handler to materialize early mappings on demand
Linear mode (CR0.PG = 0) is mutually exclusive with 64-bit mode; all
64-bit code has to use page tables. This makes it awkward before we
have first set up properly all-covering page tables to access objects
that are outside the static kernel range.
So far we have dealt with that simply by mapping a fixed amount of
low memory, but that fails in at least two upcoming use cases:
1. We will support load and run kernel, struct boot_params, ramdisk,
command line, etc. above the 4 GiB mark.
2. need to access ramdisk early to get microcode to update that as
early possible.
We could use early_iomap to access them too, but it will make code to
messy and hard to be unified with 32 bit.
Hence, set up a #PF table and use a fixed number of buffers to set up
page tables on demand. If the buffers fill up then we simply flush
them and start over. These buffers are all in __initdata, so it does
not increase RAM usage at runtime.
Thus, with the help of the #PF handler, we can set the final kernel
mapping from blank, and switch to init_level4_pgt later.
During the switchover in head_64.S, before #PF handler is available,
we use three pages to handle kernel crossing 1G, 512G boundaries with
sharing page by playing games with page aliasing: the same page is
mapped twice in the higher-level tables with appropriate wraparound.
The kernel region itself will be properly mapped; other mappings may
be spurious.
early_make_pgtable is using kernel high mapping address to access pages
to set page table.
-v4: Add phys_base offset to make kexec happy, and add
init_mapping_kernel() - Yinghai
-v5: fix compiling with xen, and add back ident level3 and level2 for xen
also move back init_level4_pgt from BSS to DATA again.
because we have to clear it anyway. - Yinghai
-v6: switch to init_level4_pgt in init_mem_mapping. - Yinghai
-v7: remove not needed clear_page for init_level4_page
it is with fill 512,8,0 already in head_64.S - Yinghai
-v8: we need to keep that handler alive until init_mem_mapping and don't
let early_trap_init to trash that early #PF handler.
So split early_trap_pf_init out and move it down. - Yinghai
-v9: switchover only cover kernel space instead of 1G so could avoid
touch possible mem holes. - Yinghai
-v11: change far jmp back to far return to initial_code, that is needed
to fix failure that is reported by Konrad on AMD systems. - Yinghai
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Link: http://lkml.kernel.org/r/1359058816-7615-12-git-send-email-yinghai@kernel.org
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
2013-01-24 20:19:52 +00:00
|
|
|
* Don't set NX because code runs from these pages.
|
2018-04-06 20:55:13 +00:00
|
|
|
*
|
|
|
|
|
* Note: This sets _PAGE_GLOBAL despite whether
|
|
|
|
|
* the CPU supports it or it is enabled. But,
|
|
|
|
|
* the CPU should ignore the bit.
|
x86, 64bit: Use a #PF handler to materialize early mappings on demand
Linear mode (CR0.PG = 0) is mutually exclusive with 64-bit mode; all
64-bit code has to use page tables. This makes it awkward before we
have first set up properly all-covering page tables to access objects
that are outside the static kernel range.
So far we have dealt with that simply by mapping a fixed amount of
low memory, but that fails in at least two upcoming use cases:
1. We will support load and run kernel, struct boot_params, ramdisk,
command line, etc. above the 4 GiB mark.
2. need to access ramdisk early to get microcode to update that as
early possible.
We could use early_iomap to access them too, but it will make code to
messy and hard to be unified with 32 bit.
Hence, set up a #PF table and use a fixed number of buffers to set up
page tables on demand. If the buffers fill up then we simply flush
them and start over. These buffers are all in __initdata, so it does
not increase RAM usage at runtime.
Thus, with the help of the #PF handler, we can set the final kernel
mapping from blank, and switch to init_level4_pgt later.
During the switchover in head_64.S, before #PF handler is available,
we use three pages to handle kernel crossing 1G, 512G boundaries with
sharing page by playing games with page aliasing: the same page is
mapped twice in the higher-level tables with appropriate wraparound.
The kernel region itself will be properly mapped; other mappings may
be spurious.
early_make_pgtable is using kernel high mapping address to access pages
to set page table.
-v4: Add phys_base offset to make kexec happy, and add
init_mapping_kernel() - Yinghai
-v5: fix compiling with xen, and add back ident level3 and level2 for xen
also move back init_level4_pgt from BSS to DATA again.
because we have to clear it anyway. - Yinghai
-v6: switch to init_level4_pgt in init_mem_mapping. - Yinghai
-v7: remove not needed clear_page for init_level4_page
it is with fill 512,8,0 already in head_64.S - Yinghai
-v8: we need to keep that handler alive until init_mem_mapping and don't
let early_trap_init to trash that early #PF handler.
So split early_trap_pf_init out and move it down. - Yinghai
-v9: switchover only cover kernel space instead of 1G so could avoid
touch possible mem holes. - Yinghai
-v11: change far jmp back to far return to initial_code, that is needed
to fix failure that is reported by Konrad on AMD systems. - Yinghai
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Link: http://lkml.kernel.org/r/1359058816-7615-12-git-send-email-yinghai@kernel.org
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
2013-01-24 20:19:52 +00:00
|
|
|
*/
|
|
|
|
|
PMDS(0, __PAGE_KERNEL_IDENT_LARGE_EXEC, PTRS_PER_PMD)
|
2019-10-11 11:50:51 +00:00
|
|
|
SYM_DATA_END(level2_ident_pgt)
|
2017-09-29 14:08:19 +00:00
|
|
|
#else
|
2019-10-11 11:50:51 +00:00
|
|
|
SYM_DATA_START_PTI_ALIGNED(init_top_pgt)
|
2017-09-29 14:08:19 +00:00
|
|
|
.fill 512,8,0
|
2017-12-04 14:07:39 +00:00
|
|
|
.fill PTI_USER_PGD_FILL,8,0
|
2019-10-11 11:50:51 +00:00
|
|
|
SYM_DATA_END(init_top_pgt)
|
x86, 64bit: Use a #PF handler to materialize early mappings on demand
Linear mode (CR0.PG = 0) is mutually exclusive with 64-bit mode; all
64-bit code has to use page tables. This makes it awkward before we
have first set up properly all-covering page tables to access objects
that are outside the static kernel range.
So far we have dealt with that simply by mapping a fixed amount of
low memory, but that fails in at least two upcoming use cases:
1. We will support load and run kernel, struct boot_params, ramdisk,
command line, etc. above the 4 GiB mark.
2. need to access ramdisk early to get microcode to update that as
early possible.
We could use early_iomap to access them too, but it will make code to
messy and hard to be unified with 32 bit.
Hence, set up a #PF table and use a fixed number of buffers to set up
page tables on demand. If the buffers fill up then we simply flush
them and start over. These buffers are all in __initdata, so it does
not increase RAM usage at runtime.
Thus, with the help of the #PF handler, we can set the final kernel
mapping from blank, and switch to init_level4_pgt later.
During the switchover in head_64.S, before #PF handler is available,
we use three pages to handle kernel crossing 1G, 512G boundaries with
sharing page by playing games with page aliasing: the same page is
mapped twice in the higher-level tables with appropriate wraparound.
The kernel region itself will be properly mapped; other mappings may
be spurious.
early_make_pgtable is using kernel high mapping address to access pages
to set page table.
-v4: Add phys_base offset to make kexec happy, and add
init_mapping_kernel() - Yinghai
-v5: fix compiling with xen, and add back ident level3 and level2 for xen
also move back init_level4_pgt from BSS to DATA again.
because we have to clear it anyway. - Yinghai
-v6: switch to init_level4_pgt in init_mem_mapping. - Yinghai
-v7: remove not needed clear_page for init_level4_page
it is with fill 512,8,0 already in head_64.S - Yinghai
-v8: we need to keep that handler alive until init_mem_mapping and don't
let early_trap_init to trash that early #PF handler.
So split early_trap_pf_init out and move it down. - Yinghai
-v9: switchover only cover kernel space instead of 1G so could avoid
touch possible mem holes. - Yinghai
-v11: change far jmp back to far return to initial_code, that is needed
to fix failure that is reported by Konrad on AMD systems. - Yinghai
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Link: http://lkml.kernel.org/r/1359058816-7615-12-git-send-email-yinghai@kernel.org
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
2013-01-24 20:19:52 +00:00
|
|
|
#endif
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2017-06-06 11:31:28 +00:00
|
|
|
#ifdef CONFIG_X86_5LEVEL
|
2019-10-11 11:50:51 +00:00
|
|
|
SYM_DATA_START_PAGE_ALIGNED(level4_kernel_pgt)
|
2017-06-06 11:31:28 +00:00
|
|
|
.fill 511,8,0
|
2017-07-17 21:10:07 +00:00
|
|
|
.quad level3_kernel_pgt - __START_KERNEL_map + _PAGE_TABLE_NOENC
|
2019-10-11 11:50:51 +00:00
|
|
|
SYM_DATA_END(level4_kernel_pgt)
|
2017-06-06 11:31:28 +00:00
|
|
|
#endif
|
|
|
|
|
|
2019-10-11 11:50:51 +00:00
|
|
|
SYM_DATA_START_PAGE_ALIGNED(level3_kernel_pgt)
|
2008-06-25 04:19:16 +00:00
|
|
|
.fill L3_START_KERNEL,8,0
|
2005-04-16 22:20:36 +00:00
|
|
|
/* (2^48-(2*1024*1024*1024)-((2^39)*511))/(2^30) = 510 */
|
2017-07-17 21:10:07 +00:00
|
|
|
.quad level2_kernel_pgt - __START_KERNEL_map + _KERNPG_TABLE_NOENC
|
|
|
|
|
.quad level2_fixmap_pgt - __START_KERNEL_map + _PAGE_TABLE_NOENC
|
2019-10-11 11:50:51 +00:00
|
|
|
SYM_DATA_END(level3_kernel_pgt)
|
2007-07-16 06:37:28 +00:00
|
|
|
|
2019-10-11 11:50:51 +00:00
|
|
|
SYM_DATA_START_PAGE_ALIGNED(level2_kernel_pgt)
|
2008-02-21 10:04:11 +00:00
|
|
|
/*
|
2008-02-21 11:50:51 +00:00
|
|
|
* 512 MB kernel mapping. We spend a full page on this pagetable
|
2008-02-21 10:04:11 +00:00
|
|
|
* anyway.
|
|
|
|
|
*
|
|
|
|
|
* The kernel code+data+bss must not be bigger than that.
|
|
|
|
|
*
|
2008-02-21 11:50:51 +00:00
|
|
|
* (NOTE: at +512MB starts the module area, see MODULES_VADDR.
|
2008-02-21 10:04:11 +00:00
|
|
|
* If you want to increase this then increase MODULES_VADDR
|
|
|
|
|
* too.)
|
2018-04-06 20:55:13 +00:00
|
|
|
*
|
|
|
|
|
* This table is eventually used by the kernel during normal
|
|
|
|
|
* runtime. Care must be taken to clear out undesired bits
|
|
|
|
|
* later, like _PAGE_RW or _PAGE_GLOBAL in some cases.
|
2008-02-21 10:04:11 +00:00
|
|
|
*/
|
2008-07-01 23:46:35 +00:00
|
|
|
PMDS(0, __PAGE_KERNEL_LARGE_EXEC,
|
2008-02-21 12:39:30 +00:00
|
|
|
KERNEL_IMAGE_SIZE/PMD_SIZE)
|
2019-10-11 11:50:51 +00:00
|
|
|
SYM_DATA_END(level2_kernel_pgt)
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2019-10-11 11:50:51 +00:00
|
|
|
SYM_DATA_START_PAGE_ALIGNED(level2_fixmap_pgt)
|
2018-09-20 02:58:28 +00:00
|
|
|
.fill (512 - 4 - FIXMAP_PMD_NUM),8,0
|
|
|
|
|
pgtno = 0
|
|
|
|
|
.rept (FIXMAP_PMD_NUM)
|
|
|
|
|
.quad level1_fixmap_pgt + (pgtno << PAGE_SHIFT) - __START_KERNEL_map \
|
|
|
|
|
+ _PAGE_TABLE_NOENC;
|
|
|
|
|
pgtno = pgtno + 1
|
|
|
|
|
.endr
|
|
|
|
|
/* 6 MB reserved space + a 2MB hole */
|
|
|
|
|
.fill 4,8,0
|
2019-10-11 11:50:51 +00:00
|
|
|
SYM_DATA_END(level2_fixmap_pgt)
|
x86, 64bit: Use a #PF handler to materialize early mappings on demand
Linear mode (CR0.PG = 0) is mutually exclusive with 64-bit mode; all
64-bit code has to use page tables. This makes it awkward before we
have first set up properly all-covering page tables to access objects
that are outside the static kernel range.
So far we have dealt with that simply by mapping a fixed amount of
low memory, but that fails in at least two upcoming use cases:
1. We will support load and run kernel, struct boot_params, ramdisk,
command line, etc. above the 4 GiB mark.
2. need to access ramdisk early to get microcode to update that as
early possible.
We could use early_iomap to access them too, but it will make code to
messy and hard to be unified with 32 bit.
Hence, set up a #PF table and use a fixed number of buffers to set up
page tables on demand. If the buffers fill up then we simply flush
them and start over. These buffers are all in __initdata, so it does
not increase RAM usage at runtime.
Thus, with the help of the #PF handler, we can set the final kernel
mapping from blank, and switch to init_level4_pgt later.
During the switchover in head_64.S, before #PF handler is available,
we use three pages to handle kernel crossing 1G, 512G boundaries with
sharing page by playing games with page aliasing: the same page is
mapped twice in the higher-level tables with appropriate wraparound.
The kernel region itself will be properly mapped; other mappings may
be spurious.
early_make_pgtable is using kernel high mapping address to access pages
to set page table.
-v4: Add phys_base offset to make kexec happy, and add
init_mapping_kernel() - Yinghai
-v5: fix compiling with xen, and add back ident level3 and level2 for xen
also move back init_level4_pgt from BSS to DATA again.
because we have to clear it anyway. - Yinghai
-v6: switch to init_level4_pgt in init_mem_mapping. - Yinghai
-v7: remove not needed clear_page for init_level4_page
it is with fill 512,8,0 already in head_64.S - Yinghai
-v8: we need to keep that handler alive until init_mem_mapping and don't
let early_trap_init to trash that early #PF handler.
So split early_trap_pf_init out and move it down. - Yinghai
-v9: switchover only cover kernel space instead of 1G so could avoid
touch possible mem holes. - Yinghai
-v11: change far jmp back to far return to initial_code, that is needed
to fix failure that is reported by Konrad on AMD systems. - Yinghai
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Link: http://lkml.kernel.org/r/1359058816-7615-12-git-send-email-yinghai@kernel.org
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
2013-01-24 20:19:52 +00:00
|
|
|
|
2019-10-11 11:50:51 +00:00
|
|
|
SYM_DATA_START_PAGE_ALIGNED(level1_fixmap_pgt)
|
2018-09-20 02:58:28 +00:00
|
|
|
.rept (FIXMAP_PMD_NUM)
|
x86, 64bit: Use a #PF handler to materialize early mappings on demand
Linear mode (CR0.PG = 0) is mutually exclusive with 64-bit mode; all
64-bit code has to use page tables. This makes it awkward before we
have first set up properly all-covering page tables to access objects
that are outside the static kernel range.
So far we have dealt with that simply by mapping a fixed amount of
low memory, but that fails in at least two upcoming use cases:
1. We will support load and run kernel, struct boot_params, ramdisk,
command line, etc. above the 4 GiB mark.
2. need to access ramdisk early to get microcode to update that as
early possible.
We could use early_iomap to access them too, but it will make code to
messy and hard to be unified with 32 bit.
Hence, set up a #PF table and use a fixed number of buffers to set up
page tables on demand. If the buffers fill up then we simply flush
them and start over. These buffers are all in __initdata, so it does
not increase RAM usage at runtime.
Thus, with the help of the #PF handler, we can set the final kernel
mapping from blank, and switch to init_level4_pgt later.
During the switchover in head_64.S, before #PF handler is available,
we use three pages to handle kernel crossing 1G, 512G boundaries with
sharing page by playing games with page aliasing: the same page is
mapped twice in the higher-level tables with appropriate wraparound.
The kernel region itself will be properly mapped; other mappings may
be spurious.
early_make_pgtable is using kernel high mapping address to access pages
to set page table.
-v4: Add phys_base offset to make kexec happy, and add
init_mapping_kernel() - Yinghai
-v5: fix compiling with xen, and add back ident level3 and level2 for xen
also move back init_level4_pgt from BSS to DATA again.
because we have to clear it anyway. - Yinghai
-v6: switch to init_level4_pgt in init_mem_mapping. - Yinghai
-v7: remove not needed clear_page for init_level4_page
it is with fill 512,8,0 already in head_64.S - Yinghai
-v8: we need to keep that handler alive until init_mem_mapping and don't
let early_trap_init to trash that early #PF handler.
So split early_trap_pf_init out and move it down. - Yinghai
-v9: switchover only cover kernel space instead of 1G so could avoid
touch possible mem holes. - Yinghai
-v11: change far jmp back to far return to initial_code, that is needed
to fix failure that is reported by Konrad on AMD systems. - Yinghai
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Link: http://lkml.kernel.org/r/1359058816-7615-12-git-send-email-yinghai@kernel.org
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
2013-01-24 20:19:52 +00:00
|
|
|
.fill 512,8,0
|
2018-09-20 02:58:28 +00:00
|
|
|
.endr
|
2019-10-11 11:50:51 +00:00
|
|
|
SYM_DATA_END(level1_fixmap_pgt)
|
2007-05-02 17:27:07 +00:00
|
|
|
|
2007-05-02 17:27:06 +00:00
|
|
|
#undef PMDS
|
2005-04-16 22:20:36 +00:00
|
|
|
|
2006-01-17 06:03:32 +00:00
|
|
|
.data
|
2005-04-16 22:20:36 +00:00
|
|
|
.align 16
|
2019-10-11 11:50:51 +00:00
|
|
|
|
|
|
|
|
SYM_DATA(early_gdt_descr, .word GDT_ENTRIES*8-1)
|
|
|
|
|
SYM_DATA_LOCAL(early_gdt_descr_base, .quad INIT_PER_CPU_VAR(gdt_page))
|
|
|
|
|
|
|
|
|
|
.align 16
|
|
|
|
|
/* This must match the first entry in level2_kernel_pgt */
|
|
|
|
|
SYM_DATA(phys_base, .quad 0x0)
|
2016-01-11 16:04:34 +00:00
|
|
|
EXPORT_SYMBOL(phys_base)
|
2007-05-02 17:27:07 +00:00
|
|
|
|
2008-07-08 22:06:44 +00:00
|
|
|
#include "../../x86/xen/xen-head.S"
|
2017-09-19 02:43:37 +00:00
|
|
|
|
2009-09-20 22:14:14 +00:00
|
|
|
__PAGE_ALIGNED_BSS
|
2019-10-11 11:50:51 +00:00
|
|
|
SYM_DATA_START_PAGE_ALIGNED(empty_zero_page)
|
2006-03-25 15:30:01 +00:00
|
|
|
.skip PAGE_SIZE
|
2019-10-11 11:50:51 +00:00
|
|
|
SYM_DATA_END(empty_zero_page)
|
2016-01-11 16:04:34 +00:00
|
|
|
EXPORT_SYMBOL(empty_zero_page)
|
2015-02-13 22:39:25 +00:00
|
|
|
|