License cleanup: add SPDX GPL-2.0 license identifier to files with no license
Many source files in the tree are missing licensing information, which
makes it harder for compliance tools to determine the correct license.
By default all files without license information are under the default
license of the kernel, which is GPL version 2.
Update the files which contain no license information with the 'GPL-2.0'
SPDX license identifier. The SPDX identifier is a legally binding
shorthand, which can be used instead of the full boiler plate text.
This patch is based on work done by Thomas Gleixner and Kate Stewart and
Philippe Ombredanne.
How this work was done:
Patches were generated and checked against linux-4.14-rc6 for a subset of
the use cases:
- file had no licensing information it it.
- file was a */uapi/* one with no licensing information in it,
- file was a */uapi/* one with existing licensing information,
Further patches will be generated in subsequent months to fix up cases
where non-standard license headers were used, and references to license
had to be inferred by heuristics based on keywords.
The analysis to determine which SPDX License Identifier to be applied to
a file was done in a spreadsheet of side by side results from of the
output of two independent scanners (ScanCode & Windriver) producing SPDX
tag:value files created by Philippe Ombredanne. Philippe prepared the
base worksheet, and did an initial spot review of a few 1000 files.
The 4.13 kernel was the starting point of the analysis with 60,537 files
assessed. Kate Stewart did a file by file comparison of the scanner
results in the spreadsheet to determine which SPDX license identifier(s)
to be applied to the file. She confirmed any determination that was not
immediately clear with lawyers working with the Linux Foundation.
Criteria used to select files for SPDX license identifier tagging was:
- Files considered eligible had to be source code files.
- Make and config files were included as candidates if they contained >5
lines of source
- File already had some variant of a license header in it (even if <5
lines).
All documentation files were explicitly excluded.
The following heuristics were used to determine which SPDX license
identifiers to apply.
- when both scanners couldn't find any license traces, file was
considered to have no license information in it, and the top level
COPYING file license applied.
For non */uapi/* files that summary was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 11139
and resulted in the first patch in this series.
If that file was a */uapi/* path one, it was "GPL-2.0 WITH
Linux-syscall-note" otherwise it was "GPL-2.0". Results of that was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 WITH Linux-syscall-note 930
and resulted in the second patch in this series.
- if a file had some form of licensing information in it, and was one
of the */uapi/* ones, it was denoted with the Linux-syscall-note if
any GPL family license was found in the file or had no licensing in
it (per prior point). Results summary:
SPDX license identifier # files
---------------------------------------------------|------
GPL-2.0 WITH Linux-syscall-note 270
GPL-2.0+ WITH Linux-syscall-note 169
((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) 21
((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) 17
LGPL-2.1+ WITH Linux-syscall-note 15
GPL-1.0+ WITH Linux-syscall-note 14
((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) 5
LGPL-2.0+ WITH Linux-syscall-note 4
LGPL-2.1 WITH Linux-syscall-note 3
((GPL-2.0 WITH Linux-syscall-note) OR MIT) 3
((GPL-2.0 WITH Linux-syscall-note) AND MIT) 1
and that resulted in the third patch in this series.
- when the two scanners agreed on the detected license(s), that became
the concluded license(s).
- when there was disagreement between the two scanners (one detected a
license but the other didn't, or they both detected different
licenses) a manual inspection of the file occurred.
- In most cases a manual inspection of the information in the file
resulted in a clear resolution of the license that should apply (and
which scanner probably needed to revisit its heuristics).
- When it was not immediately clear, the license identifier was
confirmed with lawyers working with the Linux Foundation.
- If there was any question as to the appropriate license identifier,
the file was flagged for further research and to be revisited later
in time.
In total, over 70 hours of logged manual review was done on the
spreadsheet to determine the SPDX license identifiers to apply to the
source files by Kate, Philippe, Thomas and, in some cases, confirmation
by lawyers working with the Linux Foundation.
Kate also obtained a third independent scan of the 4.13 code base from
FOSSology, and compared selected files where the other two scanners
disagreed against that SPDX file, to see if there was new insights. The
Windriver scanner is based on an older version of FOSSology in part, so
they are related.
Thomas did random spot checks in about 500 files from the spreadsheets
for the uapi headers and agreed with SPDX license identifier in the
files he inspected. For the non-uapi files Thomas did random spot checks
in about 15000 files.
In initial set of patches against 4.14-rc6, 3 files were found to have
copy/paste license identifier errors, and have been fixed to reflect the
correct identifier.
Additionally Philippe spent 10 hours this week doing a detailed manual
inspection and review of the 12,461 patched files from the initial patch
version early this week with:
- a full scancode scan run, collecting the matched texts, detected
license ids and scores
- reviewing anything where there was a license detected (about 500+
files) to ensure that the applied SPDX license was correct
- reviewing anything where there was no detection but the patch license
was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied
SPDX license was correct
This produced a worksheet with 20 files needing minor correction. This
worksheet was then exported into 3 different .csv files for the
different types of files to be modified.
These .csv files were then reviewed by Greg. Thomas wrote a script to
parse the csv files and add the proper SPDX tag to the file, in the
format that the file expected. This script was further refined by Greg
based on the output to detect more types of files automatically and to
distinguish between header and source .c files (which need different
comment types.) Finally Greg ran the script using the .csv files to
generate the patches.
Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Philippe Ombredanne <pombredanne@nexb.com>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2017-11-01 14:07:57 +00:00
|
|
|
/* SPDX-License-Identifier: GPL-2.0 */
|
2008-10-23 05:26:29 +00:00
|
|
|
#ifndef _ASM_X86_PROCESSOR_H
|
|
|
|
|
#define _ASM_X86_PROCESSOR_H
|
2008-01-30 12:31:03 +00:00
|
|
|
|
2008-01-30 12:31:27 +00:00
|
|
|
#include <asm/processor-flags.h>
|
|
|
|
|
|
2008-01-30 12:31:27 +00:00
|
|
|
/* Forward declaration, a strange C thing */
|
|
|
|
|
struct task_struct;
|
|
|
|
|
struct mm_struct;
|
2019-11-11 22:03:21 +00:00
|
|
|
struct io_bitmap;
|
2015-07-29 05:41:16 +00:00
|
|
|
struct vm86;
|
2008-01-30 12:31:27 +00:00
|
|
|
|
2008-01-30 12:31:57 +00:00
|
|
|
#include <asm/math_emu.h>
|
|
|
|
|
#include <asm/segment.h>
|
|
|
|
|
#include <asm/types.h>
|
2015-09-05 07:32:43 +00:00
|
|
|
#include <uapi/asm/sigcontext.h>
|
2008-01-30 12:31:57 +00:00
|
|
|
#include <asm/current.h>
|
2016-01-26 21:12:04 +00:00
|
|
|
#include <asm/cpufeatures.h>
|
2022-11-24 15:31:27 +00:00
|
|
|
#include <asm/cpuid.h>
|
2008-01-30 12:31:57 +00:00
|
|
|
#include <asm/page.h>
|
2009-02-11 18:20:05 +00:00
|
|
|
#include <asm/pgtable_types.h>
|
2008-01-30 12:31:33 +00:00
|
|
|
#include <asm/percpu.h>
|
2008-01-30 12:31:57 +00:00
|
|
|
#include <asm/msr.h>
|
|
|
|
|
#include <asm/desc_defs.h>
|
2008-01-30 12:32:38 +00:00
|
|
|
#include <asm/nops.h>
|
2012-03-28 17:11:12 +00:00
|
|
|
#include <asm/special_insns.h>
|
2015-04-22 07:57:24 +00:00
|
|
|
#include <asm/fpu/types.h>
|
2017-07-11 15:33:45 +00:00
|
|
|
#include <asm/unwind_hints.h>
|
x86/vmx: Introduce VMX_FEATURES_*
Add a VMX-specific variant of X86_FEATURE_* flags, which will eventually
supplant the synthetic VMX flags defined in cpufeatures word 8. Use the
Intel-defined layouts for the major VMX execution controls so that their
word entries can be directly populated from their respective MSRs, and
so that the VMX_FEATURE_* flags can be used to define the existing bit
definitions in asm/vmx.h, i.e. force developers to define a VMX_FEATURE
flag when adding support for a new hardware feature.
The majority of Intel's (and compatible CPU's) VMX capabilities are
enumerated via MSRs and not CPUID, i.e. querying /proc/cpuinfo doesn't
naturally provide any insight into the virtualization capabilities of
VMX enabled CPUs. Commit
e38e05a85828d ("x86: extended "flags" to show virtualization HW feature
in /proc/cpuinfo")
attempted to address the issue by synthesizing select VMX features into
a Linux-defined word in cpufeatures.
Lack of reporting of VMX capabilities via /proc/cpuinfo is problematic
because there is no sane way for a user to query the capabilities of
their platform, e.g. when trying to find a platform to test a feature or
debug an issue that has a hardware dependency. Lack of reporting is
especially problematic when the user isn't familiar with VMX, e.g. the
format of the MSRs is non-standard, existence of some MSRs is reported
by bits in other MSRs, several "features" from KVM's point of view are
enumerated as 3+ distinct features by hardware, etc...
The synthetic cpufeatures approach has several flaws:
- The set of synthesized VMX flags has become extremely stale with
respect to the full set of VMX features, e.g. only one new flag
(EPT A/D) has been added in the the decade since the introduction of
the synthetic VMX features. Failure to keep the VMX flags up to
date is likely due to the lack of a mechanism that forces developers
to consider whether or not a new feature is worth reporting.
- The synthetic flags may incorrectly be misinterpreted as affecting
kernel behavior, i.e. KVM, the kernel's sole consumer of VMX,
completely ignores the synthetic flags.
- New CPU vendors that support VMX have duplicated the hideous code
that propagates VMX features from MSRs to cpufeatures. Bringing the
synthetic VMX flags up to date would exacerbate the copy+paste
trainwreck.
Define separate VMX_FEATURE flags to set the stage for enumerating VMX
capabilities outside of the cpu_has() framework, and for adding
functional usage of VMX_FEATURE_* to help ensure the features reported
via /proc/cpuinfo is up to date with respect to kernel recognition of
VMX capabilities.
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20191221044513.21680-10-sean.j.christopherson@intel.com
2019-12-21 04:45:03 +00:00
|
|
|
#include <asm/vmxfeatures.h>
|
2020-03-20 14:53:48 +00:00
|
|
|
#include <asm/vdso/processor.h>
|
2023-06-13 00:10:52 +00:00
|
|
|
#include <asm/shstk.h>
|
2008-02-21 03:24:40 +00:00
|
|
|
|
2008-01-30 12:31:57 +00:00
|
|
|
#include <linux/personality.h>
|
2008-01-30 12:31:33 +00:00
|
|
|
#include <linux/cache.h>
|
2008-01-30 12:31:57 +00:00
|
|
|
#include <linux/threads.h>
|
2009-09-02 09:49:52 +00:00
|
|
|
#include <linux/math64.h>
|
2010-03-25 13:51:50 +00:00
|
|
|
#include <linux/err.h>
|
2012-03-28 17:11:12 +00:00
|
|
|
#include <linux/irqflags.h>
|
2017-07-17 21:10:07 +00:00
|
|
|
#include <linux/mem_encrypt.h>
|
2012-03-28 17:11:12 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* We handle most unaligned accesses in hardware. On the other hand
|
|
|
|
|
* unaligned DMA can be quite expensive on some Nehalem processors.
|
|
|
|
|
*
|
|
|
|
|
* Based on this we disable the IP header alignment in network drivers.
|
|
|
|
|
*/
|
|
|
|
|
#define NET_IP_ALIGN 0
|
2008-01-30 12:31:27 +00:00
|
|
|
|
2009-06-01 18:13:10 +00:00
|
|
|
#define HBP_NUM 4
|
2008-01-30 12:31:27 +00:00
|
|
|
|
2015-05-24 07:58:12 +00:00
|
|
|
/*
|
|
|
|
|
* These alignment constraints are for performance in the vSMP case,
|
|
|
|
|
* but in the task_struct case we must also meet hardware imposed
|
|
|
|
|
* alignment requirements of the FPU state:
|
|
|
|
|
*/
|
2008-01-30 12:31:31 +00:00
|
|
|
#ifdef CONFIG_X86_VSMP
|
2008-02-21 03:24:40 +00:00
|
|
|
# define ARCH_MIN_TASKALIGN (1 << INTERNODE_CACHE_SHIFT)
|
|
|
|
|
# define ARCH_MIN_MMSTRUCT_ALIGN (1 << INTERNODE_CACHE_SHIFT)
|
2008-01-30 12:31:31 +00:00
|
|
|
#else
|
2015-05-24 07:58:12 +00:00
|
|
|
# define ARCH_MIN_TASKALIGN __alignof__(union fpregs_state)
|
2008-02-21 03:24:40 +00:00
|
|
|
# define ARCH_MIN_MMSTRUCT_ALIGN 0
|
2008-01-30 12:31:31 +00:00
|
|
|
#endif
|
|
|
|
|
|
x86/tlb_info: get last level TLB entry number of CPU
For 4KB pages, x86 CPU has 2 or 1 level TLB, first level is data TLB and
instruction TLB, second level is shared TLB for both data and instructions.
For hupe page TLB, usually there is just one level and seperated by 2MB/4MB
and 1GB.
Although each levels TLB size is important for performance tuning, but for
genernal and rude optimizing, last level TLB entry number is suitable. And
in fact, last level TLB always has the biggest entry number.
This patch will get the biggest TLB entry number and use it in furture TLB
optimizing.
Accroding Borislav's suggestion, except tlb_ll[i/d]_* array, other
function and data will be released after system boot up.
For all kinds of x86 vendor friendly, vendor specific code was moved to its
specific files.
Signed-off-by: Alex Shi <alex.shi@intel.com>
Link: http://lkml.kernel.org/r/1340845344-27557-2-git-send-email-alex.shi@intel.com
Signed-off-by: H. Peter Anvin <hpa@zytor.com>
2012-06-28 01:02:16 +00:00
|
|
|
enum tlb_infos {
|
|
|
|
|
ENTRIES,
|
|
|
|
|
NR_INFO
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
extern u16 __read_mostly tlb_lli_4k[NR_INFO];
|
|
|
|
|
extern u16 __read_mostly tlb_lli_2m[NR_INFO];
|
|
|
|
|
extern u16 __read_mostly tlb_lli_4m[NR_INFO];
|
|
|
|
|
extern u16 __read_mostly tlb_lld_4k[NR_INFO];
|
|
|
|
|
extern u16 __read_mostly tlb_lld_2m[NR_INFO];
|
|
|
|
|
extern u16 __read_mostly tlb_lld_4m[NR_INFO];
|
x86, cpu: Detect more TLB configuration
The Intel Software Developer’s Manual covers few more TLB
configurations exposed as CPUID 2 descriptors:
61H Instruction TLB: 4 KByte pages, fully associative, 48 entries
63H Data TLB: 1 GByte pages, 4-way set associative, 4 entries
76H Instruction TLB: 2M/4M pages, fully associative, 8 entries
B5H Instruction TLB: 4KByte pages, 8-way set associative, 64 entries
B6H Instruction TLB: 4KByte pages, 8-way set associative, 128 entries
C1H Shared 2nd-Level TLB: 4 KByte/2MByte pages, 8-way associative, 1024 entries
C2H DTLB DTLB: 2 MByte/$MByte pages, 4-way associative, 16 entries
Let's detect them as well.
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Link: http://lkml.kernel.org/r/1387801018-14499-1-git-send-email-kirill.shutemov@linux.intel.com
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
2013-12-23 12:16:58 +00:00
|
|
|
extern u16 __read_mostly tlb_lld_1g[NR_INFO];
|
2012-06-28 01:02:19 +00:00
|
|
|
|
2008-01-30 12:31:33 +00:00
|
|
|
/*
|
2023-08-14 08:18:29 +00:00
|
|
|
* CPU type and hardware bug flags. Kept separately for each CPU.
|
2008-01-30 12:31:33 +00:00
|
|
|
*/
|
|
|
|
|
|
2023-08-14 08:18:29 +00:00
|
|
|
struct cpuinfo_topology {
|
|
|
|
|
// Real APIC ID read from the local APIC
|
|
|
|
|
u32 apicid;
|
|
|
|
|
// The initial APIC ID provided by CPUID
|
|
|
|
|
u32 initial_apicid;
|
2023-08-14 08:18:30 +00:00
|
|
|
|
|
|
|
|
// Physical package ID
|
|
|
|
|
u32 pkg_id;
|
2023-08-14 08:18:31 +00:00
|
|
|
|
|
|
|
|
// Physical die ID on AMD, Relative on Intel
|
|
|
|
|
u32 die_id;
|
2023-08-14 08:18:34 +00:00
|
|
|
|
2023-08-14 08:18:35 +00:00
|
|
|
// Compute unit ID - AMD specific
|
|
|
|
|
u32 cu_id;
|
|
|
|
|
|
2023-08-14 08:18:34 +00:00
|
|
|
// Core ID relative to the package
|
|
|
|
|
u32 core_id;
|
2023-08-14 08:18:37 +00:00
|
|
|
|
|
|
|
|
// Logical ID mappings
|
|
|
|
|
u32 logical_pkg_id;
|
|
|
|
|
u32 logical_die_id;
|
2023-08-14 08:18:38 +00:00
|
|
|
|
|
|
|
|
// Cache level topology IDs
|
|
|
|
|
u32 llc_id;
|
|
|
|
|
u32 l2c_id;
|
2023-08-14 08:18:29 +00:00
|
|
|
};
|
|
|
|
|
|
2008-01-30 12:31:33 +00:00
|
|
|
struct cpuinfo_x86 {
|
2008-02-21 03:24:40 +00:00
|
|
|
__u8 x86; /* CPU family */
|
|
|
|
|
__u8 x86_vendor; /* CPU vendor */
|
|
|
|
|
__u8 x86_model;
|
2018-01-01 01:52:10 +00:00
|
|
|
__u8 x86_stepping;
|
2017-02-12 21:12:08 +00:00
|
|
|
#ifdef CONFIG_X86_64
|
2008-02-21 03:24:40 +00:00
|
|
|
/* Number of 4K pages in DTLB/ITLB combined(in pages): */
|
2009-01-24 01:18:52 +00:00
|
|
|
int x86_tlbsize;
|
2019-12-21 04:45:04 +00:00
|
|
|
#endif
|
|
|
|
|
#ifdef CONFIG_X86_VMX_FEATURE_NAMES
|
|
|
|
|
__u32 vmx_capability[NVMXINTS];
|
2009-03-12 12:37:34 +00:00
|
|
|
#endif
|
2008-02-21 03:24:40 +00:00
|
|
|
__u8 x86_virt_bits;
|
|
|
|
|
__u8 x86_phys_bits;
|
|
|
|
|
/* CPUID returned core id bits: */
|
|
|
|
|
__u8 x86_coreid_bits;
|
|
|
|
|
/* Max extended CPUID function supported: */
|
|
|
|
|
__u32 extended_cpuid_level;
|
|
|
|
|
/* Maximum supported CPUID level, -1=no CPUID: */
|
|
|
|
|
int cpuid_level;
|
2019-09-16 22:39:58 +00:00
|
|
|
/*
|
|
|
|
|
* Align to size of unsigned long because the x86_capability array
|
|
|
|
|
* is passed to bitops which require the alignment. Use unnamed
|
|
|
|
|
* union to enforce the array is aligned to size of unsigned long.
|
|
|
|
|
*/
|
|
|
|
|
union {
|
|
|
|
|
__u32 x86_capability[NCAPINTS + NBUGINTS];
|
|
|
|
|
unsigned long x86_capability_alignment;
|
|
|
|
|
};
|
2008-02-21 03:24:40 +00:00
|
|
|
char x86_vendor_id[16];
|
|
|
|
|
char x86_model_id[64];
|
2023-08-14 08:18:29 +00:00
|
|
|
struct cpuinfo_topology topo;
|
2008-02-21 03:24:40 +00:00
|
|
|
/* in KB - valid for CPUS which support this call: */
|
2018-02-13 19:22:08 +00:00
|
|
|
unsigned int x86_cache_size;
|
2008-02-21 03:24:40 +00:00
|
|
|
int x86_cache_alignment; /* In bytes */
|
2020-05-05 22:36:17 +00:00
|
|
|
/* Cache QoS architectural values, valid only on the BSP: */
|
2015-01-23 18:45:43 +00:00
|
|
|
int x86_cache_max_rmid; /* max index */
|
|
|
|
|
int x86_cache_occ_scale; /* scale to bytes */
|
2020-05-05 22:36:17 +00:00
|
|
|
int x86_cache_mbm_width_offset;
|
2008-02-21 03:24:40 +00:00
|
|
|
int x86_power;
|
|
|
|
|
unsigned long loops_per_jiffy;
|
2022-01-31 23:01:09 +00:00
|
|
|
/* protected processor identification number */
|
|
|
|
|
u64 ppin;
|
2008-02-21 03:24:40 +00:00
|
|
|
/* cpuid returned max cores value: */
|
2019-05-13 17:58:45 +00:00
|
|
|
u16 x86_max_cores;
|
2008-02-21 03:24:40 +00:00
|
|
|
u16 x86_clflush_size;
|
|
|
|
|
/* number of cores as seen by the OS: */
|
|
|
|
|
u16 booted_cores;
|
|
|
|
|
/* Index into per_cpu list: */
|
|
|
|
|
u16 cpu_index;
|
2021-01-08 12:10:52 +00:00
|
|
|
/* Is SMT active on this core? */
|
|
|
|
|
bool smt_active;
|
2011-10-13 00:46:33 +00:00
|
|
|
u32 microcode;
|
2018-08-24 17:03:50 +00:00
|
|
|
/* Address space bits used by the cache internally */
|
|
|
|
|
u8 x86_cache_bits;
|
2017-11-14 12:42:56 +00:00
|
|
|
unsigned initialized : 1;
|
2016-10-28 08:22:25 +00:00
|
|
|
} __randomize_layout;
|
2008-01-30 12:31:33 +00:00
|
|
|
|
2008-02-21 03:24:40 +00:00
|
|
|
#define X86_VENDOR_INTEL 0
|
|
|
|
|
#define X86_VENDOR_CYRIX 1
|
|
|
|
|
#define X86_VENDOR_AMD 2
|
|
|
|
|
#define X86_VENDOR_UMC 3
|
|
|
|
|
#define X86_VENDOR_CENTAUR 5
|
|
|
|
|
#define X86_VENDOR_TRANSMETA 7
|
|
|
|
|
#define X86_VENDOR_NSC 8
|
2018-09-23 09:33:12 +00:00
|
|
|
#define X86_VENDOR_HYGON 9
|
2019-06-18 08:37:05 +00:00
|
|
|
#define X86_VENDOR_ZHAOXIN 10
|
2021-10-17 09:44:10 +00:00
|
|
|
#define X86_VENDOR_VORTEX 11
|
|
|
|
|
#define X86_VENDOR_NUM 12
|
2008-02-21 03:24:40 +00:00
|
|
|
|
|
|
|
|
#define X86_VENDOR_UNKNOWN 0xff
|
2008-01-30 12:31:33 +00:00
|
|
|
|
2008-01-30 12:31:39 +00:00
|
|
|
/*
|
|
|
|
|
* capabilities of CPUs
|
|
|
|
|
*/
|
2008-02-21 03:24:40 +00:00
|
|
|
extern struct cpuinfo_x86 boot_cpu_data;
|
|
|
|
|
extern struct cpuinfo_x86 new_cpu_data;
|
|
|
|
|
|
2017-12-04 14:07:32 +00:00
|
|
|
extern __u32 cpu_caps_cleared[NCAPINTS + NBUGINTS];
|
|
|
|
|
extern __u32 cpu_caps_set[NCAPINTS + NBUGINTS];
|
2008-01-30 12:31:33 +00:00
|
|
|
|
|
|
|
|
#ifdef CONFIG_SMP
|
2014-11-04 08:26:42 +00:00
|
|
|
DECLARE_PER_CPU_READ_MOSTLY(struct cpuinfo_x86, cpu_info);
|
2008-01-30 12:31:33 +00:00
|
|
|
#define cpu_data(cpu) per_cpu(cpu_info, cpu)
|
|
|
|
|
#else
|
2010-12-18 15:30:05 +00:00
|
|
|
#define cpu_info boot_cpu_data
|
2008-01-30 12:31:33 +00:00
|
|
|
#define cpu_data(cpu) boot_cpu_data
|
|
|
|
|
#endif
|
|
|
|
|
|
2008-07-21 17:10:37 +00:00
|
|
|
extern const struct seq_operations cpuinfo_op;
|
|
|
|
|
|
2008-02-21 03:24:40 +00:00
|
|
|
#define cache_line_size() (boot_cpu_data.x86_cache_alignment)
|
|
|
|
|
|
|
|
|
|
extern void cpu_detect(struct cpuinfo_x86 *c);
|
2008-01-30 12:31:39 +00:00
|
|
|
|
2018-08-20 09:58:35 +00:00
|
|
|
static inline unsigned long long l1tf_pfn_limit(void)
|
2018-06-13 22:48:26 +00:00
|
|
|
{
|
2018-08-24 17:03:50 +00:00
|
|
|
return BIT_ULL(boot_cpu_data.x86_cache_bits - 1 - PAGE_SHIFT);
|
2018-06-13 22:48:26 +00:00
|
|
|
}
|
|
|
|
|
|
2008-06-21 10:24:19 +00:00
|
|
|
extern void early_cpu_init(void);
|
2008-01-30 12:31:39 +00:00
|
|
|
extern void identify_secondary_cpu(struct cpuinfo_x86 *);
|
2008-01-30 12:31:33 +00:00
|
|
|
extern void print_cpu_info(struct cpuinfo_x86 *);
|
2012-02-12 17:53:57 +00:00
|
|
|
void print_cpu_msr(struct cpuinfo_x86 *);
|
2008-01-30 12:31:39 +00:00
|
|
|
|
2017-06-12 17:26:14 +00:00
|
|
|
/*
|
|
|
|
|
* Friendlier CR3 helpers.
|
|
|
|
|
*/
|
|
|
|
|
static inline unsigned long read_cr3_pa(void)
|
|
|
|
|
{
|
|
|
|
|
return __read_cr3() & CR3_ADDR_MASK;
|
|
|
|
|
}
|
|
|
|
|
|
2017-07-17 21:10:08 +00:00
|
|
|
static inline unsigned long native_read_cr3_pa(void)
|
|
|
|
|
{
|
|
|
|
|
return __native_read_cr3() & CR3_ADDR_MASK;
|
|
|
|
|
}
|
|
|
|
|
|
2008-01-30 12:31:27 +00:00
|
|
|
static inline void load_cr3(pgd_t *pgdir)
|
|
|
|
|
{
|
2017-07-17 21:10:07 +00:00
|
|
|
write_cr3(__sme_pa(pgdir));
|
2008-01-30 12:31:27 +00:00
|
|
|
}
|
2008-01-30 12:31:03 +00:00
|
|
|
|
2017-12-04 14:07:17 +00:00
|
|
|
/*
|
|
|
|
|
* Note that while the legacy 'TSS' name comes from 'Task State Segment',
|
|
|
|
|
* on modern x86 CPUs the TSS also holds information important to 64-bit mode,
|
|
|
|
|
* unrelated to the task-switch mechanism:
|
|
|
|
|
*/
|
2008-01-30 12:31:31 +00:00
|
|
|
#ifdef CONFIG_X86_32
|
|
|
|
|
/* This is the TSS defined by the hardware. */
|
|
|
|
|
struct x86_hw_tss {
|
2008-02-21 03:24:40 +00:00
|
|
|
unsigned short back_link, __blh;
|
|
|
|
|
unsigned long sp0;
|
|
|
|
|
unsigned short ss0, __ss0h;
|
2015-04-02 19:41:45 +00:00
|
|
|
unsigned long sp1;
|
2015-03-10 18:06:00 +00:00
|
|
|
|
|
|
|
|
/*
|
2015-04-02 19:41:45 +00:00
|
|
|
* We don't use ring 1, so ss1 is a convenient scratch space in
|
|
|
|
|
* the same cacheline as sp0. We use ss1 to cache the value in
|
|
|
|
|
* MSR_IA32_SYSENTER_CS. When we context switch
|
|
|
|
|
* MSR_IA32_SYSENTER_CS, we first check if the new value being
|
|
|
|
|
* written matches ss1, and, if it's not, then we wrmsr the new
|
|
|
|
|
* value and update ss1.
|
2015-03-10 18:06:00 +00:00
|
|
|
*
|
2015-04-02 19:41:45 +00:00
|
|
|
* The only reason we context switch MSR_IA32_SYSENTER_CS is
|
|
|
|
|
* that we set it to zero in vm86 tasks to avoid corrupting the
|
|
|
|
|
* stack if we were to go through the sysenter path from vm86
|
|
|
|
|
* mode.
|
2015-03-10 18:06:00 +00:00
|
|
|
*/
|
|
|
|
|
unsigned short ss1; /* MSR_IA32_SYSENTER_CS */
|
|
|
|
|
|
|
|
|
|
unsigned short __ss1h;
|
2008-02-21 03:24:40 +00:00
|
|
|
unsigned long sp2;
|
|
|
|
|
unsigned short ss2, __ss2h;
|
|
|
|
|
unsigned long __cr3;
|
|
|
|
|
unsigned long ip;
|
|
|
|
|
unsigned long flags;
|
|
|
|
|
unsigned long ax;
|
|
|
|
|
unsigned long cx;
|
|
|
|
|
unsigned long dx;
|
|
|
|
|
unsigned long bx;
|
|
|
|
|
unsigned long sp;
|
|
|
|
|
unsigned long bp;
|
|
|
|
|
unsigned long si;
|
|
|
|
|
unsigned long di;
|
|
|
|
|
unsigned short es, __esh;
|
|
|
|
|
unsigned short cs, __csh;
|
|
|
|
|
unsigned short ss, __ssh;
|
|
|
|
|
unsigned short ds, __dsh;
|
|
|
|
|
unsigned short fs, __fsh;
|
|
|
|
|
unsigned short gs, __gsh;
|
|
|
|
|
unsigned short ldt, __ldth;
|
|
|
|
|
unsigned short trace;
|
|
|
|
|
unsigned short io_bitmap_base;
|
|
|
|
|
|
2008-01-30 12:31:31 +00:00
|
|
|
} __attribute__((packed));
|
|
|
|
|
#else
|
|
|
|
|
struct x86_hw_tss {
|
2008-02-21 03:24:40 +00:00
|
|
|
u32 reserved1;
|
|
|
|
|
u64 sp0;
|
|
|
|
|
u64 sp1;
|
2017-12-04 14:07:21 +00:00
|
|
|
|
2018-09-03 22:59:43 +00:00
|
|
|
/*
|
|
|
|
|
* Since Linux does not use ring 2, the 'sp2' slot is unused by
|
|
|
|
|
* hardware. entry_SYSCALL_64 uses it as scratch space to stash
|
|
|
|
|
* the user RSP value.
|
|
|
|
|
*/
|
2008-02-21 03:24:40 +00:00
|
|
|
u64 sp2;
|
2018-09-03 22:59:43 +00:00
|
|
|
|
2008-02-21 03:24:40 +00:00
|
|
|
u64 reserved2;
|
|
|
|
|
u64 ist[7];
|
|
|
|
|
u32 reserved3;
|
|
|
|
|
u32 reserved4;
|
|
|
|
|
u16 reserved5;
|
|
|
|
|
u16 io_bitmap_base;
|
|
|
|
|
|
2017-02-20 16:56:13 +00:00
|
|
|
} __attribute__((packed));
|
2008-01-30 12:31:31 +00:00
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
/*
|
2008-02-21 03:24:40 +00:00
|
|
|
* IO-bitmap sizes:
|
2008-01-30 12:31:31 +00:00
|
|
|
*/
|
2008-02-21 03:24:40 +00:00
|
|
|
#define IO_BITMAP_BITS 65536
|
2019-11-12 17:45:29 +00:00
|
|
|
#define IO_BITMAP_BYTES (IO_BITMAP_BITS / BITS_PER_BYTE)
|
|
|
|
|
#define IO_BITMAP_LONGS (IO_BITMAP_BYTES / sizeof(long))
|
2019-11-11 22:03:20 +00:00
|
|
|
|
2019-11-11 22:03:28 +00:00
|
|
|
#define IO_BITMAP_OFFSET_VALID_MAP \
|
2019-11-12 17:45:29 +00:00
|
|
|
(offsetof(struct tss_struct, io_bitmap.bitmap) - \
|
2019-11-11 22:03:20 +00:00
|
|
|
offsetof(struct tss_struct, x86_tss))
|
|
|
|
|
|
2019-11-11 22:03:28 +00:00
|
|
|
#define IO_BITMAP_OFFSET_VALID_ALL \
|
|
|
|
|
(offsetof(struct tss_struct, io_bitmap.mapall) - \
|
|
|
|
|
offsetof(struct tss_struct, x86_tss))
|
|
|
|
|
|
2019-11-12 20:40:33 +00:00
|
|
|
#ifdef CONFIG_X86_IOPL_IOPERM
|
2019-11-11 22:03:20 +00:00
|
|
|
/*
|
2019-11-11 22:03:28 +00:00
|
|
|
* sizeof(unsigned long) coming from an extra "long" at the end of the
|
|
|
|
|
* iobitmap. The limit is inclusive, i.e. the last valid byte.
|
2019-11-11 22:03:20 +00:00
|
|
|
*/
|
2019-11-12 20:40:33 +00:00
|
|
|
# define __KERNEL_TSS_LIMIT \
|
2019-11-11 22:03:28 +00:00
|
|
|
(IO_BITMAP_OFFSET_VALID_ALL + IO_BITMAP_BYTES + \
|
|
|
|
|
sizeof(unsigned long) - 1)
|
2019-11-12 20:40:33 +00:00
|
|
|
#else
|
|
|
|
|
# define __KERNEL_TSS_LIMIT \
|
|
|
|
|
(offsetof(struct tss_struct, x86_tss) + sizeof(struct x86_hw_tss) - 1)
|
|
|
|
|
#endif
|
2019-11-11 22:03:20 +00:00
|
|
|
|
|
|
|
|
/* Base offset outside of TSS_LIMIT so unpriviledged IO causes #GP */
|
|
|
|
|
#define IO_BITMAP_OFFSET_INVALID (__KERNEL_TSS_LIMIT + 1)
|
2008-01-30 12:31:31 +00:00
|
|
|
|
2017-12-05 01:25:07 +00:00
|
|
|
struct entry_stack {
|
2020-06-17 16:25:57 +00:00
|
|
|
char stack[PAGE_SIZE];
|
2017-12-04 14:07:28 +00:00
|
|
|
};
|
|
|
|
|
|
2017-12-05 01:25:07 +00:00
|
|
|
struct entry_stack_page {
|
|
|
|
|
struct entry_stack stack;
|
2017-12-04 14:07:29 +00:00
|
|
|
} __aligned(PAGE_SIZE);
|
2017-12-04 14:07:19 +00:00
|
|
|
|
2019-11-12 17:45:29 +00:00
|
|
|
/*
|
|
|
|
|
* All IO bitmap related data stored in the TSS:
|
|
|
|
|
*/
|
|
|
|
|
struct x86_io_bitmap {
|
2019-11-11 22:03:22 +00:00
|
|
|
/* The sequence number of the last active bitmap. */
|
|
|
|
|
u64 prev_sequence;
|
|
|
|
|
|
2008-02-21 03:24:40 +00:00
|
|
|
/*
|
2019-11-11 22:03:20 +00:00
|
|
|
* Store the dirty size of the last io bitmap offender. The next
|
|
|
|
|
* one will have to do the cleanup as the switch out to a non io
|
|
|
|
|
* bitmap user will just set x86_tss.io_bitmap_base to a value
|
|
|
|
|
* outside of the TSS limit. So for sane tasks there is no need to
|
|
|
|
|
* actually touch the io_bitmap at all.
|
2008-02-21 03:24:40 +00:00
|
|
|
*/
|
2019-11-12 17:45:29 +00:00
|
|
|
unsigned int prev_max;
|
2008-01-30 12:31:31 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* The extra 1 is there because the CPU will access an
|
|
|
|
|
* additional byte beyond the end of the IO permission
|
|
|
|
|
* bitmap. The extra byte must be all 1 bits, and must
|
|
|
|
|
* be within the limit.
|
|
|
|
|
*/
|
2019-11-12 17:45:29 +00:00
|
|
|
unsigned long bitmap[IO_BITMAP_LONGS + 1];
|
2019-11-11 22:03:28 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Special I/O bitmap to emulate IOPL(3). All bytes zero,
|
|
|
|
|
* except the additional byte at the end.
|
|
|
|
|
*/
|
|
|
|
|
unsigned long mapall[IO_BITMAP_LONGS + 1];
|
2019-11-12 17:45:29 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
|
|
struct tss_struct {
|
|
|
|
|
/*
|
|
|
|
|
* The fixed hardware portion. This must not cross a page boundary
|
|
|
|
|
* at risk of violating the SDM's advice and potentially triggering
|
|
|
|
|
* errata.
|
|
|
|
|
*/
|
|
|
|
|
struct x86_hw_tss x86_tss;
|
|
|
|
|
|
|
|
|
|
struct x86_io_bitmap io_bitmap;
|
2017-12-04 14:07:19 +00:00
|
|
|
} __aligned(PAGE_SIZE);
|
2008-02-21 03:24:40 +00:00
|
|
|
|
2017-12-04 14:07:29 +00:00
|
|
|
DECLARE_PER_CPU_PAGE_ALIGNED(struct tss_struct, cpu_tss_rw);
|
2008-01-30 12:31:31 +00:00
|
|
|
|
2019-04-14 16:00:06 +00:00
|
|
|
/* Per CPU interrupt stacks */
|
|
|
|
|
struct irq_stack {
|
|
|
|
|
char stack[IRQ_STACK_SIZE];
|
|
|
|
|
} __aligned(IRQ_STACK_SIZE);
|
|
|
|
|
|
2008-03-03 17:12:56 +00:00
|
|
|
#ifdef CONFIG_X86_64
|
2019-04-14 16:00:06 +00:00
|
|
|
struct fixed_percpu_data {
|
2009-01-19 03:21:28 +00:00
|
|
|
/*
|
|
|
|
|
* GCC hardcodes the stack canary as %gs:40. Since the
|
|
|
|
|
* irq_stack is the object at %gs:0, we reserve the bottom
|
|
|
|
|
* 48 bytes of the irq stack for the canary.
|
x86/stackprotector/32: Make the canary into a regular percpu variable
On 32-bit kernels, the stackprotector canary is quite nasty -- it is
stored at %gs:(20), which is nasty because 32-bit kernels use %fs for
percpu storage. It's even nastier because it means that whether %gs
contains userspace state or kernel state while running kernel code
depends on whether stackprotector is enabled (this is
CONFIG_X86_32_LAZY_GS), and this setting radically changes the way
that segment selectors work. Supporting both variants is a
maintenance and testing mess.
Merely rearranging so that percpu and the stack canary
share the same segment would be messy as the 32-bit percpu address
layout isn't currently compatible with putting a variable at a fixed
offset.
Fortunately, GCC 8.1 added options that allow the stack canary to be
accessed as %fs:__stack_chk_guard, effectively turning it into an ordinary
percpu variable. This lets us get rid of all of the code to manage the
stack canary GDT descriptor and the CONFIG_X86_32_LAZY_GS mess.
(That name is special. We could use any symbol we want for the
%fs-relative mode, but for CONFIG_SMP=n, gcc refuses to let us use any
name other than __stack_chk_guard.)
Forcibly disable stackprotector on older compilers that don't support
the new options and turn the stack canary into a percpu variable. The
"lazy GS" approach is now used for all 32-bit configurations.
Also makes load_gs_index() work on 32-bit kernels. On 64-bit kernels,
it loads the GS selector and updates the user GSBASE accordingly. (This
is unchanged.) On 32-bit kernels, it loads the GS selector and updates
GSBASE, which is now always the user base. This means that the overall
effect is the same on 32-bit and 64-bit, which avoids some ifdeffery.
[ bp: Massage commit message. ]
Signed-off-by: Andy Lutomirski <luto@kernel.org>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/c0ff7dba14041c7e5d1cae5d4df052f03759bef3.1613243844.git.luto@kernel.org
2021-02-13 19:19:44 +00:00
|
|
|
*
|
|
|
|
|
* Once we are willing to require -mstack-protector-guard-symbol=
|
|
|
|
|
* support for x86_64 stackprotector, we can get rid of this.
|
2009-01-19 03:21:28 +00:00
|
|
|
*/
|
2019-04-14 16:00:06 +00:00
|
|
|
char gs_base[40];
|
|
|
|
|
unsigned long stack_canary;
|
2009-01-19 03:21:28 +00:00
|
|
|
};
|
|
|
|
|
|
2019-04-14 16:00:06 +00:00
|
|
|
DECLARE_PER_CPU_FIRST(struct fixed_percpu_data, fixed_percpu_data) __visible;
|
|
|
|
|
DECLARE_INIT_PER_CPU(fixed_percpu_data);
|
2009-02-08 14:58:39 +00:00
|
|
|
|
2018-03-13 17:48:05 +00:00
|
|
|
static inline unsigned long cpu_kernelmode_gs_base(int cpu)
|
|
|
|
|
{
|
2019-04-14 16:00:06 +00:00
|
|
|
return (unsigned long)per_cpu(fixed_percpu_data.gs_base, cpu);
|
2018-03-13 17:48:05 +00:00
|
|
|
}
|
|
|
|
|
|
2023-06-23 11:14:05 +00:00
|
|
|
extern asmlinkage void entry_SYSCALL32_ignore(void);
|
2018-03-13 17:48:04 +00:00
|
|
|
|
|
|
|
|
/* Save actual FS/GS selectors and bases to current->thread */
|
2020-05-28 20:13:52 +00:00
|
|
|
void current_save_fsgs(void);
|
2009-02-09 13:17:40 +00:00
|
|
|
#else /* X86_64 */
|
Kbuild: rename CC_STACKPROTECTOR[_STRONG] config variables
The changes to automatically test for working stack protector compiler
support in the Kconfig files removed the special STACKPROTECTOR_AUTO
option that picked the strongest stack protector that the compiler
supported.
That was all a nice cleanup - it makes no sense to have the AUTO case
now that the Kconfig phase can just determine the compiler support
directly.
HOWEVER.
It also meant that doing "make oldconfig" would now _disable_ the strong
stackprotector if you had AUTO enabled, because in a legacy config file,
the sane stack protector configuration would look like
CONFIG_HAVE_CC_STACKPROTECTOR=y
# CONFIG_CC_STACKPROTECTOR_NONE is not set
# CONFIG_CC_STACKPROTECTOR_REGULAR is not set
# CONFIG_CC_STACKPROTECTOR_STRONG is not set
CONFIG_CC_STACKPROTECTOR_AUTO=y
and when you ran this through "make oldconfig" with the Kbuild changes,
it would ask you about the regular CONFIG_CC_STACKPROTECTOR (that had
been renamed from CONFIG_CC_STACKPROTECTOR_REGULAR to just
CONFIG_CC_STACKPROTECTOR), but it would think that the STRONG version
used to be disabled (because it was really enabled by AUTO), and would
disable it in the new config, resulting in:
CONFIG_HAVE_CC_STACKPROTECTOR=y
CONFIG_CC_HAS_STACKPROTECTOR_NONE=y
CONFIG_CC_STACKPROTECTOR=y
# CONFIG_CC_STACKPROTECTOR_STRONG is not set
CONFIG_CC_HAS_SANE_STACKPROTECTOR=y
That's dangerously subtle - people could suddenly find themselves with
the weaker stack protector setup without even realizing.
The solution here is to just rename not just the old RECULAR stack
protector option, but also the strong one. This does that by just
removing the CC_ prefix entirely for the user choices, because it really
is not about the compiler support (the compiler support now instead
automatially impacts _visibility_ of the options to users).
This results in "make oldconfig" actually asking the user for their
choice, so that we don't have any silent subtle security model changes.
The end result would generally look like this:
CONFIG_HAVE_CC_STACKPROTECTOR=y
CONFIG_CC_HAS_STACKPROTECTOR_NONE=y
CONFIG_STACKPROTECTOR=y
CONFIG_STACKPROTECTOR_STRONG=y
CONFIG_CC_HAS_SANE_STACKPROTECTOR=y
where the "CC_" versions really are about internal compiler
infrastructure, not the user selections.
Acked-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2018-06-14 03:21:18 +00:00
|
|
|
#ifdef CONFIG_STACKPROTECTOR
|
x86/stackprotector/32: Make the canary into a regular percpu variable
On 32-bit kernels, the stackprotector canary is quite nasty -- it is
stored at %gs:(20), which is nasty because 32-bit kernels use %fs for
percpu storage. It's even nastier because it means that whether %gs
contains userspace state or kernel state while running kernel code
depends on whether stackprotector is enabled (this is
CONFIG_X86_32_LAZY_GS), and this setting radically changes the way
that segment selectors work. Supporting both variants is a
maintenance and testing mess.
Merely rearranging so that percpu and the stack canary
share the same segment would be messy as the 32-bit percpu address
layout isn't currently compatible with putting a variable at a fixed
offset.
Fortunately, GCC 8.1 added options that allow the stack canary to be
accessed as %fs:__stack_chk_guard, effectively turning it into an ordinary
percpu variable. This lets us get rid of all of the code to manage the
stack canary GDT descriptor and the CONFIG_X86_32_LAZY_GS mess.
(That name is special. We could use any symbol we want for the
%fs-relative mode, but for CONFIG_SMP=n, gcc refuses to let us use any
name other than __stack_chk_guard.)
Forcibly disable stackprotector on older compilers that don't support
the new options and turn the stack canary into a percpu variable. The
"lazy GS" approach is now used for all 32-bit configurations.
Also makes load_gs_index() work on 32-bit kernels. On 64-bit kernels,
it loads the GS selector and updates the user GSBASE accordingly. (This
is unchanged.) On 32-bit kernels, it loads the GS selector and updates
GSBASE, which is now always the user base. This means that the overall
effect is the same on 32-bit and 64-bit, which avoids some ifdeffery.
[ bp: Massage commit message. ]
Signed-off-by: Andy Lutomirski <luto@kernel.org>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/c0ff7dba14041c7e5d1cae5d4df052f03759bef3.1613243844.git.luto@kernel.org
2021-02-13 19:19:44 +00:00
|
|
|
DECLARE_PER_CPU(unsigned long, __stack_chk_guard);
|
2007-10-11 09:20:03 +00:00
|
|
|
#endif
|
2021-02-09 23:40:44 +00:00
|
|
|
#endif /* !X86_64 */
|
2008-01-30 12:31:03 +00:00
|
|
|
|
2009-09-09 17:22:48 +00:00
|
|
|
struct perf_event;
|
|
|
|
|
|
2008-01-30 12:31:31 +00:00
|
|
|
struct thread_struct {
|
2008-02-21 03:24:40 +00:00
|
|
|
/* Cached TLS descriptors: */
|
|
|
|
|
struct desc_struct tls_array[GDT_ENTRY_TLS_ENTRIES];
|
2017-11-02 07:59:16 +00:00
|
|
|
#ifdef CONFIG_X86_32
|
2008-02-21 03:24:40 +00:00
|
|
|
unsigned long sp0;
|
2017-11-02 07:59:16 +00:00
|
|
|
#endif
|
2008-02-21 03:24:40 +00:00
|
|
|
unsigned long sp;
|
2008-01-30 12:31:31 +00:00
|
|
|
#ifdef CONFIG_X86_32
|
2008-02-21 03:24:40 +00:00
|
|
|
unsigned long sysenter_cs;
|
2008-01-30 12:31:31 +00:00
|
|
|
#else
|
2008-02-21 03:24:40 +00:00
|
|
|
unsigned short es;
|
|
|
|
|
unsigned short ds;
|
|
|
|
|
unsigned short fsindex;
|
|
|
|
|
unsigned short gsindex;
|
2008-01-30 12:31:31 +00:00
|
|
|
#endif
|
2016-09-13 21:29:21 +00:00
|
|
|
|
2009-05-03 23:29:52 +00:00
|
|
|
#ifdef CONFIG_X86_64
|
2016-04-26 19:23:29 +00:00
|
|
|
unsigned long fsbase;
|
|
|
|
|
unsigned long gsbase;
|
|
|
|
|
#else
|
|
|
|
|
/*
|
|
|
|
|
* XXX: this could presumably be unsigned short. Alternatively,
|
|
|
|
|
* 32-bit kernels could be taught to use fsindex instead.
|
|
|
|
|
*/
|
|
|
|
|
unsigned long fs;
|
|
|
|
|
unsigned long gs;
|
2009-05-03 23:29:52 +00:00
|
|
|
#endif
|
2015-04-23 10:49:20 +00:00
|
|
|
|
2009-09-09 17:22:48 +00:00
|
|
|
/* Save middle states of ptrace breakpoints */
|
|
|
|
|
struct perf_event *ptrace_bps[HBP_NUM];
|
|
|
|
|
/* Debug status used for traps, single steps, etc... */
|
2020-09-02 13:26:02 +00:00
|
|
|
unsigned long virtual_dr6;
|
2010-02-18 17:24:18 +00:00
|
|
|
/* Keep track of the exact dr7 value set by the user */
|
|
|
|
|
unsigned long ptrace_dr7;
|
2008-02-21 03:24:40 +00:00
|
|
|
/* Fault info: */
|
|
|
|
|
unsigned long cr2;
|
2012-03-12 09:25:55 +00:00
|
|
|
unsigned long trap_nr;
|
2008-02-21 03:24:40 +00:00
|
|
|
unsigned long error_code;
|
2015-07-29 05:41:16 +00:00
|
|
|
#ifdef CONFIG_VM86
|
2008-02-21 03:24:40 +00:00
|
|
|
/* Virtual 86 mode info */
|
2015-07-29 05:41:16 +00:00
|
|
|
struct vm86 *vm86;
|
2008-01-30 12:31:31 +00:00
|
|
|
#endif
|
2008-02-21 03:24:40 +00:00
|
|
|
/* IO permissions: */
|
2019-11-11 22:03:21 +00:00
|
|
|
struct io_bitmap *io_bitmap;
|
2019-11-11 22:03:28 +00:00
|
|
|
|
|
|
|
|
/*
|
2021-03-18 14:28:01 +00:00
|
|
|
* IOPL. Privilege level dependent I/O permission which is
|
2019-11-11 22:03:29 +00:00
|
|
|
* emulated via the I/O bitmap to prevent user space from disabling
|
|
|
|
|
* interrupts.
|
2019-11-11 22:03:28 +00:00
|
|
|
*/
|
|
|
|
|
unsigned long iopl_emul;
|
2015-07-17 10:28:11 +00:00
|
|
|
|
2021-09-17 09:20:04 +00:00
|
|
|
unsigned int iopl_warn:1;
|
2016-07-15 08:21:11 +00:00
|
|
|
unsigned int sig_on_uaccess_err:1;
|
2016-07-14 20:22:56 +00:00
|
|
|
|
2021-06-23 12:02:18 +00:00
|
|
|
/*
|
|
|
|
|
* Protection Keys Register for Userspace. Loaded immediately on
|
|
|
|
|
* context switch. Store it in thread_struct to avoid a lookup in
|
|
|
|
|
* the tasks's FPU xstate buffer. This value is only valid when a
|
|
|
|
|
* task is scheduled out. For 'current' the authoritative source of
|
|
|
|
|
* PKRU is the hardware itself.
|
|
|
|
|
*/
|
|
|
|
|
u32 pkru;
|
|
|
|
|
|
2023-06-13 00:10:52 +00:00
|
|
|
#ifdef CONFIG_X86_USER_SHADOW_STACK
|
|
|
|
|
unsigned long features;
|
|
|
|
|
unsigned long features_locked;
|
2023-06-13 00:10:54 +00:00
|
|
|
|
|
|
|
|
struct thread_shstk shstk;
|
2023-06-13 00:10:52 +00:00
|
|
|
#endif
|
|
|
|
|
|
2015-07-17 10:28:11 +00:00
|
|
|
/* Floating point and extended processor state */
|
|
|
|
|
struct fpu fpu;
|
|
|
|
|
/*
|
|
|
|
|
* WARNING: 'fpu' is dynamically-sized. It *MUST* be at
|
|
|
|
|
* the end.
|
|
|
|
|
*/
|
2008-01-30 12:31:31 +00:00
|
|
|
};
|
|
|
|
|
|
2021-10-13 14:55:45 +00:00
|
|
|
extern void fpu_thread_struct_whitelist(unsigned long *offset, unsigned long *size);
|
|
|
|
|
|
2017-08-16 20:26:03 +00:00
|
|
|
static inline void arch_thread_struct_whitelist(unsigned long *offset,
|
|
|
|
|
unsigned long *size)
|
|
|
|
|
{
|
2021-10-13 14:55:45 +00:00
|
|
|
fpu_thread_struct_whitelist(offset, size);
|
2017-08-16 20:26:03 +00:00
|
|
|
}
|
|
|
|
|
|
2008-02-21 03:24:40 +00:00
|
|
|
static inline void
|
2017-11-02 07:59:10 +00:00
|
|
|
native_load_sp0(unsigned long sp0)
|
2008-01-30 12:31:31 +00:00
|
|
|
{
|
2017-12-04 14:07:29 +00:00
|
|
|
this_cpu_write(cpu_tss_rw.x86_tss.sp0, sp0);
|
2008-01-30 12:31:31 +00:00
|
|
|
}
|
2008-01-30 12:31:27 +00:00
|
|
|
|
2020-05-28 20:13:50 +00:00
|
|
|
static __always_inline void native_swapgs(void)
|
2008-01-30 12:32:08 +00:00
|
|
|
{
|
|
|
|
|
#ifdef CONFIG_X86_64
|
|
|
|
|
asm volatile("swapgs" ::: "memory");
|
|
|
|
|
#endif
|
|
|
|
|
}
|
|
|
|
|
|
2022-05-07 11:37:45 +00:00
|
|
|
static __always_inline unsigned long current_top_of_stack(void)
|
2015-03-06 03:19:02 +00:00
|
|
|
{
|
2017-12-04 14:07:21 +00:00
|
|
|
/*
|
|
|
|
|
* We can't read directly from tss.sp0: sp0 on x86_32 is special in
|
|
|
|
|
* and around vm86 mode and sp0 on x86_64 is special because of the
|
|
|
|
|
* entry trampoline.
|
|
|
|
|
*/
|
2022-09-15 11:11:04 +00:00
|
|
|
return this_cpu_read_stable(pcpu_hot.top_of_stack);
|
2015-03-06 03:19:02 +00:00
|
|
|
}
|
|
|
|
|
|
2022-05-07 11:37:45 +00:00
|
|
|
static __always_inline bool on_thread_stack(void)
|
2017-11-02 07:59:17 +00:00
|
|
|
{
|
|
|
|
|
return (unsigned long)(current_top_of_stack() -
|
|
|
|
|
current_stack_pointer) < THREAD_SIZE;
|
|
|
|
|
}
|
|
|
|
|
|
2018-08-28 07:40:23 +00:00
|
|
|
#ifdef CONFIG_PARAVIRT_XXL
|
2008-01-30 12:31:31 +00:00
|
|
|
#include <asm/paravirt.h>
|
|
|
|
|
#else
|
2008-01-30 12:31:27 +00:00
|
|
|
|
2017-11-02 07:59:10 +00:00
|
|
|
static inline void load_sp0(unsigned long sp0)
|
2008-01-30 12:31:31 +00:00
|
|
|
{
|
2017-11-02 07:59:10 +00:00
|
|
|
native_load_sp0(sp0);
|
2008-01-30 12:31:31 +00:00
|
|
|
}
|
|
|
|
|
|
2018-08-28 07:40:23 +00:00
|
|
|
#endif /* CONFIG_PARAVIRT_XXL */
|
2008-01-30 12:31:27 +00:00
|
|
|
|
2021-09-29 22:02:14 +00:00
|
|
|
unsigned long __get_wchan(struct task_struct *p);
|
2008-01-30 12:31:03 +00:00
|
|
|
|
2008-01-30 12:31:27 +00:00
|
|
|
extern void select_idle_routine(const struct cpuinfo_x86 *c);
|
2016-12-09 18:29:11 +00:00
|
|
|
extern void amd_e400_c1e_apic_setup(void);
|
2008-01-30 12:31:27 +00:00
|
|
|
|
2008-02-21 03:24:40 +00:00
|
|
|
extern unsigned long boot_option_idle_override;
|
2008-01-30 12:31:27 +00:00
|
|
|
|
2010-11-03 16:06:14 +00:00
|
|
|
enum idle_boot_override {IDLE_NO_OVERRIDE=0, IDLE_HALT, IDLE_NOMWAIT,
|
2013-02-10 06:38:39 +00:00
|
|
|
IDLE_POLL};
|
2010-11-03 16:06:14 +00:00
|
|
|
|
2008-01-30 12:31:39 +00:00
|
|
|
extern void enable_sep_cpu(void);
|
|
|
|
|
|
2010-05-21 02:04:29 +00:00
|
|
|
|
2008-01-30 12:31:39 +00:00
|
|
|
/* Defined in head.S */
|
2008-02-21 03:24:40 +00:00
|
|
|
extern struct desc_ptr early_gdt_descr;
|
2008-01-30 12:31:39 +00:00
|
|
|
|
2022-09-15 11:10:42 +00:00
|
|
|
extern void switch_gdt_and_percpu_base(int);
|
2017-03-14 17:05:08 +00:00
|
|
|
extern void load_direct_gdt(int);
|
2017-03-14 17:05:07 +00:00
|
|
|
extern void load_fixmap_gdt(int);
|
2008-01-30 12:31:39 +00:00
|
|
|
extern void cpu_init(void);
|
2020-09-07 13:16:08 +00:00
|
|
|
extern void cpu_init_exception_handling(void);
|
2019-07-10 19:42:46 +00:00
|
|
|
extern void cr4_init(void);
|
2008-01-30 12:31:39 +00:00
|
|
|
|
2008-12-11 12:49:59 +00:00
|
|
|
static inline unsigned long get_debugctlmsr(void)
|
|
|
|
|
{
|
2010-03-25 13:51:51 +00:00
|
|
|
unsigned long debugctlmsr = 0;
|
2008-12-11 12:49:59 +00:00
|
|
|
|
|
|
|
|
#ifndef CONFIG_X86_DEBUGCTLMSR
|
|
|
|
|
if (boot_cpu_data.x86 < 6)
|
|
|
|
|
return 0;
|
|
|
|
|
#endif
|
|
|
|
|
rdmsrl(MSR_IA32_DEBUGCTLMSR, debugctlmsr);
|
|
|
|
|
|
2010-03-25 13:51:51 +00:00
|
|
|
return debugctlmsr;
|
2008-12-11 12:49:59 +00:00
|
|
|
}
|
|
|
|
|
|
2008-03-10 13:11:17 +00:00
|
|
|
static inline void update_debugctlmsr(unsigned long debugctlmsr)
|
|
|
|
|
{
|
|
|
|
|
#ifndef CONFIG_X86_DEBUGCTLMSR
|
|
|
|
|
if (boot_cpu_data.x86 < 6)
|
|
|
|
|
return;
|
|
|
|
|
#endif
|
|
|
|
|
wrmsrl(MSR_IA32_DEBUGCTLMSR, debugctlmsr);
|
|
|
|
|
}
|
|
|
|
|
|
2012-09-03 13:24:17 +00:00
|
|
|
extern void set_task_blockstep(struct task_struct *task, bool on);
|
|
|
|
|
|
2008-02-21 03:24:40 +00:00
|
|
|
/* Boot loader type from the setup header: */
|
|
|
|
|
extern int bootloader_type;
|
2009-05-07 23:54:11 +00:00
|
|
|
extern int bootloader_version;
|
2008-01-30 12:31:39 +00:00
|
|
|
|
2008-02-21 03:24:40 +00:00
|
|
|
extern char ignore_fpu_irq;
|
2008-01-30 12:31:27 +00:00
|
|
|
|
|
|
|
|
#define HAVE_ARCH_PICK_MMAP_LAYOUT 1
|
|
|
|
|
#define ARCH_HAS_PREFETCHW
|
|
|
|
|
|
2008-01-30 12:31:40 +00:00
|
|
|
#ifdef CONFIG_X86_32
|
x86/asm: Cleanup prefetch primitives
This is based on a patch originally by hpa.
With the current improvements to the alternatives, we can simply use %P1
as a mem8 operand constraint and rely on the toolchain to generate the
proper instruction sizes. For example, on 32-bit, where we use an empty
old instruction we get:
apply_alternatives: feat: 6*32+8, old: (c104648b, len: 4), repl: (c195566c, len: 4)
c104648b: alt_insn: 90 90 90 90
c195566c: rpl_insn: 0f 0d 4b 5c
...
apply_alternatives: feat: 6*32+8, old: (c18e09b4, len: 3), repl: (c1955948, len: 3)
c18e09b4: alt_insn: 90 90 90
c1955948: rpl_insn: 0f 0d 08
...
apply_alternatives: feat: 6*32+8, old: (c1190cf9, len: 7), repl: (c1955a79, len: 7)
c1190cf9: alt_insn: 90 90 90 90 90 90 90
c1955a79: rpl_insn: 0f 0d 0d a0 d4 85 c1
all with the proper padding done depending on the size of the
replacement instruction the compiler generates.
Signed-off-by: Borislav Petkov <bp@suse.de>
Cc: H. Peter Anvin <hpa@linux.intel.com>
2015-01-18 16:48:18 +00:00
|
|
|
# define BASE_PREFETCH ""
|
2008-02-21 03:24:40 +00:00
|
|
|
# define ARCH_HAS_PREFETCH
|
2008-01-30 12:31:40 +00:00
|
|
|
#else
|
x86/asm: Cleanup prefetch primitives
This is based on a patch originally by hpa.
With the current improvements to the alternatives, we can simply use %P1
as a mem8 operand constraint and rely on the toolchain to generate the
proper instruction sizes. For example, on 32-bit, where we use an empty
old instruction we get:
apply_alternatives: feat: 6*32+8, old: (c104648b, len: 4), repl: (c195566c, len: 4)
c104648b: alt_insn: 90 90 90 90
c195566c: rpl_insn: 0f 0d 4b 5c
...
apply_alternatives: feat: 6*32+8, old: (c18e09b4, len: 3), repl: (c1955948, len: 3)
c18e09b4: alt_insn: 90 90 90
c1955948: rpl_insn: 0f 0d 08
...
apply_alternatives: feat: 6*32+8, old: (c1190cf9, len: 7), repl: (c1955a79, len: 7)
c1190cf9: alt_insn: 90 90 90 90 90 90 90
c1955a79: rpl_insn: 0f 0d 0d a0 d4 85 c1
all with the proper padding done depending on the size of the
replacement instruction the compiler generates.
Signed-off-by: Borislav Petkov <bp@suse.de>
Cc: H. Peter Anvin <hpa@linux.intel.com>
2015-01-18 16:48:18 +00:00
|
|
|
# define BASE_PREFETCH "prefetcht0 %P1"
|
2008-01-30 12:31:40 +00:00
|
|
|
#endif
|
|
|
|
|
|
2008-02-21 03:24:40 +00:00
|
|
|
/*
|
|
|
|
|
* Prefetch instructions for Pentium III (+) and AMD Athlon (+)
|
|
|
|
|
*
|
|
|
|
|
* It's not worth to care about 3dnow prefetches for the K6
|
|
|
|
|
* because they are microcoded there and very slow.
|
|
|
|
|
*/
|
2008-01-30 12:31:40 +00:00
|
|
|
static inline void prefetch(const void *x)
|
|
|
|
|
{
|
x86/asm: Cleanup prefetch primitives
This is based on a patch originally by hpa.
With the current improvements to the alternatives, we can simply use %P1
as a mem8 operand constraint and rely on the toolchain to generate the
proper instruction sizes. For example, on 32-bit, where we use an empty
old instruction we get:
apply_alternatives: feat: 6*32+8, old: (c104648b, len: 4), repl: (c195566c, len: 4)
c104648b: alt_insn: 90 90 90 90
c195566c: rpl_insn: 0f 0d 4b 5c
...
apply_alternatives: feat: 6*32+8, old: (c18e09b4, len: 3), repl: (c1955948, len: 3)
c18e09b4: alt_insn: 90 90 90
c1955948: rpl_insn: 0f 0d 08
...
apply_alternatives: feat: 6*32+8, old: (c1190cf9, len: 7), repl: (c1955a79, len: 7)
c1190cf9: alt_insn: 90 90 90 90 90 90 90
c1955a79: rpl_insn: 0f 0d 0d a0 d4 85 c1
all with the proper padding done depending on the size of the
replacement instruction the compiler generates.
Signed-off-by: Borislav Petkov <bp@suse.de>
Cc: H. Peter Anvin <hpa@linux.intel.com>
2015-01-18 16:48:18 +00:00
|
|
|
alternative_input(BASE_PREFETCH, "prefetchnta %P1",
|
2008-01-30 12:31:40 +00:00
|
|
|
X86_FEATURE_XMM,
|
x86/asm: Cleanup prefetch primitives
This is based on a patch originally by hpa.
With the current improvements to the alternatives, we can simply use %P1
as a mem8 operand constraint and rely on the toolchain to generate the
proper instruction sizes. For example, on 32-bit, where we use an empty
old instruction we get:
apply_alternatives: feat: 6*32+8, old: (c104648b, len: 4), repl: (c195566c, len: 4)
c104648b: alt_insn: 90 90 90 90
c195566c: rpl_insn: 0f 0d 4b 5c
...
apply_alternatives: feat: 6*32+8, old: (c18e09b4, len: 3), repl: (c1955948, len: 3)
c18e09b4: alt_insn: 90 90 90
c1955948: rpl_insn: 0f 0d 08
...
apply_alternatives: feat: 6*32+8, old: (c1190cf9, len: 7), repl: (c1955a79, len: 7)
c1190cf9: alt_insn: 90 90 90 90 90 90 90
c1955a79: rpl_insn: 0f 0d 0d a0 d4 85 c1
all with the proper padding done depending on the size of the
replacement instruction the compiler generates.
Signed-off-by: Borislav Petkov <bp@suse.de>
Cc: H. Peter Anvin <hpa@linux.intel.com>
2015-01-18 16:48:18 +00:00
|
|
|
"m" (*(const char *)x));
|
2008-01-30 12:31:40 +00:00
|
|
|
}
|
|
|
|
|
|
2008-02-21 03:24:40 +00:00
|
|
|
/*
|
|
|
|
|
* 3dnow prefetch to get an exclusive cache line.
|
|
|
|
|
* Useful for spinlocks to avoid one state transition in the
|
|
|
|
|
* cache coherency protocol:
|
|
|
|
|
*/
|
2020-06-03 11:40:22 +00:00
|
|
|
static __always_inline void prefetchw(const void *x)
|
2008-01-30 12:31:40 +00:00
|
|
|
{
|
x86/asm: Cleanup prefetch primitives
This is based on a patch originally by hpa.
With the current improvements to the alternatives, we can simply use %P1
as a mem8 operand constraint and rely on the toolchain to generate the
proper instruction sizes. For example, on 32-bit, where we use an empty
old instruction we get:
apply_alternatives: feat: 6*32+8, old: (c104648b, len: 4), repl: (c195566c, len: 4)
c104648b: alt_insn: 90 90 90 90
c195566c: rpl_insn: 0f 0d 4b 5c
...
apply_alternatives: feat: 6*32+8, old: (c18e09b4, len: 3), repl: (c1955948, len: 3)
c18e09b4: alt_insn: 90 90 90
c1955948: rpl_insn: 0f 0d 08
...
apply_alternatives: feat: 6*32+8, old: (c1190cf9, len: 7), repl: (c1955a79, len: 7)
c1190cf9: alt_insn: 90 90 90 90 90 90 90
c1955a79: rpl_insn: 0f 0d 0d a0 d4 85 c1
all with the proper padding done depending on the size of the
replacement instruction the compiler generates.
Signed-off-by: Borislav Petkov <bp@suse.de>
Cc: H. Peter Anvin <hpa@linux.intel.com>
2015-01-18 16:48:18 +00:00
|
|
|
alternative_input(BASE_PREFETCH, "prefetchw %P1",
|
|
|
|
|
X86_FEATURE_3DNOWPREFETCH,
|
|
|
|
|
"m" (*(const char *)x));
|
2008-01-30 12:31:40 +00:00
|
|
|
}
|
|
|
|
|
|
2015-03-10 18:05:59 +00:00
|
|
|
#define TOP_OF_INIT_STACK ((unsigned long)&init_stack + sizeof(init_stack) - \
|
|
|
|
|
TOP_OF_KERNEL_STACK_PADDING)
|
|
|
|
|
|
2017-11-02 07:59:11 +00:00
|
|
|
#define task_top_of_stack(task) ((unsigned long)(task_pt_regs(task) + 1))
|
|
|
|
|
|
2017-11-02 07:59:16 +00:00
|
|
|
#define task_pt_regs(task) \
|
|
|
|
|
({ \
|
|
|
|
|
unsigned long __ptr = (unsigned long)task_stack_page(task); \
|
|
|
|
|
__ptr += THREAD_SIZE - TOP_OF_KERNEL_STACK_PADDING; \
|
|
|
|
|
((struct pt_regs *)__ptr) - 1; \
|
|
|
|
|
})
|
|
|
|
|
|
2008-01-30 12:31:57 +00:00
|
|
|
#ifdef CONFIG_X86_32
|
2008-02-21 03:24:40 +00:00
|
|
|
#define INIT_THREAD { \
|
2015-03-10 18:05:59 +00:00
|
|
|
.sp0 = TOP_OF_INIT_STACK, \
|
2008-02-21 03:24:40 +00:00
|
|
|
.sysenter_cs = __KERNEL_CS, \
|
2008-01-30 12:31:57 +00:00
|
|
|
}
|
|
|
|
|
|
2008-02-21 03:24:40 +00:00
|
|
|
#define KSTK_ESP(task) (task_pt_regs(task)->sp)
|
2008-01-30 12:31:57 +00:00
|
|
|
|
|
|
|
|
#else
|
2023-03-16 22:21:03 +00:00
|
|
|
extern unsigned long __end_init_task[];
|
|
|
|
|
|
|
|
|
|
#define INIT_THREAD { \
|
|
|
|
|
.sp = (unsigned long)&__end_init_task - sizeof(struct pt_regs), \
|
|
|
|
|
}
|
2008-01-30 12:31:57 +00:00
|
|
|
|
2009-11-03 09:22:40 +00:00
|
|
|
extern unsigned long KSTK_ESP(struct task_struct *task);
|
2012-02-14 21:49:48 +00:00
|
|
|
|
2008-01-30 12:31:57 +00:00
|
|
|
#endif /* CONFIG_X86_64 */
|
|
|
|
|
|
2008-02-21 04:18:40 +00:00
|
|
|
extern void start_thread(struct pt_regs *regs, unsigned long new_ip,
|
|
|
|
|
unsigned long new_sp);
|
|
|
|
|
|
2008-02-21 03:24:40 +00:00
|
|
|
/*
|
|
|
|
|
* This decides where the kernel will search for a free chunk of vm
|
2008-01-30 12:31:27 +00:00
|
|
|
* space during mmap's.
|
|
|
|
|
*/
|
2017-03-06 14:17:18 +00:00
|
|
|
#define __TASK_UNMAPPED_BASE(task_size) (PAGE_ALIGN(task_size / 3))
|
2017-07-16 22:59:52 +00:00
|
|
|
#define TASK_UNMAPPED_BASE __TASK_UNMAPPED_BASE(TASK_SIZE_LOW)
|
2008-01-30 12:31:27 +00:00
|
|
|
|
2008-02-21 03:24:40 +00:00
|
|
|
#define KSTK_EIP(task) (task_pt_regs(task)->ip)
|
2008-01-30 12:31:27 +00:00
|
|
|
|
2008-04-13 22:24:18 +00:00
|
|
|
/* Get/set a process' ability to use the timestamp counter instruction */
|
|
|
|
|
#define GET_TSC_CTL(adr) get_tsc_mode((adr))
|
|
|
|
|
#define SET_TSC_CTL(val) set_tsc_mode((val))
|
|
|
|
|
|
|
|
|
|
extern int get_tsc_mode(unsigned long adr);
|
|
|
|
|
extern int set_tsc_mode(unsigned int val);
|
|
|
|
|
|
2017-03-20 08:16:26 +00:00
|
|
|
DECLARE_PER_CPU(u64, msr_misc_features_shadow);
|
|
|
|
|
|
2023-08-14 08:18:40 +00:00
|
|
|
static inline u32 per_cpu_llc_id(unsigned int cpu)
|
2023-08-14 08:18:38 +00:00
|
|
|
{
|
|
|
|
|
return per_cpu(cpu_info.topo.llc_id, cpu);
|
|
|
|
|
}
|
|
|
|
|
|
2023-08-14 08:18:40 +00:00
|
|
|
static inline u32 per_cpu_l2c_id(unsigned int cpu)
|
2023-08-14 08:18:38 +00:00
|
|
|
{
|
|
|
|
|
return per_cpu(cpu_info.topo.l2c_id, cpu);
|
|
|
|
|
}
|
2021-08-17 22:10:46 +00:00
|
|
|
|
2017-06-13 16:28:30 +00:00
|
|
|
#ifdef CONFIG_CPU_SUP_AMD
|
2015-06-15 08:28:15 +00:00
|
|
|
extern u32 amd_get_nodes_per_socket(void);
|
2021-04-25 07:34:51 +00:00
|
|
|
extern u32 amd_get_highest_perf(void);
|
2023-08-04 22:06:43 +00:00
|
|
|
extern void amd_clear_divider(void);
|
2023-07-25 12:17:39 +00:00
|
|
|
extern void amd_check_microcode(void);
|
2017-06-13 16:28:30 +00:00
|
|
|
#else
|
|
|
|
|
static inline u32 amd_get_nodes_per_socket(void) { return 0; }
|
2021-04-25 07:34:51 +00:00
|
|
|
static inline u32 amd_get_highest_perf(void) { return 0; }
|
2023-08-04 22:06:43 +00:00
|
|
|
static inline void amd_clear_divider(void) { }
|
2023-07-25 12:17:39 +00:00
|
|
|
static inline void amd_check_microcode(void) { }
|
2017-06-13 16:28:30 +00:00
|
|
|
#endif
|
2009-09-16 09:33:40 +00:00
|
|
|
|
2012-03-28 17:11:12 +00:00
|
|
|
extern unsigned long arch_align_stack(unsigned long sp);
|
2018-12-28 08:36:03 +00:00
|
|
|
void free_init_pages(const char *what, unsigned long begin, unsigned long end);
|
2019-10-29 21:13:49 +00:00
|
|
|
extern void free_kernel_image_pages(const char *what, void *begin, void *end);
|
2012-03-28 17:11:12 +00:00
|
|
|
|
|
|
|
|
void default_idle(void);
|
2013-02-10 04:08:07 +00:00
|
|
|
#ifdef CONFIG_XEN
|
|
|
|
|
bool xen_set_default_idle(void);
|
|
|
|
|
#else
|
|
|
|
|
#define xen_set_default_idle 0
|
|
|
|
|
#endif
|
2012-03-28 17:11:12 +00:00
|
|
|
|
2022-03-08 15:30:47 +00:00
|
|
|
void __noreturn stop_this_cpu(void *dummy);
|
2023-01-09 15:35:50 +00:00
|
|
|
void microcode_check(struct cpuinfo_x86 *prev_info);
|
2023-01-09 15:35:51 +00:00
|
|
|
void store_cpu_caps(struct cpuinfo_x86 *info);
|
x86/bugs, kvm: Introduce boot-time control of L1TF mitigations
Introduce the 'l1tf=' kernel command line option to allow for boot-time
switching of mitigation that is used on processors affected by L1TF.
The possible values are:
full
Provides all available mitigations for the L1TF vulnerability. Disables
SMT and enables all mitigations in the hypervisors. SMT control via
/sys/devices/system/cpu/smt/control is still possible after boot.
Hypervisors will issue a warning when the first VM is started in
a potentially insecure configuration, i.e. SMT enabled or L1D flush
disabled.
full,force
Same as 'full', but disables SMT control. Implies the 'nosmt=force'
command line option. sysfs control of SMT and the hypervisor flush
control is disabled.
flush
Leaves SMT enabled and enables the conditional hypervisor mitigation.
Hypervisors will issue a warning when the first VM is started in a
potentially insecure configuration, i.e. SMT enabled or L1D flush
disabled.
flush,nosmt
Disables SMT and enables the conditional hypervisor mitigation. SMT
control via /sys/devices/system/cpu/smt/control is still possible
after boot. If SMT is reenabled or flushing disabled at runtime
hypervisors will issue a warning.
flush,nowarn
Same as 'flush', but hypervisors will not warn when
a VM is started in a potentially insecure configuration.
off
Disables hypervisor mitigations and doesn't emit any warnings.
Default is 'flush'.
Let KVM adhere to these semantics, which means:
- 'lt1f=full,force' : Performe L1D flushes. No runtime control
possible.
- 'l1tf=full'
- 'l1tf-flush'
- 'l1tf=flush,nosmt' : Perform L1D flushes and warn on VM start if
SMT has been runtime enabled or L1D flushing
has been run-time enabled
- 'l1tf=flush,nowarn' : Perform L1D flushes and no warnings are emitted.
- 'l1tf=off' : L1D flushes are not performed and no warnings
are emitted.
KVM can always override the L1D flushing behavior using its 'vmentry_l1d_flush'
module parameter except when lt1f=full,force is set.
This makes KVM's private 'nosmt' option redundant, and as it is a bit
non-systematic anyway (this is something to control globally, not on
hypervisor level), remove that option.
Add the missing Documentation entry for the l1tf vulnerability sysfs file
while at it.
Signed-off-by: Jiri Kosina <jkosina@suse.cz>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Tested-by: Jiri Kosina <jkosina@suse.cz>
Reviewed-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-by: Josh Poimboeuf <jpoimboe@redhat.com>
Link: https://lkml.kernel.org/r/20180713142323.202758176@linutronix.de
2018-07-13 14:23:25 +00:00
|
|
|
|
|
|
|
|
enum l1tf_mitigations {
|
|
|
|
|
L1TF_MITIGATION_OFF,
|
|
|
|
|
L1TF_MITIGATION_FLUSH_NOWARN,
|
|
|
|
|
L1TF_MITIGATION_FLUSH,
|
|
|
|
|
L1TF_MITIGATION_FLUSH_NOSMT,
|
|
|
|
|
L1TF_MITIGATION_FULL,
|
|
|
|
|
L1TF_MITIGATION_FULL_FORCE
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
extern enum l1tf_mitigations l1tf_mitigation;
|
|
|
|
|
|
2019-02-18 21:04:08 +00:00
|
|
|
enum mds_mitigations {
|
|
|
|
|
MDS_MITIGATION_OFF,
|
|
|
|
|
MDS_MITIGATION_FULL,
|
2019-02-20 08:40:40 +00:00
|
|
|
MDS_MITIGATION_VMWERV,
|
2019-02-18 21:04:08 +00:00
|
|
|
};
|
|
|
|
|
|
2023-08-09 13:05:00 +00:00
|
|
|
extern bool gds_ucode_mitigated(void);
|
|
|
|
|
|
x86/barrier: Do not serialize MSR accesses on AMD
AMD does not have the requirement for a synchronization barrier when
acccessing a certain group of MSRs. Do not incur that unnecessary
penalty there.
There will be a CPUID bit which explicitly states that a MFENCE is not
needed. Once that bit is added to the APM, this will be extended with
it.
While at it, move to processor.h to avoid include hell. Untangling that
file properly is a matter for another day.
Some notes on the performance aspect of why this is relevant, courtesy
of Kishon VijayAbraham <Kishon.VijayAbraham@amd.com>:
On a AMD Zen4 system with 96 cores, a modified ipi-bench[1] on a VM
shows x2AVIC IPI rate is 3% to 4% lower than AVIC IPI rate. The
ipi-bench is modified so that the IPIs are sent between two vCPUs in the
same CCX. This also requires to pin the vCPU to a physical core to
prevent any latencies. This simulates the use case of pinning vCPUs to
the thread of a single CCX to avoid interrupt IPI latency.
In order to avoid run-to-run variance (for both x2AVIC and AVIC), the
below configurations are done:
1) Disable Power States in BIOS (to prevent the system from going to
lower power state)
2) Run the system at fixed frequency 2500MHz (to prevent the system
from increasing the frequency when the load is more)
With the above configuration:
*) Performance measured using ipi-bench for AVIC:
Average Latency: 1124.98ns [Time to send IPI from one vCPU to another vCPU]
Cumulative throughput: 42.6759M/s [Total number of IPIs sent in a second from
48 vCPUs simultaneously]
*) Performance measured using ipi-bench for x2AVIC:
Average Latency: 1172.42ns [Time to send IPI from one vCPU to another vCPU]
Cumulative throughput: 40.9432M/s [Total number of IPIs sent in a second from
48 vCPUs simultaneously]
From above, x2AVIC latency is ~4% more than AVIC. However, the expectation is
x2AVIC performance to be better or equivalent to AVIC. Upon analyzing
the perf captures, it is observed significant time is spent in
weak_wrmsr_fence() invoked by x2apic_send_IPI().
With the fix to skip weak_wrmsr_fence()
*) Performance measured using ipi-bench for x2AVIC:
Average Latency: 1117.44ns [Time to send IPI from one vCPU to another vCPU]
Cumulative throughput: 42.9608M/s [Total number of IPIs sent in a second from
48 vCPUs simultaneously]
Comparing the performance of x2AVIC with and without the fix, it can be seen
the performance improves by ~4%.
Performance captured using an unmodified ipi-bench using the 'mesh-ipi' option
with and without weak_wrmsr_fence() on a Zen4 system also showed significant
performance improvement without weak_wrmsr_fence(). The 'mesh-ipi' option ignores
CCX or CCD and just picks random vCPU.
Average throughput (10 iterations) with weak_wrmsr_fence(),
Cumulative throughput: 4933374 IPI/s
Average throughput (10 iterations) without weak_wrmsr_fence(),
Cumulative throughput: 6355156 IPI/s
[1] https://github.com/bytedance/kvm-utils/tree/master/microbenchmark/ipi-bench
Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://lore.kernel.org/r/20230622095212.20940-1-bp@alien8.de
2023-10-27 12:24:16 +00:00
|
|
|
/*
|
|
|
|
|
* Make previous memory operations globally visible before
|
|
|
|
|
* a WRMSR.
|
|
|
|
|
*
|
|
|
|
|
* MFENCE makes writes visible, but only affects load/store
|
|
|
|
|
* instructions. WRMSR is unfortunately not a load/store
|
|
|
|
|
* instruction and is unaffected by MFENCE. The LFENCE ensures
|
|
|
|
|
* that the WRMSR is not reordered.
|
|
|
|
|
*
|
|
|
|
|
* Most WRMSRs are full serializing instructions themselves and
|
|
|
|
|
* do not require this barrier. This is only required for the
|
|
|
|
|
* IA32_TSC_DEADLINE and X2APIC MSRs.
|
|
|
|
|
*/
|
|
|
|
|
static inline void weak_wrmsr_fence(void)
|
|
|
|
|
{
|
|
|
|
|
alternative("mfence; lfence", "", ALT_NOT(X86_FEATURE_APIC_MSRS_FENCE));
|
|
|
|
|
}
|
|
|
|
|
|
2008-10-23 05:26:29 +00:00
|
|
|
#endif /* _ASM_X86_PROCESSOR_H */
|