mirrors/linux-stable
1
0
Fork 0
mirror of https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git synced 2024-09-12 21:57:43 +00:00
Code Issues Releases Wiki Activity
linux-stable/arch/x86/kernel
History
Andy Lutomirski 3386bc8aed x86/entry/64: Create a per-CPU SYSCALL entry trampoline 
Handling SYSCALL is tricky: the SYSCALL handler is entered with every
single register (except FLAGS), including RSP, live.  It somehow needs
to set RSP to point to a valid stack, which means it needs to save the
user RSP somewhere and find its own stack pointer.  The canonical way
to do this is with SWAPGS, which lets us access percpu data using the
%gs prefix.

With PAGE_TABLE_ISOLATION-like pagetable switching, this is
problematic.  Without a scratch register, switching CR3 is impossible, so
%gs-based percpu memory would need to be mapped in the user pagetables.
Doing that without information leaks is difficult or impossible.

Instead, use a different sneaky trick.  Map a copy of the first part
of the SYSCALL asm at a different address for each CPU.  Now RIP
varies depending on the CPU, so we can use RIP-relative memory access
to access percpu memory.  By putting the relevant information (one
scratch slot and the stack address) at a constant offset relative to
RIP, we can make SYSCALL work without relying on %gs.

A nice thing about this approach is that we can easily switch it on
and off if we want pagetable switching to be configurable.

The compat variant of SYSCALL doesn't have this problem in the first
place -- there are plenty of scratch registers, since we don't care
about preserving r8-r15.  This patch therefore doesn't touch SYSCALL32
at all.

This patch actually seems to be a small speedup.  With this patch,
SYSCALL touches an extra cache line and an extra virtual page, but
the pipeline no longer stalls waiting for SWAPGS.  It seems that, at
least in a tight loop, the latter outweights the former.

Thanks to David Laight for an optimization tip.

Signed-off-by: Andy Lutomirski <luto@kernel.org>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Borislav Petkov <bpetkov@suse.de>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Dave Hansen <dave.hansen@intel.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: David Laight <David.Laight@aculab.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: Eduardo Valentin <eduval@amazon.com>
Cc: Greg KH <gregkh@linuxfoundation.org>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Juergen Gross <jgross@suse.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rik van Riel <riel@redhat.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: aliguori@amazon.com
Cc: daniel.gruss@iaik.tugraz.at
Cc: hughd@google.com
Cc: keescook@google.com
Link: https://lkml.kernel.org/r/20171204150606.403607157@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-12-17 14:27:50 +01:00
..
acpi License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
apic x86/virt, x86/platform: Merge 'struct x86_hyper' into 'struct x86_platform' and 'struct x86_init' 2017-11-10 10:03:12 +01:00
cpu x86/entry/64: Create a per-CPU SYSCALL entry trampoline 2017-12-17 14:27:50 +01:00
fpu Merge commit 'upstream-x86-entry' into WIP.x86/mm 2017-12-17 12:58:53 +01:00
kprobes License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
.gitignore
alternative.c x86: Clarify/fix no-op barriers for text_poke_bp() 2017-08-10 17:35:19 +02:00
amd_gart_64.c x86: remove arch specific dma_supported implementation 2017-06-28 06:54:46 -07:00
amd_nb.c x86/cpu/AMD: Apply the Erratum 688 fix when the BIOS doesn't 2017-10-22 13:06:02 +02:00
apb_timer.c
aperture_64.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
apm_32.c dmi: Mark all struct dmi_system_id instances const 2017-09-14 11:59:30 +02:00
asm-offsets.c x86/entry/64: Create a per-CPU SYSCALL entry trampoline 2017-12-17 14:27:50 +01:00
asm-offsets_32.c x86/entry/64: Allocate and enable the SYSENTER stack 2017-12-17 13:59:53 +01:00
asm-offsets_64.c x86/entry/64: Separate cpu_current_top_of_stack from TSS.sp0 2017-12-17 13:59:56 +01:00
audit_64.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
bootflag.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
check.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
cpuid.c
crash.c kexec: move vmcoreinfo out of the kernel's .bss section 2017-07-12 16:25:59 -07:00
crash_dump_32.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
crash_dump_64.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
devicetree.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
doublefault.c x86/entry: Fix assumptions that the HW TSS is at the beginning of cpu_tss 2017-12-17 13:59:55 +01:00
dumpstack.c x86/entry: Remap the TSS into the CPU entry area 2017-12-17 13:59:56 +01:00
dumpstack_32.c x86/dumpstack: Add get_stack_info() support for the SYSENTER stack 2017-12-17 13:59:54 +01:00
dumpstack_64.c x86/dumpstack: Add get_stack_info() support for the SYSENTER stack 2017-12-17 13:59:54 +01:00
e820.c x86/boot/e820: Add support to determine the E820 type of an address 2017-07-18 11:38:01 +02:00
early-quirks.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
early_printk.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
ebda.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
eisa.c x86/eisa: Add missing include 2017-08-31 21:34:48 +02:00
espfix_64.c x86/mm: Provide general kernel support for memory encryption 2017-07-18 11:38:00 +02:00
ftrace.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
ftrace_32.S License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
ftrace_64.S License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
head32.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
head64.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
head_32.S Merge commit 'upstream-x86-entry' into WIP.x86/mm 2017-12-17 12:58:53 +01:00
head_64.S Merge commit 'upstream-x86-entry' into WIP.x86/mm 2017-12-17 12:58:53 +01:00
hpet.c x86/hpet: Cure interface abuse in the resume path 2017-08-01 13:02:37 +02:00
hw_breakpoint.c
i8237.c
i8253.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
i8259.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
idt.c x86/idt: Remove X86_TRAP_BP initialization in idt_setup_traps() 2017-11-08 21:05:23 +01:00
io_delay.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
ioport.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
irq.c x86/irq: Remove an old outdated comment about context tracking races 2017-12-17 13:59:53 +01:00
irq_32.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
irq_64.c x86/irq/64: Print the offending IP in the stack overflow warning 2017-12-17 13:59:53 +01:00
irq_work.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
irqinit.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
itmt.c
jump_label.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
kdebugfs.c x86, mpparse, x86/acpi, x86/PCI, x86/dmi, SFI: Use memremap() for RAM mappings 2017-07-18 11:37:58 +02:00
kexec-bzimage64.c x86/boot/e820: Rename the e820_table_firmware to e820_table_kexec 2017-07-05 10:09:02 +02:00
kgdb.c
ksysfs.c x86/sysfs: Fix off-by-one error in loop termination 2017-09-25 09:36:16 +02:00
kvm.c x86/virt: Add enum for hypervisors to replace x86_hyper 2017-11-10 10:03:12 +01:00
kvmclock.c kvm: Return -ENODEV from update_persistent_clock 2017-11-02 18:23:18 +01:00
ldt.c locking/barriers: Convert users of lockless_dereference() to READ_ONCE() 2017-12-17 13:57:15 +01:00
livepatch.c
machine_kexec_32.c x86/idt: Consolidate IDT invalidation 2017-08-29 12:07:26 +02:00
machine_kexec_64.c x86/mm, kexec: Fix memory corruption with SME on successive kexecs 2017-07-30 12:09:12 +02:00
Makefile Merge commit 'upstream-x86-entry' into WIP.x86/mm 2017-12-17 12:58:53 +01:00
mmconf-fam10h_64.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
module.c x86/module: Detect and skip invalid relocations 2017-11-05 09:52:16 +01:00
mpparse.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
msr.c
nmi.c x86/nmi: Use raw lock 2017-08-16 20:40:09 +02:00
nmi_selftest.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
paravirt-spinlocks.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
paravirt.c x86/paravirt: Remove no longer used paravirt functions 2017-09-13 10:55:15 +02:00
paravirt_patch_32.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
paravirt_patch_64.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
pci-calgary_64.c x86: remove arch specific dma_supported implementation 2017-06-28 06:54:46 -07:00
pci-dma.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
pci-iommu_table.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
pci-nommu.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
pci-swiotlb.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
pcspeaker.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
perf_regs.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
platform-quirks.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
pmem.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
probe_roms.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
process.c x86/entry/64: Separate cpu_current_top_of_stack from TSS.sp0 2017-12-17 13:59:56 +01:00
process_32.c x86/entry/64: Remove all remaining direct thread_struct::sp0 reads 2017-11-02 11:04:47 +01:00
process_64.c x86/entry/64: Separate cpu_current_top_of_stack from TSS.sp0 2017-12-17 13:59:56 +01:00
ptrace.c
pvclock.c
quirks.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
reboot.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
reboot_fixups_32.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
relocate_kernel_32.S
relocate_kernel_64.S x86/mm, kexec: Fix memory corruption with SME on successive kexecs 2017-07-30 12:09:12 +02:00
resource.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
rtc.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
setup.c x86/mm/64: Initialize CR4.PCIDE early 2017-09-13 09:54:43 +02:00
setup_percpu.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
signal.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
signal_compat.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
smp.c x86/tracing: Disentangle pagefault and resched IPI tracing key 2017-08-29 11:42:29 +02:00
smpboot.c Merge commit 'upstream-x86-entry' into WIP.x86/mm 2017-12-17 12:58:53 +01:00
stacktrace.c
step.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
sys_x86_64.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
sysfb.c
sysfb_efi.c
sysfb_simplefb.c
tboot.c
tce_64.c
time.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
tls.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
tls.h
topology.c
trace_clock.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
tracepoint.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
traps.c x86/entry/64: Use a per-CPU trampoline stack for IDT entries 2017-12-17 14:27:38 +01:00
tsc.c x86/smpboot: Make optimization of delay calibration work correctly 2017-11-07 16:04:54 +01:00
tsc_msr.c
tsc_sync.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
unwind_frame.c x86/unwind: Disable unwinder warnings on 32-bit 2017-10-10 12:49:49 +02:00
unwind_guess.c x86/unwind: Add the ORC unwinder 2017-07-26 13:18:20 +02:00
unwind_orc.c x86/unwinder: Handle stack overflows more gracefully 2017-12-17 13:59:52 +01:00
uprobes.c
verify_cpu.S x86/boot: Annotate verify_cpu() as a callable function 2017-09-28 09:39:03 +02:00
vm86_32.c Merge commit 'upstream-x86-entry' into WIP.x86/mm 2017-12-17 12:58:53 +01:00
vmlinux.lds.S x86/entry/64: Create a per-CPU SYSCALL entry trampoline 2017-12-17 14:27:50 +01:00
vsmp_64.c
x86_init.c x86/virt, x86/platform: Merge 'struct x86_hyper' into 'struct x86_platform' and 'struct x86_init' 2017-11-10 10:03:12 +01:00