License cleanup: add SPDX GPL-2.0 license identifier to files with no license
Many source files in the tree are missing licensing information, which
makes it harder for compliance tools to determine the correct license.
By default all files without license information are under the default
license of the kernel, which is GPL version 2.
Update the files which contain no license information with the 'GPL-2.0'
SPDX license identifier. The SPDX identifier is a legally binding
shorthand, which can be used instead of the full boiler plate text.
This patch is based on work done by Thomas Gleixner and Kate Stewart and
Philippe Ombredanne.
How this work was done:
Patches were generated and checked against linux-4.14-rc6 for a subset of
the use cases:
- file had no licensing information it it.
- file was a */uapi/* one with no licensing information in it,
- file was a */uapi/* one with existing licensing information,
Further patches will be generated in subsequent months to fix up cases
where non-standard license headers were used, and references to license
had to be inferred by heuristics based on keywords.
The analysis to determine which SPDX License Identifier to be applied to
a file was done in a spreadsheet of side by side results from of the
output of two independent scanners (ScanCode & Windriver) producing SPDX
tag:value files created by Philippe Ombredanne. Philippe prepared the
base worksheet, and did an initial spot review of a few 1000 files.
The 4.13 kernel was the starting point of the analysis with 60,537 files
assessed. Kate Stewart did a file by file comparison of the scanner
results in the spreadsheet to determine which SPDX license identifier(s)
to be applied to the file. She confirmed any determination that was not
immediately clear with lawyers working with the Linux Foundation.
Criteria used to select files for SPDX license identifier tagging was:
- Files considered eligible had to be source code files.
- Make and config files were included as candidates if they contained >5
lines of source
- File already had some variant of a license header in it (even if <5
lines).
All documentation files were explicitly excluded.
The following heuristics were used to determine which SPDX license
identifiers to apply.
- when both scanners couldn't find any license traces, file was
considered to have no license information in it, and the top level
COPYING file license applied.
For non */uapi/* files that summary was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 11139
and resulted in the first patch in this series.
If that file was a */uapi/* path one, it was "GPL-2.0 WITH
Linux-syscall-note" otherwise it was "GPL-2.0". Results of that was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 WITH Linux-syscall-note 930
and resulted in the second patch in this series.
- if a file had some form of licensing information in it, and was one
of the */uapi/* ones, it was denoted with the Linux-syscall-note if
any GPL family license was found in the file or had no licensing in
it (per prior point). Results summary:
SPDX license identifier # files
---------------------------------------------------|------
GPL-2.0 WITH Linux-syscall-note 270
GPL-2.0+ WITH Linux-syscall-note 169
((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) 21
((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) 17
LGPL-2.1+ WITH Linux-syscall-note 15
GPL-1.0+ WITH Linux-syscall-note 14
((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) 5
LGPL-2.0+ WITH Linux-syscall-note 4
LGPL-2.1 WITH Linux-syscall-note 3
((GPL-2.0 WITH Linux-syscall-note) OR MIT) 3
((GPL-2.0 WITH Linux-syscall-note) AND MIT) 1
and that resulted in the third patch in this series.
- when the two scanners agreed on the detected license(s), that became
the concluded license(s).
- when there was disagreement between the two scanners (one detected a
license but the other didn't, or they both detected different
licenses) a manual inspection of the file occurred.
- In most cases a manual inspection of the information in the file
resulted in a clear resolution of the license that should apply (and
which scanner probably needed to revisit its heuristics).
- When it was not immediately clear, the license identifier was
confirmed with lawyers working with the Linux Foundation.
- If there was any question as to the appropriate license identifier,
the file was flagged for further research and to be revisited later
in time.
In total, over 70 hours of logged manual review was done on the
spreadsheet to determine the SPDX license identifiers to apply to the
source files by Kate, Philippe, Thomas and, in some cases, confirmation
by lawyers working with the Linux Foundation.
Kate also obtained a third independent scan of the 4.13 code base from
FOSSology, and compared selected files where the other two scanners
disagreed against that SPDX file, to see if there was new insights. The
Windriver scanner is based on an older version of FOSSology in part, so
they are related.
Thomas did random spot checks in about 500 files from the spreadsheets
for the uapi headers and agreed with SPDX license identifier in the
files he inspected. For the non-uapi files Thomas did random spot checks
in about 15000 files.
In initial set of patches against 4.14-rc6, 3 files were found to have
copy/paste license identifier errors, and have been fixed to reflect the
correct identifier.
Additionally Philippe spent 10 hours this week doing a detailed manual
inspection and review of the 12,461 patched files from the initial patch
version early this week with:
- a full scancode scan run, collecting the matched texts, detected
license ids and scores
- reviewing anything where there was a license detected (about 500+
files) to ensure that the applied SPDX license was correct
- reviewing anything where there was no detection but the patch license
was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied
SPDX license was correct
This produced a worksheet with 20 files needing minor correction. This
worksheet was then exported into 3 different .csv files for the
different types of files to be modified.
These .csv files were then reviewed by Greg. Thomas wrote a script to
parse the csv files and add the proper SPDX tag to the file, in the
format that the file expected. This script was further refined by Greg
based on the output to detect more types of files automatically and to
distinguish between header and source .c files (which need different
comment types.) Finally Greg ran the script using the .csv files to
generate the patches.
Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Philippe Ombredanne <pombredanne@nexb.com>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2017-11-01 14:07:57 +00:00
|
|
|
/* SPDX-License-Identifier: GPL-2.0 */
|
2014-05-08 19:21:52 +00:00
|
|
|
/*
|
|
|
|
|
* Copyright (C) 2014 Steven Rostedt, Red Hat Inc
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#include <linux/linkage.h>
|
|
|
|
|
#include <asm/ptrace.h>
|
|
|
|
|
#include <asm/ftrace.h>
|
2016-01-11 16:04:34 +00:00
|
|
|
#include <asm/export.h>
|
2018-01-11 21:46:29 +00:00
|
|
|
#include <asm/nospec-branch.h>
|
2018-01-23 04:07:46 +00:00
|
|
|
#include <asm/unwind_hints.h>
|
2019-05-07 21:25:50 +00:00
|
|
|
#include <asm/frame.h>
|
2014-05-08 19:21:52 +00:00
|
|
|
|
|
|
|
|
.code64
|
2020-03-25 18:45:26 +00:00
|
|
|
.section .text, "ax"
|
2014-05-08 19:21:52 +00:00
|
|
|
|
2014-11-24 23:08:48 +00:00
|
|
|
#ifdef CONFIG_FRAME_POINTER
|
|
|
|
|
/* Save parent and function stack frames (rip and rbp) */
|
|
|
|
|
# define MCOUNT_FRAME_SIZE (8+16*2)
|
|
|
|
|
#else
|
|
|
|
|
/* No need to save a stack frame */
|
2018-01-23 04:07:46 +00:00
|
|
|
# define MCOUNT_FRAME_SIZE 0
|
2014-11-24 23:08:48 +00:00
|
|
|
#endif /* CONFIG_FRAME_POINTER */
|
|
|
|
|
|
2014-11-24 19:26:38 +00:00
|
|
|
/* Size of stack used to save mcount regs in save_mcount_regs */
|
2020-04-01 14:50:40 +00:00
|
|
|
#define MCOUNT_REG_SIZE (FRAME_SIZE + MCOUNT_FRAME_SIZE)
|
2014-11-24 19:26:38 +00:00
|
|
|
|
2014-11-24 16:43:39 +00:00
|
|
|
/*
|
|
|
|
|
* gcc -pg option adds a call to 'mcount' in most functions.
|
|
|
|
|
* When -mfentry is used, the call is to 'fentry' and not 'mcount'
|
|
|
|
|
* and is done before the function's stack frame is set up.
|
|
|
|
|
* They both require a set of regs to be saved before calling
|
|
|
|
|
* any C code and restored before returning back to the function.
|
|
|
|
|
*
|
|
|
|
|
* On boot up, all these calls are converted into nops. When tracing
|
|
|
|
|
* is enabled, the call can jump to either ftrace_caller or
|
|
|
|
|
* ftrace_regs_caller. Callbacks (tracing functions) that require
|
|
|
|
|
* ftrace_regs_caller (like kprobes) need to have pt_regs passed to
|
|
|
|
|
* it. For this reason, the size of the pt_regs structure will be
|
|
|
|
|
* allocated on the stack and the required mcount registers will
|
|
|
|
|
* be saved in the locations that pt_regs has them in.
|
|
|
|
|
*/
|
|
|
|
|
|
2014-11-25 02:38:40 +00:00
|
|
|
/*
|
|
|
|
|
* @added: the amount of stack added before calling this
|
|
|
|
|
*
|
|
|
|
|
* After this is called, the following registers contain:
|
|
|
|
|
*
|
|
|
|
|
* %rdi - holds the address that called the trampoline
|
|
|
|
|
* %rsi - holds the parent function (traced function's return address)
|
|
|
|
|
* %rdx - holds the original %rbp
|
|
|
|
|
*/
|
2014-11-24 18:06:05 +00:00
|
|
|
.macro save_mcount_regs added=0
|
2014-11-24 23:08:48 +00:00
|
|
|
|
2018-01-23 04:07:46 +00:00
|
|
|
#ifdef CONFIG_FRAME_POINTER
|
|
|
|
|
/* Save the original rbp */
|
2014-11-24 23:08:48 +00:00
|
|
|
pushq %rbp
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Stack traces will stop at the ftrace trampoline if the frame pointer
|
|
|
|
|
* is not set up properly. If fentry is used, we need to save a frame
|
|
|
|
|
* pointer for the parent as well as the function traced, because the
|
|
|
|
|
* fentry is called before the stack frame is set up, where as mcount
|
|
|
|
|
* is called afterward.
|
|
|
|
|
*/
|
ftrace/x86: Remove mcount support
There's two methods of enabling function tracing in Linux on x86. One is
with just "gcc -pg" and the other is "gcc -pg -mfentry". The former will use
calls to a special function "mcount" after the frame is set up in all C
functions. The latter will add calls to a special function called "fentry"
as the very first instruction of all C functions.
At compile time, there is a check to see if gcc supports, -mfentry, and if
it does, it will use that, because it is more versatile and less error prone
for function tracing.
Starting with v4.19, the minimum gcc supported to build the Linux kernel,
was raised to version 4.6. That also happens to be the first gcc version to
support -mfentry. Since on x86, using gcc versions from 4.6 and beyond will
unconditionally enable the -mfentry, it will no longer use mcount as the
method for inserting calls into the C functions of the kernel. This means
that there is no point in continuing to maintain mcount in x86.
Remove support for using mcount. This makes the code less complex, and will
also allow it to be simplified in the future.
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Jiri Kosina <jkosina@suse.cz>
Acked-by: Josh Poimboeuf <jpoimboe@redhat.com>
Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
2019-05-09 19:32:05 +00:00
|
|
|
|
2014-11-24 23:08:48 +00:00
|
|
|
/* Save the parent pointer (skip orig rbp and our return address) */
|
|
|
|
|
pushq \added+8*2(%rsp)
|
|
|
|
|
pushq %rbp
|
|
|
|
|
movq %rsp, %rbp
|
|
|
|
|
/* Save the return address (now skip orig rbp, rbp and parent) */
|
|
|
|
|
pushq \added+8*3(%rsp)
|
|
|
|
|
pushq %rbp
|
|
|
|
|
movq %rsp, %rbp
|
|
|
|
|
#endif /* CONFIG_FRAME_POINTER */
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* We add enough stack to save all regs.
|
|
|
|
|
*/
|
2020-04-01 14:50:40 +00:00
|
|
|
subq $(FRAME_SIZE), %rsp
|
2014-11-24 16:30:58 +00:00
|
|
|
movq %rax, RAX(%rsp)
|
|
|
|
|
movq %rcx, RCX(%rsp)
|
|
|
|
|
movq %rdx, RDX(%rsp)
|
|
|
|
|
movq %rsi, RSI(%rsp)
|
|
|
|
|
movq %rdi, RDI(%rsp)
|
|
|
|
|
movq %r8, R8(%rsp)
|
|
|
|
|
movq %r9, R9(%rsp)
|
2019-11-08 18:11:39 +00:00
|
|
|
movq $0, ORIG_RAX(%rsp)
|
2014-11-24 23:08:48 +00:00
|
|
|
/*
|
|
|
|
|
* Save the original RBP. Even though the mcount ABI does not
|
|
|
|
|
* require this, it helps out callers.
|
|
|
|
|
*/
|
2018-01-23 04:07:46 +00:00
|
|
|
#ifdef CONFIG_FRAME_POINTER
|
2014-11-24 23:08:48 +00:00
|
|
|
movq MCOUNT_REG_SIZE-8(%rsp), %rdx
|
2018-01-23 04:07:46 +00:00
|
|
|
#else
|
|
|
|
|
movq %rbp, %rdx
|
|
|
|
|
#endif
|
2014-11-24 23:08:48 +00:00
|
|
|
movq %rdx, RBP(%rsp)
|
|
|
|
|
|
2014-11-25 02:38:40 +00:00
|
|
|
/* Copy the parent address into %rsi (second parameter) */
|
|
|
|
|
movq MCOUNT_REG_SIZE+8+\added(%rsp), %rsi
|
|
|
|
|
|
2014-11-24 16:30:58 +00:00
|
|
|
/* Move RIP to its proper location */
|
2014-11-24 19:26:38 +00:00
|
|
|
movq MCOUNT_REG_SIZE+\added(%rsp), %rdi
|
2014-11-24 18:21:09 +00:00
|
|
|
movq %rdi, RIP(%rsp)
|
2014-11-25 02:38:40 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Now %rdi (the first parameter) has the return address of
|
|
|
|
|
* where ftrace_call returns. But the callbacks expect the
|
2014-11-25 02:00:34 +00:00
|
|
|
* address of the call itself.
|
2014-11-25 02:38:40 +00:00
|
|
|
*/
|
|
|
|
|
subq $MCOUNT_INSN_SIZE, %rdi
|
2014-11-24 16:30:58 +00:00
|
|
|
.endm
|
|
|
|
|
|
2019-11-08 18:11:39 +00:00
|
|
|
.macro restore_mcount_regs save=0
|
|
|
|
|
|
|
|
|
|
/* ftrace_regs_caller or frame pointers require this */
|
|
|
|
|
movq RBP(%rsp), %rbp
|
|
|
|
|
|
2014-11-24 16:30:58 +00:00
|
|
|
movq R9(%rsp), %r9
|
|
|
|
|
movq R8(%rsp), %r8
|
|
|
|
|
movq RDI(%rsp), %rdi
|
|
|
|
|
movq RSI(%rsp), %rsi
|
|
|
|
|
movq RDX(%rsp), %rdx
|
|
|
|
|
movq RCX(%rsp), %rcx
|
|
|
|
|
movq RAX(%rsp), %rax
|
2014-11-24 23:08:48 +00:00
|
|
|
|
2019-11-08 18:11:39 +00:00
|
|
|
addq $MCOUNT_REG_SIZE-\save, %rsp
|
2014-11-24 23:08:48 +00:00
|
|
|
|
2014-11-24 16:30:58 +00:00
|
|
|
.endm
|
|
|
|
|
|
2014-11-24 19:54:27 +00:00
|
|
|
#ifdef CONFIG_DYNAMIC_FTRACE
|
|
|
|
|
|
2019-10-21 15:18:23 +00:00
|
|
|
SYM_FUNC_START(__fentry__)
|
2021-12-04 13:43:40 +00:00
|
|
|
RET
|
2019-10-21 15:18:23 +00:00
|
|
|
SYM_FUNC_END(__fentry__)
|
|
|
|
|
EXPORT_SYMBOL(__fentry__)
|
2014-11-24 19:54:27 +00:00
|
|
|
|
2019-10-11 11:51:04 +00:00
|
|
|
SYM_FUNC_START(ftrace_caller)
|
2014-11-25 02:38:40 +00:00
|
|
|
/* save_mcount_regs fills in first two parameters */
|
|
|
|
|
save_mcount_regs
|
|
|
|
|
|
2020-10-27 14:55:55 +00:00
|
|
|
/* Stack - skipping return address of ftrace_caller */
|
|
|
|
|
leaq MCOUNT_REG_SIZE+8(%rsp), %rcx
|
|
|
|
|
movq %rcx, RSP(%rsp)
|
|
|
|
|
|
2019-10-11 11:50:57 +00:00
|
|
|
SYM_INNER_LABEL(ftrace_caller_op_ptr, SYM_L_GLOBAL)
|
2022-03-08 15:30:41 +00:00
|
|
|
ANNOTATE_NOENDBR
|
2014-11-25 02:38:40 +00:00
|
|
|
/* Load the ftrace_ops into the 3rd parameter */
|
|
|
|
|
movq function_trace_op(%rip), %rdx
|
|
|
|
|
|
2020-10-27 14:55:55 +00:00
|
|
|
/* regs go into 4th parameter */
|
|
|
|
|
leaq (%rsp), %rcx
|
|
|
|
|
|
|
|
|
|
/* Only ops with REGS flag set should have CS register set */
|
|
|
|
|
movq $0, CS(%rsp)
|
2014-05-08 19:21:52 +00:00
|
|
|
|
2019-10-11 11:50:57 +00:00
|
|
|
SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL)
|
2022-03-08 15:30:41 +00:00
|
|
|
ANNOTATE_NOENDBR
|
2014-05-08 19:21:52 +00:00
|
|
|
call ftrace_stub
|
|
|
|
|
|
2020-10-28 21:15:27 +00:00
|
|
|
/* Handlers can change the RIP */
|
|
|
|
|
movq RIP(%rsp), %rax
|
|
|
|
|
movq %rax, MCOUNT_REG_SIZE(%rsp)
|
|
|
|
|
|
2014-11-24 16:43:39 +00:00
|
|
|
restore_mcount_regs
|
ftrace/x86: Add dynamic allocated trampoline for ftrace_ops
The current method of handling multiple function callbacks is to register
a list function callback that calls all the other callbacks based on
their hash tables and compare it to the function that the callback was
called on. But this is very inefficient.
For example, if you are tracing all functions in the kernel and then
add a kprobe to a function such that the kprobe uses ftrace, the
mcount trampoline will switch from calling the function trace callback
to calling the list callback that will iterate over all registered
ftrace_ops (in this case, the function tracer and the kprobes callback).
That means for every function being traced it checks the hash of the
ftrace_ops for function tracing and kprobes, even though the kprobes
is only set at a single function. The kprobes ftrace_ops is checked
for every function being traced!
Instead of calling the list function for functions that are only being
traced by a single callback, we can call a dynamically allocated
trampoline that calls the callback directly. The function graph tracer
already uses a direct call trampoline when it is being traced by itself
but it is not dynamically allocated. It's trampoline is static in the
kernel core. The infrastructure that called the function graph trampoline
can also be used to call a dynamically allocated one.
For now, only ftrace_ops that are not dynamically allocated can have
a trampoline. That is, users such as function tracer or stack tracer.
kprobes and perf allocate their ftrace_ops, and until there's a safe
way to free the trampoline, it can not be used. The dynamically allocated
ftrace_ops may, although, use the trampoline if the kernel is not
compiled with CONFIG_PREEMPT. But that will come later.
Tested-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Tested-by: Jiri Kosina <jkosina@suse.cz>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2014-07-03 03:23:31 +00:00
|
|
|
|
|
|
|
|
/*
|
2016-02-16 08:43:21 +00:00
|
|
|
* The code up to this label is copied into trampolines so
|
|
|
|
|
* think twice before adding any new code or changing the
|
|
|
|
|
* layout here.
|
ftrace/x86: Add dynamic allocated trampoline for ftrace_ops
The current method of handling multiple function callbacks is to register
a list function callback that calls all the other callbacks based on
their hash tables and compare it to the function that the callback was
called on. But this is very inefficient.
For example, if you are tracing all functions in the kernel and then
add a kprobe to a function such that the kprobe uses ftrace, the
mcount trampoline will switch from calling the function trace callback
to calling the list callback that will iterate over all registered
ftrace_ops (in this case, the function tracer and the kprobes callback).
That means for every function being traced it checks the hash of the
ftrace_ops for function tracing and kprobes, even though the kprobes
is only set at a single function. The kprobes ftrace_ops is checked
for every function being traced!
Instead of calling the list function for functions that are only being
traced by a single callback, we can call a dynamically allocated
trampoline that calls the callback directly. The function graph tracer
already uses a direct call trampoline when it is being traced by itself
but it is not dynamically allocated. It's trampoline is static in the
kernel core. The infrastructure that called the function graph trampoline
can also be used to call a dynamically allocated one.
For now, only ftrace_ops that are not dynamically allocated can have
a trampoline. That is, users such as function tracer or stack tracer.
kprobes and perf allocate their ftrace_ops, and until there's a safe
way to free the trampoline, it can not be used. The dynamically allocated
ftrace_ops may, although, use the trampoline if the kernel is not
compiled with CONFIG_PREEMPT. But that will come later.
Tested-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Tested-by: Jiri Kosina <jkosina@suse.cz>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2014-07-03 03:23:31 +00:00
|
|
|
*/
|
x86,ftrace: Fix ftrace_regs_caller() unwind
The ftrace_regs_caller() trampoline does something 'funny' when there
is a direct-caller present. In that case it stuffs the 'direct-caller'
address on the return stack and then exits the function. This then
results in 'returning' to the direct-caller with the exact registers
we came in with -- an indirect tail-call without using a register.
This however (rightfully) confuses objtool because the function shares
a few instruction in order to have a single exit path, but the stack
layout is different for them, depending through which path we came
there.
This is currently cludged by forcing the stack state to the non-direct
case, but this generates actively wrong (ORC) unwind information for
the direct case, leading to potential broken unwinds.
Fix this issue by fully separating the exit paths. This results in
having to poke a second RET into the trampoline copy, see
ftrace_regs_caller_ret.
This brings us to a second objtool problem, in order for it to
perceive the 'jmp ftrace_epilogue' as a function exit, it needs to be
recognised as a tail call. In order to make that happen,
ftrace_epilogue needs to be the start of an STT_FUNC, so re-arrange
code to make this so.
Finally, a third issue is that objtool requires functions to exit with
the same stack layout they started with, which is obviously violated
in the direct case, employ the new HINT_RET_OFFSET to tell objtool
this is an expected exception.
Together, this results in generating correct ORC unwind information
for the ftrace_regs_caller() function and it's trampoline copies.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Miroslav Benes <mbenes@suse.cz>
Reviewed-by: Alexandre Chartre <alexandre.chartre@oracle.com>
Acked-by: Josh Poimboeuf <jpoimboe@redhat.com>
Link: https://lkml.kernel.org/r/20200416115118.749606694@infradead.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2020-04-01 14:53:19 +00:00
|
|
|
SYM_INNER_LABEL(ftrace_caller_end, SYM_L_GLOBAL)
|
2022-03-08 15:30:41 +00:00
|
|
|
ANNOTATE_NOENDBR
|
2014-05-08 19:21:52 +00:00
|
|
|
|
x86,ftrace: Fix ftrace_regs_caller() unwind
The ftrace_regs_caller() trampoline does something 'funny' when there
is a direct-caller present. In that case it stuffs the 'direct-caller'
address on the return stack and then exits the function. This then
results in 'returning' to the direct-caller with the exact registers
we came in with -- an indirect tail-call without using a register.
This however (rightfully) confuses objtool because the function shares
a few instruction in order to have a single exit path, but the stack
layout is different for them, depending through which path we came
there.
This is currently cludged by forcing the stack state to the non-direct
case, but this generates actively wrong (ORC) unwind information for
the direct case, leading to potential broken unwinds.
Fix this issue by fully separating the exit paths. This results in
having to poke a second RET into the trampoline copy, see
ftrace_regs_caller_ret.
This brings us to a second objtool problem, in order for it to
perceive the 'jmp ftrace_epilogue' as a function exit, it needs to be
recognised as a tail call. In order to make that happen,
ftrace_epilogue needs to be the start of an STT_FUNC, so re-arrange
code to make this so.
Finally, a third issue is that objtool requires functions to exit with
the same stack layout they started with, which is obviously violated
in the direct case, employ the new HINT_RET_OFFSET to tell objtool
this is an expected exception.
Together, this results in generating correct ORC unwind information
for the ftrace_regs_caller() function and it's trampoline copies.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Miroslav Benes <mbenes@suse.cz>
Reviewed-by: Alexandre Chartre <alexandre.chartre@oracle.com>
Acked-by: Josh Poimboeuf <jpoimboe@redhat.com>
Link: https://lkml.kernel.org/r/20200416115118.749606694@infradead.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2020-04-01 14:53:19 +00:00
|
|
|
jmp ftrace_epilogue
|
|
|
|
|
SYM_FUNC_END(ftrace_caller);
|
2022-06-03 15:04:44 +00:00
|
|
|
STACK_FRAME_NON_STANDARD_FP(ftrace_caller)
|
x86,ftrace: Fix ftrace_regs_caller() unwind
The ftrace_regs_caller() trampoline does something 'funny' when there
is a direct-caller present. In that case it stuffs the 'direct-caller'
address on the return stack and then exits the function. This then
results in 'returning' to the direct-caller with the exact registers
we came in with -- an indirect tail-call without using a register.
This however (rightfully) confuses objtool because the function shares
a few instruction in order to have a single exit path, but the stack
layout is different for them, depending through which path we came
there.
This is currently cludged by forcing the stack state to the non-direct
case, but this generates actively wrong (ORC) unwind information for
the direct case, leading to potential broken unwinds.
Fix this issue by fully separating the exit paths. This results in
having to poke a second RET into the trampoline copy, see
ftrace_regs_caller_ret.
This brings us to a second objtool problem, in order for it to
perceive the 'jmp ftrace_epilogue' as a function exit, it needs to be
recognised as a tail call. In order to make that happen,
ftrace_epilogue needs to be the start of an STT_FUNC, so re-arrange
code to make this so.
Finally, a third issue is that objtool requires functions to exit with
the same stack layout they started with, which is obviously violated
in the direct case, employ the new HINT_RET_OFFSET to tell objtool
this is an expected exception.
Together, this results in generating correct ORC unwind information
for the ftrace_regs_caller() function and it's trampoline copies.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Miroslav Benes <mbenes@suse.cz>
Reviewed-by: Alexandre Chartre <alexandre.chartre@oracle.com>
Acked-by: Josh Poimboeuf <jpoimboe@redhat.com>
Link: https://lkml.kernel.org/r/20200416115118.749606694@infradead.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2020-04-01 14:53:19 +00:00
|
|
|
|
|
|
|
|
SYM_FUNC_START(ftrace_epilogue)
|
2018-12-08 17:58:51 +00:00
|
|
|
/*
|
|
|
|
|
* This is weak to keep gas from relaxing the jumps.
|
|
|
|
|
*/
|
2019-10-11 11:51:08 +00:00
|
|
|
SYM_INNER_LABEL_ALIGN(ftrace_stub, SYM_L_WEAK)
|
2021-01-21 21:29:21 +00:00
|
|
|
UNWIND_HINT_FUNC
|
2022-03-08 15:30:31 +00:00
|
|
|
ENDBR
|
2021-12-04 13:43:40 +00:00
|
|
|
RET
|
x86,ftrace: Fix ftrace_regs_caller() unwind
The ftrace_regs_caller() trampoline does something 'funny' when there
is a direct-caller present. In that case it stuffs the 'direct-caller'
address on the return stack and then exits the function. This then
results in 'returning' to the direct-caller with the exact registers
we came in with -- an indirect tail-call without using a register.
This however (rightfully) confuses objtool because the function shares
a few instruction in order to have a single exit path, but the stack
layout is different for them, depending through which path we came
there.
This is currently cludged by forcing the stack state to the non-direct
case, but this generates actively wrong (ORC) unwind information for
the direct case, leading to potential broken unwinds.
Fix this issue by fully separating the exit paths. This results in
having to poke a second RET into the trampoline copy, see
ftrace_regs_caller_ret.
This brings us to a second objtool problem, in order for it to
perceive the 'jmp ftrace_epilogue' as a function exit, it needs to be
recognised as a tail call. In order to make that happen,
ftrace_epilogue needs to be the start of an STT_FUNC, so re-arrange
code to make this so.
Finally, a third issue is that objtool requires functions to exit with
the same stack layout they started with, which is obviously violated
in the direct case, employ the new HINT_RET_OFFSET to tell objtool
this is an expected exception.
Together, this results in generating correct ORC unwind information
for the ftrace_regs_caller() function and it's trampoline copies.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Miroslav Benes <mbenes@suse.cz>
Reviewed-by: Alexandre Chartre <alexandre.chartre@oracle.com>
Acked-by: Josh Poimboeuf <jpoimboe@redhat.com>
Link: https://lkml.kernel.org/r/20200416115118.749606694@infradead.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2020-04-01 14:53:19 +00:00
|
|
|
SYM_FUNC_END(ftrace_epilogue)
|
2014-05-08 19:21:52 +00:00
|
|
|
|
2019-10-11 11:51:04 +00:00
|
|
|
SYM_FUNC_START(ftrace_regs_caller)
|
2014-11-24 18:06:05 +00:00
|
|
|
/* Save the current flags before any operations that can change them */
|
2014-05-08 19:21:52 +00:00
|
|
|
pushfq
|
|
|
|
|
|
2014-11-24 18:06:05 +00:00
|
|
|
/* added 8 bytes to save flags */
|
2014-11-25 02:38:40 +00:00
|
|
|
save_mcount_regs 8
|
|
|
|
|
/* save_mcount_regs fills in first two parameters */
|
|
|
|
|
|
2019-10-11 11:50:57 +00:00
|
|
|
SYM_INNER_LABEL(ftrace_regs_caller_op_ptr, SYM_L_GLOBAL)
|
2022-03-08 15:30:41 +00:00
|
|
|
ANNOTATE_NOENDBR
|
2014-11-25 02:38:40 +00:00
|
|
|
/* Load the ftrace_ops into the 3rd parameter */
|
|
|
|
|
movq function_trace_op(%rip), %rdx
|
2014-05-08 19:21:52 +00:00
|
|
|
|
|
|
|
|
/* Save the rest of pt_regs */
|
|
|
|
|
movq %r15, R15(%rsp)
|
|
|
|
|
movq %r14, R14(%rsp)
|
|
|
|
|
movq %r13, R13(%rsp)
|
|
|
|
|
movq %r12, R12(%rsp)
|
|
|
|
|
movq %r11, R11(%rsp)
|
|
|
|
|
movq %r10, R10(%rsp)
|
|
|
|
|
movq %rbx, RBX(%rsp)
|
|
|
|
|
/* Copy saved flags */
|
2014-11-24 19:26:38 +00:00
|
|
|
movq MCOUNT_REG_SIZE(%rsp), %rcx
|
2014-05-08 19:21:52 +00:00
|
|
|
movq %rcx, EFLAGS(%rsp)
|
|
|
|
|
/* Kernel segments */
|
|
|
|
|
movq $__KERNEL_DS, %rcx
|
|
|
|
|
movq %rcx, SS(%rsp)
|
|
|
|
|
movq $__KERNEL_CS, %rcx
|
|
|
|
|
movq %rcx, CS(%rsp)
|
2014-11-24 18:06:05 +00:00
|
|
|
/* Stack - skipping return address and flags */
|
2014-11-24 19:26:38 +00:00
|
|
|
leaq MCOUNT_REG_SIZE+8*2(%rsp), %rcx
|
2014-05-08 19:21:52 +00:00
|
|
|
movq %rcx, RSP(%rsp)
|
|
|
|
|
|
2019-05-07 21:25:50 +00:00
|
|
|
ENCODE_FRAME_POINTER
|
|
|
|
|
|
2014-05-08 19:21:52 +00:00
|
|
|
/* regs go into 4th parameter */
|
|
|
|
|
leaq (%rsp), %rcx
|
|
|
|
|
|
2019-10-11 11:50:57 +00:00
|
|
|
SYM_INNER_LABEL(ftrace_regs_call, SYM_L_GLOBAL)
|
2022-03-08 15:30:41 +00:00
|
|
|
ANNOTATE_NOENDBR
|
2014-05-08 19:21:52 +00:00
|
|
|
call ftrace_stub
|
|
|
|
|
|
|
|
|
|
/* Copy flags back to SS, to restore them */
|
|
|
|
|
movq EFLAGS(%rsp), %rax
|
2014-11-24 19:26:38 +00:00
|
|
|
movq %rax, MCOUNT_REG_SIZE(%rsp)
|
2014-05-08 19:21:52 +00:00
|
|
|
|
|
|
|
|
/* Handlers can change the RIP */
|
|
|
|
|
movq RIP(%rsp), %rax
|
2014-11-24 19:26:38 +00:00
|
|
|
movq %rax, MCOUNT_REG_SIZE+8(%rsp)
|
2014-05-08 19:21:52 +00:00
|
|
|
|
|
|
|
|
/* restore the rest of pt_regs */
|
|
|
|
|
movq R15(%rsp), %r15
|
|
|
|
|
movq R14(%rsp), %r14
|
|
|
|
|
movq R13(%rsp), %r13
|
|
|
|
|
movq R12(%rsp), %r12
|
|
|
|
|
movq R10(%rsp), %r10
|
|
|
|
|
movq RBX(%rsp), %rbx
|
|
|
|
|
|
2019-11-08 18:11:39 +00:00
|
|
|
movq ORIG_RAX(%rsp), %rax
|
|
|
|
|
movq %rax, MCOUNT_REG_SIZE-8(%rsp)
|
|
|
|
|
|
x86,ftrace: Fix ftrace_regs_caller() unwind
The ftrace_regs_caller() trampoline does something 'funny' when there
is a direct-caller present. In that case it stuffs the 'direct-caller'
address on the return stack and then exits the function. This then
results in 'returning' to the direct-caller with the exact registers
we came in with -- an indirect tail-call without using a register.
This however (rightfully) confuses objtool because the function shares
a few instruction in order to have a single exit path, but the stack
layout is different for them, depending through which path we came
there.
This is currently cludged by forcing the stack state to the non-direct
case, but this generates actively wrong (ORC) unwind information for
the direct case, leading to potential broken unwinds.
Fix this issue by fully separating the exit paths. This results in
having to poke a second RET into the trampoline copy, see
ftrace_regs_caller_ret.
This brings us to a second objtool problem, in order for it to
perceive the 'jmp ftrace_epilogue' as a function exit, it needs to be
recognised as a tail call. In order to make that happen,
ftrace_epilogue needs to be the start of an STT_FUNC, so re-arrange
code to make this so.
Finally, a third issue is that objtool requires functions to exit with
the same stack layout they started with, which is obviously violated
in the direct case, employ the new HINT_RET_OFFSET to tell objtool
this is an expected exception.
Together, this results in generating correct ORC unwind information
for the ftrace_regs_caller() function and it's trampoline copies.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Miroslav Benes <mbenes@suse.cz>
Reviewed-by: Alexandre Chartre <alexandre.chartre@oracle.com>
Acked-by: Josh Poimboeuf <jpoimboe@redhat.com>
Link: https://lkml.kernel.org/r/20200416115118.749606694@infradead.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2020-04-01 14:53:19 +00:00
|
|
|
/*
|
|
|
|
|
* If ORIG_RAX is anything but zero, make this a call to that.
|
|
|
|
|
* See arch_ftrace_set_direct_caller().
|
|
|
|
|
*/
|
2020-04-01 14:51:11 +00:00
|
|
|
testq %rax, %rax
|
2020-04-22 16:25:42 +00:00
|
|
|
SYM_INNER_LABEL(ftrace_regs_caller_jmp, SYM_L_GLOBAL)
|
2022-03-08 15:30:41 +00:00
|
|
|
ANNOTATE_NOENDBR
|
2020-04-22 16:25:40 +00:00
|
|
|
jnz 1f
|
2019-11-08 18:11:39 +00:00
|
|
|
|
2020-04-22 16:25:40 +00:00
|
|
|
restore_mcount_regs
|
2014-05-08 19:21:52 +00:00
|
|
|
/* Restore flags */
|
|
|
|
|
popfq
|
|
|
|
|
|
ftrace/x86: Add dynamic allocated trampoline for ftrace_ops
The current method of handling multiple function callbacks is to register
a list function callback that calls all the other callbacks based on
their hash tables and compare it to the function that the callback was
called on. But this is very inefficient.
For example, if you are tracing all functions in the kernel and then
add a kprobe to a function such that the kprobe uses ftrace, the
mcount trampoline will switch from calling the function trace callback
to calling the list callback that will iterate over all registered
ftrace_ops (in this case, the function tracer and the kprobes callback).
That means for every function being traced it checks the hash of the
ftrace_ops for function tracing and kprobes, even though the kprobes
is only set at a single function. The kprobes ftrace_ops is checked
for every function being traced!
Instead of calling the list function for functions that are only being
traced by a single callback, we can call a dynamically allocated
trampoline that calls the callback directly. The function graph tracer
already uses a direct call trampoline when it is being traced by itself
but it is not dynamically allocated. It's trampoline is static in the
kernel core. The infrastructure that called the function graph trampoline
can also be used to call a dynamically allocated one.
For now, only ftrace_ops that are not dynamically allocated can have
a trampoline. That is, users such as function tracer or stack tracer.
kprobes and perf allocate their ftrace_ops, and until there's a safe
way to free the trampoline, it can not be used. The dynamically allocated
ftrace_ops may, although, use the trampoline if the kernel is not
compiled with CONFIG_PREEMPT. But that will come later.
Tested-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Tested-by: Jiri Kosina <jkosina@suse.cz>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2014-07-03 03:23:31 +00:00
|
|
|
/*
|
2016-02-16 08:43:21 +00:00
|
|
|
* As this jmp to ftrace_epilogue can be a short jump
|
ftrace/x86: Add dynamic allocated trampoline for ftrace_ops
The current method of handling multiple function callbacks is to register
a list function callback that calls all the other callbacks based on
their hash tables and compare it to the function that the callback was
called on. But this is very inefficient.
For example, if you are tracing all functions in the kernel and then
add a kprobe to a function such that the kprobe uses ftrace, the
mcount trampoline will switch from calling the function trace callback
to calling the list callback that will iterate over all registered
ftrace_ops (in this case, the function tracer and the kprobes callback).
That means for every function being traced it checks the hash of the
ftrace_ops for function tracing and kprobes, even though the kprobes
is only set at a single function. The kprobes ftrace_ops is checked
for every function being traced!
Instead of calling the list function for functions that are only being
traced by a single callback, we can call a dynamically allocated
trampoline that calls the callback directly. The function graph tracer
already uses a direct call trampoline when it is being traced by itself
but it is not dynamically allocated. It's trampoline is static in the
kernel core. The infrastructure that called the function graph trampoline
can also be used to call a dynamically allocated one.
For now, only ftrace_ops that are not dynamically allocated can have
a trampoline. That is, users such as function tracer or stack tracer.
kprobes and perf allocate their ftrace_ops, and until there's a safe
way to free the trampoline, it can not be used. The dynamically allocated
ftrace_ops may, although, use the trampoline if the kernel is not
compiled with CONFIG_PREEMPT. But that will come later.
Tested-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Tested-by: Jiri Kosina <jkosina@suse.cz>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2014-07-03 03:23:31 +00:00
|
|
|
* it must not be copied into the trampoline.
|
|
|
|
|
* The trampoline will add the code to jump
|
|
|
|
|
* to the return.
|
|
|
|
|
*/
|
2020-04-22 16:25:41 +00:00
|
|
|
SYM_INNER_LABEL(ftrace_regs_caller_end, SYM_L_GLOBAL)
|
2022-03-08 15:30:41 +00:00
|
|
|
ANNOTATE_NOENDBR
|
2016-02-16 08:43:21 +00:00
|
|
|
jmp ftrace_epilogue
|
2014-06-25 15:59:45 +00:00
|
|
|
|
2020-04-22 16:25:40 +00:00
|
|
|
/* Swap the flags with orig_rax */
|
|
|
|
|
1: movq MCOUNT_REG_SIZE(%rsp), %rdi
|
|
|
|
|
movq %rdi, MCOUNT_REG_SIZE-8(%rsp)
|
|
|
|
|
movq %rax, MCOUNT_REG_SIZE(%rsp)
|
|
|
|
|
|
|
|
|
|
restore_mcount_regs 8
|
|
|
|
|
/* Restore flags */
|
|
|
|
|
popfq
|
2021-01-21 21:29:24 +00:00
|
|
|
UNWIND_HINT_FUNC
|
2020-04-22 16:25:40 +00:00
|
|
|
jmp ftrace_epilogue
|
|
|
|
|
|
2019-10-11 11:51:04 +00:00
|
|
|
SYM_FUNC_END(ftrace_regs_caller)
|
2022-06-03 15:04:44 +00:00
|
|
|
STACK_FRAME_NON_STANDARD_FP(ftrace_regs_caller)
|
2014-05-08 19:21:52 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
#else /* ! CONFIG_DYNAMIC_FTRACE */
|
|
|
|
|
|
2019-10-21 15:18:23 +00:00
|
|
|
SYM_FUNC_START(__fentry__)
|
2014-05-08 19:21:52 +00:00
|
|
|
cmpq $ftrace_stub, ftrace_trace_function
|
|
|
|
|
jnz trace
|
|
|
|
|
|
2019-10-11 11:50:57 +00:00
|
|
|
SYM_INNER_LABEL(ftrace_stub, SYM_L_GLOBAL)
|
2022-03-08 15:30:31 +00:00
|
|
|
ENDBR
|
2021-12-04 13:43:40 +00:00
|
|
|
RET
|
2014-05-08 19:21:52 +00:00
|
|
|
|
|
|
|
|
trace:
|
2014-11-25 02:38:40 +00:00
|
|
|
/* save_mcount_regs fills in first two parameters */
|
|
|
|
|
save_mcount_regs
|
2014-05-08 19:21:52 +00:00
|
|
|
|
2015-11-17 00:43:24 +00:00
|
|
|
/*
|
|
|
|
|
* When DYNAMIC_FTRACE is not defined, ARCH_SUPPORTS_FTRACE_OPS is not
|
|
|
|
|
* set (see include/asm/ftrace.h and include/linux/ftrace.h). Only the
|
|
|
|
|
* ip and parent ip are used and the list function is called when
|
|
|
|
|
* function tracing is enabled.
|
|
|
|
|
*/
|
2018-01-11 21:46:29 +00:00
|
|
|
movq ftrace_trace_function, %r8
|
2020-04-22 15:16:40 +00:00
|
|
|
CALL_NOSPEC r8
|
2014-11-24 16:43:39 +00:00
|
|
|
restore_mcount_regs
|
2014-05-08 19:21:52 +00:00
|
|
|
|
2021-10-08 09:13:31 +00:00
|
|
|
jmp ftrace_stub
|
2019-10-21 15:18:23 +00:00
|
|
|
SYM_FUNC_END(__fentry__)
|
|
|
|
|
EXPORT_SYMBOL(__fentry__)
|
2022-06-03 15:04:44 +00:00
|
|
|
STACK_FRAME_NON_STANDARD_FP(__fentry__)
|
|
|
|
|
|
2014-05-08 19:21:52 +00:00
|
|
|
#endif /* CONFIG_DYNAMIC_FTRACE */
|
|
|
|
|
|
|
|
|
|
#ifdef CONFIG_FUNCTION_GRAPH_TRACER
|
2022-06-03 15:04:44 +00:00
|
|
|
SYM_CODE_START(return_to_handler)
|
|
|
|
|
UNWIND_HINT_EMPTY
|
|
|
|
|
ANNOTATE_NOENDBR
|
2022-03-08 15:30:31 +00:00
|
|
|
subq $16, %rsp
|
2014-05-08 19:21:52 +00:00
|
|
|
|
|
|
|
|
/* Save the return values */
|
|
|
|
|
movq %rax, (%rsp)
|
|
|
|
|
movq %rdx, 8(%rsp)
|
|
|
|
|
movq %rbp, %rdi
|
|
|
|
|
|
|
|
|
|
call ftrace_return_to_handler
|
|
|
|
|
|
|
|
|
|
movq %rax, %rdi
|
|
|
|
|
movq 8(%rsp), %rdx
|
|
|
|
|
movq (%rsp), %rax
|
2022-03-08 15:30:31 +00:00
|
|
|
|
|
|
|
|
addq $16, %rsp
|
|
|
|
|
/*
|
|
|
|
|
* Jump back to the old return address. This cannot be JMP_NOSPEC rdi
|
|
|
|
|
* since IBT would demand that contain ENDBR, which simply isn't so for
|
|
|
|
|
* return addresses. Use a retpoline here to keep the RSB balanced.
|
|
|
|
|
*/
|
|
|
|
|
ANNOTATE_INTRA_FUNCTION_CALL
|
|
|
|
|
call .Ldo_rop
|
|
|
|
|
int3
|
|
|
|
|
.Ldo_rop:
|
|
|
|
|
mov %rdi, (%rsp)
|
|
|
|
|
RET
|
2022-06-03 15:04:44 +00:00
|
|
|
SYM_CODE_END(return_to_handler)
|
2014-05-08 19:21:52 +00:00
|
|
|
#endif
|