Alexei Starovoitov e21d2b9235 bpf: Fix fexit trampoline. ... [ Upstream commit e21aa34178 ] The fexit/fmod_ret programs can be attached to kernel functions that can sleep. The synchronize_rcu_tasks() will not wait for such tasks to complete. In such case the trampoline image will be freed and when the task wakes up the return IP will point to freed memory causing the crash. Solve this by adding percpu_ref_get/put for the duration of trampoline and separate trampoline vs its image life times. The "half page" optimization has to be removed, since first_half->second_half->first_half transition cannot be guaranteed to complete in deterministic time. Every trampoline update becomes a new image. The image with fmod_ret or fexit progs will be freed via percpu_ref_kill and call_rcu_tasks. Together they will wait for the original function and trampoline asm to complete. The trampoline is patched from nop to jmp to skip fexit progs. They are freed independently from the trampoline. The image with fentry progs only will be freed via call_rcu_tasks_trace+call_rcu_tasks which will wait for both sleepable and non-sleepable progs to complete. Fixes: fec56f5890 ("bpf: Introduce BPF trampoline") Reported-by: Andrii Nakryiko <andrii@kernel.org> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: Paul E. McKenney <paulmck@kernel.org> # for RCU Link: https://lore.kernel.org/bpf/20210316210007.38949-1-alexei.starovoitov@gmail.com Signed-off-by: Sasha Levin <sashal@kernel.org>		2021-04-07 15:00:03 +02:00
..
bpf	bpf: Fix fexit trampoline.	2021-04-07 15:00:03 +02:00
cgroup	cgroup-v1: add disabled controller check in cgroup1_parse_param()	2021-02-17 11:02:25 +01:00
configs
debug	kgdb: fix to kill breakpoints on initmem after boot	2021-03-04 11:38:46 +01:00
dma	swiotlb: remove the tbl_dma_addr argument to swiotlb_tbl_map_single	2020-11-02 10:10:39 -05:00
entry	x86/entry: Move nmi entry/exit into common code	2021-03-17 17:06:36 +01:00
events	perf/core: Flush PMU internal buffers for per-CPU events	2021-03-17 17:06:34 +01:00
gcov	gcov: fix clang-11+ support	2021-03-30 14:31:54 +02:00
irq	genirq: Disable interrupts for force threaded handlers	2021-03-25 09:04:18 +01:00
kcsan	kcsan: Rewrite kcsan_prandom_u32_max() without prandom_u32_state()	2021-03-04 11:37:37 +01:00
livepatch	kernel/: fix repeated words in comments	2020-10-16 11:11:19 -07:00
locking	locking/lockdep: Avoid unmatched unlock	2021-03-04 11:37:47 +01:00
power	PM: EM: postpone creating the debugfs dir till fs_initcall	2021-03-30 14:32:04 +02:00
printk	printk: fix deadlock when kernel panic	2021-03-04 11:38:41 +01:00
rcu	rcu/nocb: Perform deferred wake up before last idle's need_resched() check	2021-03-04 11:38:35 +01:00
sched	sched/membarrier: fix missing local execution of ipi_sync_rq_state()	2021-03-17 17:06:35 +01:00
time	kernel, fs: Introduce and use set_restart_fn() and arch_set_restart_data()	2021-03-25 09:04:16 +01:00
trace	ftrace: Fix modify_ftrace_direct.	2021-03-30 14:32:00 +02:00
.gitignore
acct.c	kernel: acct.c: fix some kernel-doc nits	2020-10-16 11:11:19 -07:00
async.c
audit.c	audit: Remove redundant null check	2020-08-26 09:10:39 -04:00
audit.h
audit_fsnotify.c	fsnotify: generalize handle_inode_event()	2020-12-30 11:54:18 +01:00
audit_tree.c	fsnotify: generalize handle_inode_event()	2020-12-30 11:54:18 +01:00
audit_watch.c	fsnotify: generalize handle_inode_event()	2020-12-30 11:54:18 +01:00
auditfilter.c
auditsc.c
backtracetest.c
bounds.c
capability.c	LSM: Signal to SafeSetID when setting group IDs	2020-10-13 09:17:34 -07:00
compat.c
configs.c
context_tracking.c
cpu.c	kernel/cpu: add arch override for clear_tasks_mm_cpumask() mm handling	2020-11-27 00:10:39 +11:00
cpu_pm.c	notifier: Fix broken error handling pattern	2020-09-01 09:58:03 +02:00
crash_core.c
crash_dump.c
cred.c
delayacct.c
dma.c
exec_domain.c
exit.c	kernel/io_uring: cancel io_uring before task works	2021-01-30 13:55:18 +01:00
extable.c
fail_function.c	fail_function: Remove a redundant mutex unlock	2020-11-19 11:58:16 -08:00
fork.c	mm/fork: clear PASID for new mm	2021-03-30 14:31:52 +02:00
freezer.c
futex.c	kernel, fs: Introduce and use set_restart_fn() and arch_set_restart_data()	2021-03-25 09:04:16 +01:00
gen_kheaders.sh
groups.c	LSM: Signal to SafeSetID when setting group IDs	2020-10-13 09:17:34 -07:00
hung_task.c	kernel/hung_task.c: make type annotations consistent	2020-11-02 12:14:19 -08:00
iomem.c
irq_work.c
jump_label.c	static_call: Fix static_call_update() sanity check	2021-03-25 09:04:18 +01:00
kallsyms.c	treewide: Convert macro and uses of __section(foo) to __section("foo")	2020-10-25 14:51:49 -07:00
kcmp.c	exec: Transform exec_update_mutex into a rw_semaphore	2021-01-09 13:46:24 +01:00
Kconfig.freezer
Kconfig.hz
Kconfig.locks
Kconfig.preempt
kcov.c
kexec.c	LSM: Introduce kernel_post_load_data() hook	2020-10-05 13:37:03 +02:00
kexec_core.c	kernel: kexec: remove the lock operation of system_transition_mutex	2021-02-03 23:28:37 +01:00
kexec_elf.c
kexec_file.c	ima: Free IMA measurement buffer after kexec syscall	2021-03-04 11:37:50 +01:00
kexec_internal.h
kheaders.c
kmod.c
kprobes.c	kprobes: Fix to delay the kprobes jump optimization	2021-03-04 11:38:35 +01:00
ksysfs.c
kthread.c	kthread: Extract KTHREAD_IS_PER_CPU	2021-02-07 15:37:17 +01:00
latencytop.c
Makefile	kcmp: Support selection of SYS_kcmp without CHECKPOINT_RESTORE	2021-03-04 11:38:41 +01:00
module-internal.h
module.c	module: harden ELF info handling	2021-03-25 09:04:11 +01:00
module_signature.c	module: harden ELF info handling	2021-03-25 09:04:11 +01:00
module_signing.c	module: harden ELF info handling	2021-03-25 09:04:11 +01:00
notifier.c	notifier: Fix broken error handling pattern	2020-09-01 09:58:03 +02:00
nsproxy.c
padata.c	padata: fix possible padata_works_lock deadlock	2020-09-04 17:51:55 +10:00
panic.c	panic: don't dump stack twice on warn	2020-11-14 11:26:04 -08:00
params.c	params: Replace zero-length array with flexible-array member	2020-10-29 17:22:59 -05:00
pid.c	exec: Transform exec_update_mutex into a rw_semaphore	2021-01-09 13:46:24 +01:00
pid_namespace.c	kernel/: fix repeated words in comments	2020-10-16 11:11:19 -07:00
profile.c
ptrace.c	ptrace: Set PF_SUPERPRIV when checking capability	2020-11-17 12:53:22 -08:00
range.c	kernel.h: split out min()/max() et al. helpers	2020-10-16 11:11:19 -07:00
reboot.c	reboot: fix overflow parsing reboot cpu number	2020-11-14 11:26:03 -08:00
regset.c
relay.c	kernel/relay.c: drop unneeded initialization	2020-10-16 11:11:22 -07:00
resource.c	kernel/resource: make iomem_resource implicit in release_mem_region_adjustable()	2020-10-16 11:11:18 -07:00
rseq.c
scftorture.c	scftorture: Add cond_resched() to test loop	2020-08-24 18:38:38 -07:00
scs.c
seccomp.c	seccomp: Add missing return in non-void function	2021-03-04 11:38:32 +01:00
signal.c	ptrace: fix task_join_group_stop() for the case when current is traced	2020-11-02 12:14:19 -08:00
smp.c	smp: Process pending softirqs in flush_smp_call_function_from_idle()	2021-03-04 11:37:51 +01:00
smpboot.c	kthread: Extract KTHREAD_IS_PER_CPU	2021-02-07 15:37:17 +01:00
smpboot.h
softirq.c	softirq: Add debug check to __raise_softirq_irqoff()	2020-09-16 15:18:56 +02:00
stackleak.c	stackleak: let stack_erasing_sysctl take a kernel pointer buffer	2020-09-19 13:13:39 -07:00
stacktrace.c	stacktrace: Remove reliable argument from arch_stack_walk() callback	2020-09-18 14:24:16 +01:00
static_call.c	static_call: Fix static_call_set_init()	2021-03-30 14:31:53 +02:00
stop_machine.c	stop_machine, rcu: Mark functions as notrace	2020-10-26 12:12:27 +01:00
sys.c	kernel/sys.c: fix prototype of prctl_get_tid_address()	2020-10-25 11:44:16 -07:00
sys_ni.c	mm/madvise: introduce process_madvise() syscall: an external memory hinting API	2020-10-18 09:27:10 -07:00
sysctl-test.c
sysctl.c	sysctl.c: fix underflow value setting risk in vm_table	2021-03-17 17:06:25 +01:00
task_work.c	task_work: cleanup notification modes	2020-10-17 15:05:30 -06:00
taskstats.c	taskstats: move specifying netlink policy back to ops	2020-10-02 19:11:12 -07:00
test_kprobes.c
torture.c
tracepoint.c	tracepoint: Do not fail unregistering a probe due to memory failure	2021-03-04 11:38:03 +01:00
tsacct.c
ucount.c
uid16.c
uid16.h
umh.c	usermodehelper: reset umask to default before executing user process	2020-10-06 10:31:52 -07:00
up.c
user-return-notifier.c
user.c
user_namespace.c	kernel/: fix repeated words in comments	2020-10-16 11:11:19 -07:00
usermode_driver.c	bpf: Fix umd memory leak in copy_process()	2021-03-30 14:32:03 +02:00
utsname.c
utsname_sysctl.c
watch_queue.c
watchdog.c	kernel/watchdog: fix watchdog_allowed_mask not used warning	2020-11-14 11:26:03 -08:00
watchdog_hld.c
workqueue.c	workqueue: Restrict affinity change to rescuer	2021-02-07 15:37:17 +01:00
workqueue_internal.h