mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
synced 2024-11-01 17:08:10 +00:00
d5dd3b4448
commit 4f612ed3f7 upstream.
We have observed that on very large machines with newer CPUs, the static
key/branch switching delay is on the order of milliseconds. This is due
to the required broadcast IPIs, which simply does not scale well to
hundreds of CPUs (cores). If done too frequently, this can adversely
affect tail latencies of various workloads.
One workaround is to increase the sample interval to several seconds,
while decreasing sampled allocation coverage, but the problem still
exists and could still increase tail latencies.
As already noted in the Kconfig help text, there are trade-offs: at
lower sample intervals the dynamic branch results in better performance;
however, at very large sample intervals, the static keys mode can result
in better performance -- careful benchmarking is recommended.
Our initial benchmarking showed that with large enough sample intervals
and workloads stressing the allocator, the static keys mode was slightly
better. Evaluating and observing the possible system-wide side-effects
of the static-key-switching induced broadcast IPIs, however, was a blind
spot (in particular on large machines with 100s of cores).
Therefore, a major downside of the static keys mode is, unfortunately,
that it is hard to predict performance on new system architectures and
topologies, but also making conclusions about performance of new
workloads based on a limited set of benchmarks.
Most distributions will simply select the defaults, while targeting a
large variety of different workloads and system architectures. As such,
the better default is CONFIG_KFENCE_STATIC_KEYS=n, and re-enabling it is
only recommended after careful evaluation.
For reference, on x86-64 the condition in kfence_alloc() generates
exactly
2 instructions in the kmem_cache_alloc() fast-path:
| ...
| cmpl $0x0,0x1a8021c(%rip) # ffffffff82d560d0 <kfence_allocation_gate>
| je ffffffff812d6003 <kmem_cache_alloc+0x243>
| ...
which, given kfence_allocation_gate is infrequently modified, should be
well predicted by most CPUs.
Link: https://lkml.kernel.org/r/20211019102524.2807208-2-elver@google.com
Signed-off-by: Marco Elver <elver@google.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Jann Horn <jannh@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
87 lines
3.2 KiB
Text
87 lines
3.2 KiB
Text
# SPDX-License-Identifier: GPL-2.0-only
|
|
|
|
config HAVE_ARCH_KFENCE
|
|
bool
|
|
|
|
menuconfig KFENCE
|
|
bool "KFENCE: low-overhead sampling-based memory safety error detector"
|
|
depends on HAVE_ARCH_KFENCE && (SLAB || SLUB)
|
|
select STACKTRACE
|
|
select IRQ_WORK
|
|
help
|
|
KFENCE is a low-overhead sampling-based detector of heap out-of-bounds
|
|
access, use-after-free, and invalid-free errors. KFENCE is designed
|
|
to have negligible cost to permit enabling it in production
|
|
environments.
|
|
|
|
See <file:Documentation/dev-tools/kfence.rst> for more details.
|
|
|
|
Note that, KFENCE is not a substitute for explicit testing with tools
|
|
such as KASAN. KFENCE can detect a subset of bugs that KASAN can
|
|
detect, albeit at very different performance profiles. If you can
|
|
afford to use KASAN, continue using KASAN, for example in test
|
|
environments. If your kernel targets production use, and cannot
|
|
enable KASAN due to its cost, consider using KFENCE.
|
|
|
|
if KFENCE
|
|
|
|
config KFENCE_SAMPLE_INTERVAL
|
|
int "Default sample interval in milliseconds"
|
|
default 100
|
|
help
|
|
The KFENCE sample interval determines the frequency with which heap
|
|
allocations will be guarded by KFENCE. May be overridden via boot
|
|
parameter "kfence.sample_interval".
|
|
|
|
Set this to 0 to disable KFENCE by default, in which case only
|
|
setting "kfence.sample_interval" to a non-zero value enables KFENCE.
|
|
|
|
config KFENCE_NUM_OBJECTS
|
|
int "Number of guarded objects available"
|
|
range 1 65535
|
|
default 255
|
|
help
|
|
The number of guarded objects available. For each KFENCE object, 2
|
|
pages are required; with one containing the object and two adjacent
|
|
ones used as guard pages.
|
|
|
|
config KFENCE_STATIC_KEYS
|
|
bool "Use static keys to set up allocations" if EXPERT
|
|
depends on JUMP_LABEL
|
|
help
|
|
Use static keys (static branches) to set up KFENCE allocations. This
|
|
option is only recommended when using very large sample intervals, or
|
|
performance has carefully been evaluated with this option.
|
|
|
|
Using static keys comes with trade-offs that need to be carefully
|
|
evaluated given target workloads and system architectures. Notably,
|
|
enabling and disabling static keys invoke IPI broadcasts, the latency
|
|
and impact of which is much harder to predict than a dynamic branch.
|
|
|
|
Say N if you are unsure.
|
|
|
|
config KFENCE_STRESS_TEST_FAULTS
|
|
int "Stress testing of fault handling and error reporting" if EXPERT
|
|
default 0
|
|
help
|
|
The inverse probability with which to randomly protect KFENCE object
|
|
pages, resulting in spurious use-after-frees. The main purpose of
|
|
this option is to stress test KFENCE with concurrent error reports
|
|
and allocations/frees. A value of 0 disables stress testing logic.
|
|
|
|
Only for KFENCE testing; set to 0 if you are not a KFENCE developer.
|
|
|
|
config KFENCE_KUNIT_TEST
|
|
tristate "KFENCE integration test suite" if !KUNIT_ALL_TESTS
|
|
default KUNIT_ALL_TESTS
|
|
depends on TRACEPOINTS && KUNIT
|
|
help
|
|
Test suite for KFENCE, testing various error detection scenarios with
|
|
various allocation types, and checking that reports are correctly
|
|
output to console.
|
|
|
|
Say Y here if you want the test to be built into the kernel and run
|
|
during boot; say M if you want the test to build as a module; say N
|
|
if you are unsure.
|
|
|
|
endif # KFENCE
|