No description
Find a file
Eduard Zingerman 1ade237119 bpf: Inline calls to bpf_loop when callback is known
Calls to `bpf_loop` are replaced with direct loops to avoid
indirection. E.g. the following:

  bpf_loop(10, foo, NULL, 0);

Is replaced by equivalent of the following:

  for (int i = 0; i < 10; ++i)
    foo(i, NULL);

This transformation could be applied when:
- callback is known and does not change during program execution;
- flags passed to `bpf_loop` are always zero.

Inlining logic works as follows:

- During execution simulation function `update_loop_inline_state`
  tracks the following information for each `bpf_loop` call
  instruction:
  - is callback known and constant?
  - are flags constant and zero?
- Function `optimize_bpf_loop` increases stack depth for functions
  where `bpf_loop` calls can be inlined and invokes `inline_bpf_loop`
  to apply the inlining. The additional stack space is used to spill
  registers R6, R7 and R8. These registers are used as loop counter,
  loop maximal bound and callback context parameter;

Measurements using `benchs/run_bench_bpf_loop.sh` inside QEMU / KVM on
i7-4710HQ CPU show a drop in latency from 14 ns/op to 2 ns/op.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Acked-by: Song Liu <songliubraving@fb.com>
Link: https://lore.kernel.org/r/20220620235344.569325-4-eddyz87@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2022-06-20 17:40:51 -07:00
arch Merge https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next 2022-06-17 19:35:19 -07:00
block block: remove bioset_init_from_src 2022-06-08 14:04:14 -04:00
certs certs: fix and refactor CONFIG_SYSTEM_BLACKLIST_HASH_LIST build 2022-06-15 21:52:32 +03:00
crypto This update includes the following changes: 2022-05-27 18:06:49 -07:00
Documentation Merge https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next 2022-06-17 19:35:19 -07:00
drivers mlxsw: reg: Add support for VLAN RIF as part of RITR register 2022-06-20 10:03:34 +01:00
fs fs.fixes.v5.19-rc3 2022-06-15 09:04:55 -07:00
include bpf: Inline calls to bpf_loop when callback is known 2022-06-20 17:40:51 -07:00
init gcc-12: disable '-Warray-bounds' universally for now 2022-06-09 10:11:12 -07:00
ipc These changes update the ipc sysctls so that they are fundamentally 2022-06-03 15:54:57 -07:00
kernel bpf: Inline calls to bpf_loop when callback is known 2022-06-20 17:40:51 -07:00
lib Random number generator fixes for Linux 5.19-rc2. 2022-06-12 10:33:38 -07:00
LICENSES LICENSES/LGPL-2.1: Add LGPL-2.1-or-later as valid identifiers 2021-12-16 14:33:10 +01:00
mm usercopy: Make usercopy resilient against ridiculously large copies 2022-06-13 09:54:52 -07:00
net skmsg: Get rid of unncessary memset() 2022-06-20 14:05:52 +02:00
samples samples/bpf: Check detach prog exist or not in xdp_fwd 2022-06-15 17:26:20 -07:00
scripts Merge https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next 2022-06-17 19:35:19 -07:00
security KEYS: trusted: tpm2: Fix migratable logic 2022-06-08 14:12:13 +03:00
sound sound fixes for 5.19-rc2 2022-06-10 10:20:57 -07:00
tools selftests/bpf: allow BTF specs and func infos in test_verifier tests 2022-06-20 17:40:51 -07:00
usr Not a lot of material this cycle. Many singleton patches against various 2022-05-27 11:22:03 -07:00
virt KVM: x86: disable preemption around the call to kvm_arch_vcpu_{un|}blocking 2022-06-09 10:52:20 -04:00
.clang-format clang-format: Fix space after for_each macros 2022-05-20 19:27:16 +02:00
.cocciconfig scripts: add Linux .cocciconfig for coccinelle 2016-07-22 12:13:39 +02:00
.get_maintainer.ignore Opt out of scripts/get_maintainer.pl 2019-05-16 10:53:40 -07:00
.gitattributes .gitattributes: use 'dts' diff driver for dts files 2019-12-04 19:44:11 -08:00
.gitignore kbuild: split the second line of *.mod into *.usyms 2022-05-08 03:16:59 +09:00
.mailmap Hot fixes for 5.19-rc1. 2022-06-05 17:05:38 -07:00
COPYING COPYING: state that all contributions really are covered by this file 2020-02-10 13:32:20 -08:00
CREDITS MAINTAINERS: replace a Microchip AT91 maintainer 2022-02-09 11:30:01 +01:00
Kbuild kbuild: rename hostprogs-y/always to hostprogs/always-y 2020-02-04 01:53:07 +09:00
Kconfig kbuild: ensure full rebuild when the compiler is updated 2020-05-12 13:28:33 +09:00
MAINTAINERS Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2022-06-16 20:13:52 -07:00
Makefile Linux 5.19-rc2 2022-06-12 16:11:37 -07:00
README Drop all 00-INDEX files from Documentation/ 2018-09-09 15:08:58 -06:00

Linux kernel
============

There are several guides for kernel developers and users. These guides can
be rendered in a number of formats, like HTML and PDF. Please read
Documentation/admin-guide/README.rst first.

In order to build the documentation, use ``make htmldocs`` or
``make pdfdocs``.  The formatted documentation can also be read online at:

    https://www.kernel.org/doc/html/latest/

There are various text files in the Documentation/ subdirectory,
several of them using the Restructured Text markup notation.

Please read the Documentation/process/changes.rst file, as it contains the
requirements for building and running the kernel, and information about
the problems which may result by upgrading your kernel.