mirrors
/
linux-stable
mirror of https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
1
0
Fork 0
Code Issues Releases Wiki Activity

bpf-next-for-netdev 

-----BEGIN PGP SIGNATURE-----
 
 iHUEABYIAB0WIQTFp0I1jqZrAX+hPRXbK58LschIgwUCZeEKVAAKCRDbK58LschI
 g7oYAQD5Jlv4fIVTvxvfZrTTZ2tU+OsPa75mc8SDKwpash3YygEA8kvESy8+t6pg
 D6QmSf1DIZdFoSp/bV+pfkNWMeR8gwg=
 =mTAj
 -----END PGP SIGNATURE-----

Merge tag 'for-netdev' of https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next

Daniel Borkmann says:

====================
pull-request: bpf-next 2024-02-29

We've added 119 non-merge commits during the last 32 day(s) which contain
a total of 150 files changed, 3589 insertions(+), 995 deletions(-).

The main changes are:

1) Extend the BPF verifier to enable static subprog calls in spin lock
   critical sections, from Kumar Kartikeya Dwivedi.

2) Fix confusing and incorrect inference of PTR_TO_CTX argument type
   in BPF global subprogs, from Andrii Nakryiko.

3) Larger batch of riscv BPF JIT improvements and enabling inlining
   of the bpf_kptr_xchg() for RV64, from Pu Lehui.

4) Allow skeleton users to change the values of the fields in struct_ops
   maps at runtime, from Kui-Feng Lee.

5) Extend the verifier's capabilities of tracking scalars when they
   are spilled to stack, especially when the spill or fill is narrowing,
   from Maxim Mikityanskiy & Eduard Zingerman.

6) Various BPF selftest improvements to fix errors under gcc BPF backend,
   from Jose E. Marchesi.

7) Avoid module loading failure when the module trying to register
   a struct_ops has its BTF section stripped, from Geliang Tang.

8) Annotate all kfuncs in .BTF_ids section which eventually allows
   for automatic kfunc prototype generation from bpftool, from Daniel Xu.

9) Several updates to the instruction-set.rst IETF standardization
   document, from Dave Thaler.

10) Shrink the size of struct bpf_map resp. bpf_array,
    from Alexei Starovoitov.

11) Initial small subset of BPF verifier prepwork for sleepable bpf_timer,
    from Benjamin Tissoires.

12) Fix bpftool to be more portable to musl libc by using POSIX's
    basename(), from Arnaldo Carvalho de Melo.

13) Add libbpf support to gcc in CORE macro definitions,
    from Cupertino Miranda.

14) Remove a duplicate type check in perf_event_bpf_event,
    from Florian Lehner.

15) Fix bpf_spin_{un,}lock BPF helpers to actually annotate them
    with notrace correctly, from Yonghong Song.

16) Replace the deprecated bpf_lpm_trie_key 0-length array with flexible
    array to fix build warnings, from Kees Cook.

17) Fix resolve_btfids cross-compilation to non host-native endianness,
    from Viktor Malik.

* tag 'for-netdev' of https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next: (119 commits)
  selftests/bpf: Test if shadow types work correctly.
  bpftool: Add an example for struct_ops map and shadow type.
  bpftool: Generated shadow variables for struct_ops maps.
  libbpf: Convert st_ops->data to shadow type.
  libbpf: Set btf_value_type_id of struct bpf_map for struct_ops.
  bpf: Replace bpf_lpm_trie_key 0-length array with flexible array
  bpf, arm64: use bpf_prog_pack for memory management
  arm64: patching: implement text_poke API
  bpf, arm64: support exceptions
  arm64: stacktrace: Implement arch_bpf_stack_walk() for the BPF JIT
  bpf: add is_async_callback_calling_insn() helper
  bpf: introduce in_sleepable() helper
  bpf: allow more maps in sleepable bpf programs
  selftests/bpf: Test case for lacking CFI stub functions.
  bpf: Check cfi_stubs before registering a struct_ops type.
  bpf: Clarify batch lookup/lookup_and_delete semantics
  bpf, docs: specify which BPF_ABS and BPF_IND fields were zero
  bpf, docs: Fix typos in instruction-set.rst
  selftests/bpf: update tcp_custom_syncookie to use scalar packet offset
  bpf: Shrink size of struct bpf_map/bpf_array.
  ...
====================

Link: https://lore.kernel.org/r/20240301001625.8800-1-daniel@iogearbox.net
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
This commit is contained in:
Jakub Kicinski 2024-03-02 20:50:59 -08:00
parent e960825709 0270d69121
commit 4b2765ae41
150 changed files with 3585 additions and 991 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
Documentation/bpf/kfuncs.rst
View File

@ -177,10 +177,10 @@ In addition to kfuncs' arguments, verifier may need more information about the
type of kfunc(s) being registered with the BPF subsystem. To do so, we define
flags on a set of kfuncs as follows::
BTF_SET8_START(bpf_task_set)
BTF_KFUNCS_START(bpf_task_set)
BTF_ID_FLAGS(func, bpf_get_task_pid, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_put_pid, KF_RELEASE)
BTF_SET8_END(bpf_task_set)
BTF_KFUNCS_END(bpf_task_set)
This set encodes the BTF ID of each kfunc listed above, and encodes the flags
along with it. Ofcourse, it is also allowed to specify no flags.
@ -347,10 +347,10 @@ Once the kfunc is prepared for use, the final step to making it visible is
registering it with the BPF subsystem. Registration is done per BPF program
type. An example is shown below::
BTF_SET8_START(bpf_task_set)
BTF_KFUNCS_START(bpf_task_set)
BTF_ID_FLAGS(func, bpf_get_task_pid, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_put_pid, KF_RELEASE)
BTF_SET8_END(bpf_task_set)
BTF_KFUNCS_END(bpf_task_set)
static const struct btf_kfunc_id_set bpf_task_kfunc_set = {
.owner = THIS_MODULE,

2
Documentation/bpf/map_lpm_trie.rst
View File

@ -17,7 +17,7 @@ significant byte.
LPM tries may be created with a maximum prefix length that is a multiple
of 8, in the range from 8 to 2048. The key used for lookup and update
operations is a ``struct bpf_lpm_trie_key``, extended by
operations is a ``struct bpf_lpm_trie_key_u8``, extended by
``max_prefixlen/8`` bytes.
- For IPv4 addresses the data length is 4 bytes

155
Documentation/bpf/standardization/instruction-set.rst
View File

@ -1,11 +1,11 @@
.. contents::
.. sectnum::
=======================================
BPF Instruction Set Specification, v1.0
=======================================
======================================
BPF Instruction Set Architecture (ISA)
======================================
This document specifies version 1.0 of the BPF instruction set.
This document specifies the BPF instruction set architecture (ISA).
Documentation conventions
=========================
@ -102,7 +102,7 @@ Conformance groups
An implementation does not need to support all instructions specified in this
document (e.g., deprecated instructions). Instead, a number of conformance
groups are specified. An implementation must support the "basic" conformance
groups are specified. An implementation must support the base32 conformance
group and may support additional conformance groups, where supporting a
conformance group means it must support all instructions in that conformance
group.
@ -112,12 +112,22 @@ that executes instructions, and tools as such compilers that generate
instructions for the runtime. Thus, capability discovery in terms of
conformance groups might be done manually by users or automatically by tools.
Each conformance group has a short ASCII label (e.g., "basic") that
Each conformance group has a short ASCII label (e.g., "base32") that
corresponds to a set of instructions that are mandatory. That is, each
instruction has one or more conformance groups of which it is a member.
The "basic" conformance group includes all instructions defined in this
specification unless otherwise noted.
This document defines the following conformance groups:
* base32: includes all instructions defined in this
specification unless otherwise noted.
* base64: includes base32, plus instructions explicitly noted
as being in the base64 conformance group.
* atomic32: includes 32-bit atomic operation instructions (see `Atomic operations`_).
* atomic64: includes atomic32, plus 64-bit atomic operation instructions.
* divmul32: includes 32-bit division, multiplication, and modulo instructions.
* divmul64: includes divmul32, plus 64-bit division, multiplication,
and modulo instructions.
* legacy: deprecated packet access instructions.
Instruction encoding
====================
@ -166,10 +176,10 @@ Note that most instructions do not use all of the fields.
Unused fields shall be cleared to zero.
As discussed below in `64-bit immediate instructions`_, a 64-bit immediate
instruction uses a 64-bit immediate value that is constructed as follows.
instruction uses two 32-bit immediate values that are constructed as follows.
The 64 bits following the basic instruction contain a pseudo instruction
using the same format but with opcode, dst_reg, src_reg, and offset all set to zero,
and imm containing the high 32 bits of the immediate value.
using the same format but with 'opcode', 'dst_reg', 'src_reg', and 'offset' all
set to zero, and imm containing the high 32 bits of the immediate value.
This is depicted in the following figure::
@ -181,13 +191,8 @@ This is depicted in the following figure::
'--------------'
pseudo instruction
Thus the 64-bit immediate value is constructed as follows:
imm64 = (next_imm << 32) | imm
where 'next_imm' refers to the imm value of the pseudo instruction
following the basic instruction. The unused bytes in the pseudo
instruction are reserved and shall be cleared to zero.
Here, the imm value of the pseudo instruction is called 'next_imm'. The unused
bytes in the pseudo instruction are reserved and shall be cleared to zero.
Instruction classes
-------------------
@ -239,7 +244,8 @@ Arithmetic instructions
-----------------------
``BPF_ALU`` uses 32-bit wide operands while ``BPF_ALU64`` uses 64-bit wide operands for
otherwise identical operations.
otherwise identical operations. ``BPF_ALU64`` instructions belong to the
base64 conformance group unless noted otherwise.
The 'code' field encodes the operation as below, where 'src' and 'dst' refer
to the values of the source and destination registers, respectively.
@ -284,15 +290,19 @@ where '(u32)' indicates that the upper 32 bits are zeroed.
``BPF_XOR | BPF_K | BPF_ALU`` means::
dst = (u32) dst ^ (u32) imm32
dst = (u32) dst ^ (u32) imm
``BPF_XOR | BPF_K | BPF_ALU64`` means::
dst = dst ^ imm32
dst = dst ^ imm
Note that most instructions have instruction offset of 0. Only three instructions
(``BPF_SDIV``, ``BPF_SMOD``, ``BPF_MOVSX``) have a non-zero offset.
Division, multiplication, and modulo operations for ``BPF_ALU`` are part
of the "divmul32" conformance group, and division, multiplication, and
modulo operations for ``BPF_ALU64`` are part of the "divmul64" conformance
group.
The division and modulo operations support both unsigned and signed flavors.
For unsigned operations (``BPF_DIV`` and ``BPF_MOD``), for ``BPF_ALU``,
@ -349,7 +359,9 @@ BPF_ALU64 Reserved 0x00 do byte swap unconditionally
========= ========= ===== =================================================
The 'imm' field encodes the width of the swap operations. The following widths
are supported: 16, 32 and 64.
are supported: 16, 32 and 64. Width 64 operations belong to the base64
conformance group and other swap operations belong to the base32
conformance group.
Examples:
@ -374,31 +386,33 @@ Examples:
Jump instructions
-----------------
``BPF_JMP32`` uses 32-bit wide operands while ``BPF_JMP`` uses 64-bit wide operands for
otherwise identical operations.
``BPF_JMP32`` uses 32-bit wide operands and indicates the base32
conformance group, while ``BPF_JMP`` uses 64-bit wide operands for
otherwise identical operations, and indicates the base64 conformance
group unless otherwise specified.
The 'code' field encodes the operation as below:
======== ===== === =============================== =============================================
code value src description notes
======== ===== === =============================== =============================================
BPF_JA 0x0 0x0 PC += offset BPF_JMP | BPF_K only
BPF_JA 0x0 0x0 PC += imm BPF_JMP32 | BPF_K only
BPF_JEQ 0x1 any PC += offset if dst == src
BPF_JGT 0x2 any PC += offset if dst > src unsigned
BPF_JGE 0x3 any PC += offset if dst >= src unsigned
BPF_JSET 0x4 any PC += offset if dst & src
BPF_JNE 0x5 any PC += offset if dst != src
BPF_JSGT 0x6 any PC += offset if dst > src signed
BPF_JSGE 0x7 any PC += offset if dst >= src signed
BPF_CALL 0x8 0x0 call helper function by address BPF_JMP | BPF_K only, see `Helper functions`_
BPF_CALL 0x8 0x1 call PC += imm BPF_JMP | BPF_K only, see `Program-local functions`_
BPF_CALL 0x8 0x2 call helper function by BTF ID BPF_JMP | BPF_K only, see `Helper functions`_
BPF_EXIT 0x9 0x0 return BPF_JMP | BPF_K only
BPF_JLT 0xa any PC += offset if dst < src unsigned
BPF_JLE 0xb any PC += offset if dst <= src unsigned
BPF_JSLT 0xc any PC += offset if dst < src signed
BPF_JSLE 0xd any PC += offset if dst <= src signed
======== ===== === =============================== =============================================
======== ===== ======= =============================== =============================================
code value src_reg description notes
======== ===== ======= =============================== =============================================
BPF_JA 0x0 0x0 PC += offset BPF_JMP | BPF_K only
BPF_JA 0x0 0x0 PC += imm BPF_JMP32 | BPF_K only
BPF_JEQ 0x1 any PC += offset if dst == src
BPF_JGT 0x2 any PC += offset if dst > src unsigned
BPF_JGE 0x3 any PC += offset if dst >= src unsigned
BPF_JSET 0x4 any PC += offset if dst & src
BPF_JNE 0x5 any PC += offset if dst != src
BPF_JSGT 0x6 any PC += offset if dst > src signed
BPF_JSGE 0x7 any PC += offset if dst >= src signed
BPF_CALL 0x8 0x0 call helper function by address BPF_JMP | BPF_K only, see `Helper functions`_
BPF_CALL 0x8 0x1 call PC += imm BPF_JMP | BPF_K only, see `Program-local functions`_
BPF_CALL 0x8 0x2 call helper function by BTF ID BPF_JMP | BPF_K only, see `Helper functions`_
BPF_EXIT 0x9 0x0 return BPF_JMP | BPF_K only
BPF_JLT 0xa any PC += offset if dst < src unsigned
BPF_JLE 0xb any PC += offset if dst <= src unsigned
BPF_JSLT 0xc any PC += offset if dst < src signed
BPF_JSLE 0xd any PC += offset if dst <= src signed
======== ===== ======= =============================== =============================================
The BPF program needs to store the return value into register R0 before doing a
``BPF_EXIT``.
@ -424,6 +438,9 @@ specified by the 'imm' field. A > 16-bit conditional jump may be
converted to a < 16-bit conditional jump plus a 32-bit unconditional
jump.
All ``BPF_CALL`` and ``BPF_JA`` instructions belong to the
base32 conformance group.
Helper functions
~~~~~~~~~~~~~~~~
@ -481,6 +498,8 @@ The size modifier is one of:
BPF_DW 0x18 double word (8 bytes)
============= ===== =====================
Instructions using ``BPF_DW`` belong to the base64 conformance group.
Regular load and store operations
---------------------------------
@ -493,7 +512,7 @@ instructions that transfer data between a register and memory.
``BPF_MEM | <size> | BPF_ST`` means::
*(size *) (dst + offset) = imm32
*(size *) (dst + offset) = imm
``BPF_MEM | <size> | BPF_LDX`` means::
@ -525,8 +544,10 @@ by other BPF programs or means outside of this specification.
All atomic operations supported by BPF are encoded as store operations
that use the ``BPF_ATOMIC`` mode modifier as follows:
* ``BPF_ATOMIC | BPF_W | BPF_STX`` for 32-bit operations
* ``BPF_ATOMIC | BPF_DW | BPF_STX`` for 64-bit operations
* ``BPF_ATOMIC | BPF_W | BPF_STX`` for 32-bit operations, which are
part of the "atomic32" conformance group.
* ``BPF_ATOMIC | BPF_DW | BPF_STX`` for 64-bit operations, which are
part of the "atomic64" conformance group.
* 8-bit and 16-bit wide atomic operations are not supported.
The 'imm' field is used to encode the actual atomic operation.
@ -547,7 +568,7 @@ BPF_XOR 0xa0 atomic xor
*(u32 *)(dst + offset) += src
``BPF_ATOMIC | BPF_DW | BPF_STX`` with 'imm' = BPF ADD means::
``BPF_ATOMIC | BPF_DW | BPF_STX`` with 'imm' = BPF_ADD means::
*(u64 *)(dst + offset) += src
@ -580,24 +601,24 @@ and loaded back to ``R0``.
-----------------------------
Instructions with the ``BPF_IMM`` 'mode' modifier use the wide instruction
encoding defined in `Instruction encoding`_, and use the 'src' field of the
encoding defined in `Instruction encoding`_, and use the 'src_reg' field of the
basic instruction to hold an opcode subtype.
The following table defines a set of ``BPF_IMM | BPF_DW | BPF_LD`` instructions
with opcode subtypes in the 'src' field, using new terms such as "map"
with opcode subtypes in the 'src_reg' field, using new terms such as "map"
defined further below:
========================= ====== === ========================================= =========== ==============
opcode construction opcode src pseudocode imm type dst type
========================= ====== === ========================================= =========== ==============
BPF_IMM | BPF_DW | BPF_LD 0x18 0x0 dst = imm64 integer integer
BPF_IMM | BPF_DW | BPF_LD 0x18 0x1 dst = map_by_fd(imm) map fd map
BPF_IMM | BPF_DW | BPF_LD 0x18 0x2 dst = map_val(map_by_fd(imm)) + next_imm map fd data pointer
BPF_IMM | BPF_DW | BPF_LD 0x18 0x3 dst = var_addr(imm) variable id data pointer
BPF_IMM | BPF_DW | BPF_LD 0x18 0x4 dst = code_addr(imm) integer code pointer
BPF_IMM | BPF_DW | BPF_LD 0x18 0x5 dst = map_by_idx(imm) map index map
BPF_IMM | BPF_DW | BPF_LD 0x18 0x6 dst = map_val(map_by_idx(imm)) + next_imm map index data pointer
========================= ====== === ========================================= =========== ==============
========================= ====== ======= ========================================= =========== ==============
opcode construction opcode src_reg pseudocode imm type dst type
========================= ====== ======= ========================================= =========== ==============
BPF_IMM | BPF_DW | BPF_LD 0x18 0x0 dst = (next_imm << 32) | imm integer integer
BPF_IMM | BPF_DW | BPF_LD 0x18 0x1 dst = map_by_fd(imm) map fd map
BPF_IMM | BPF_DW | BPF_LD 0x18 0x2 dst = map_val(map_by_fd(imm)) + next_imm map fd data pointer
BPF_IMM | BPF_DW | BPF_LD 0x18 0x3 dst = var_addr(imm) variable id data pointer
BPF_IMM | BPF_DW | BPF_LD 0x18 0x4 dst = code_addr(imm) integer code pointer
BPF_IMM | BPF_DW | BPF_LD 0x18 0x5 dst = map_by_idx(imm) map index map
BPF_IMM | BPF_DW | BPF_LD 0x18 0x6 dst = map_val(map_by_idx(imm)) + next_imm map index data pointer
========================= ====== ======= ========================================= =========== ==============
where
@ -635,7 +656,9 @@ Legacy BPF Packet access instructions
-------------------------------------
BPF previously introduced special instructions for access to packet data that were
carried over from classic BPF. However, these instructions are
deprecated and should no longer be used. All legacy packet access
instructions belong to the "legacy" conformance group instead of the "basic"
conformance group.
carried over from classic BPF. These instructions used an instruction
class of BPF_LD, a size modifier of BPF_W, BPF_H, or BPF_B, and a
mode modifier of BPF_ABS or BPF_IND. The 'dst_reg' and 'offset' fields were
set to zero, and 'src_reg' was set to zero for BPF_ABS. However, these
instructions are deprecated and should no longer be used. All legacy packet
access instructions belong to the "legacy" conformance group.

31
Documentation/networking/af_xdp.rst
View File

@ -329,23 +329,24 @@ XDP_SHARED_UMEM option and provide the initial socket's fd in the
sxdp_shared_umem_fd field as you registered the UMEM on that
socket. These two sockets will now share one and the same UMEM.
There is no need to supply an XDP program like the one in the previous
case where sockets were bound to the same queue id and
device. Instead, use the NIC's packet steering capabilities to steer
the packets to the right queue. In the previous example, there is only
one queue shared among sockets, so the NIC cannot do this steering. It
can only steer between queues.
In this case, it is possible to use the NIC's packet steering
capabilities to steer the packets to the right queue. This is not
possible in the previous example as there is only one queue shared
among sockets, so the NIC cannot do this steering as it can only steer
between queues.
In libbpf, you need to use the xsk_socket__create_shared() API as it
takes a reference to a FILL ring and a COMPLETION ring that will be
created for you and bound to the shared UMEM. You can use this
function for all the sockets you create, or you can use it for the
second and following ones and use xsk_socket__create() for the first
one. Both methods yield the same result.
In libxdp (or libbpf prior to version 1.0), you need to use the
xsk_socket__create_shared() API as it takes a reference to a FILL ring
and a COMPLETION ring that will be created for you and bound to the
shared UMEM. You can use this function for all the sockets you create,
or you can use it for the second and following ones and use
xsk_socket__create() for the first one. Both methods yield the same
result.
Note that a UMEM can be shared between sockets on the same queue id
and device, as well as between queues on the same device and between
devices at the same time.
devices at the same time. It is also possible to redirect to any
socket as long as it is bound to the same umem with XDP_SHARED_UMEM.
XDP_USE_NEED_WAKEUP bind flag
-----------------------------
@ -822,6 +823,10 @@ A: The short answer is no, that is not supported at the moment. The
switch, or other distribution mechanism, in your NIC to direct
traffic to the correct queue id and socket.
Note that if you are using the XDP_SHARED_UMEM option, it is
possible to switch traffic between any socket bound to the same
umem.
Q: My packets are sometimes corrupted. What is wrong?
A: Care has to be taken not to feed the same buffer in the UMEM into

2
arch/arm64/include/asm/patching.h
View File

@ -8,6 +8,8 @@ int aarch64_insn_read(void *addr, u32 *insnp);
int aarch64_insn_write(void *addr, u32 insn);
int aarch64_insn_write_literal_u64(void *addr, u64 val);
void *aarch64_insn_set(void *dst, u32 insn, size_t len);
void *aarch64_insn_copy(void *dst, void *src, size_t len);
int aarch64_insn_patch_text_nosync(void *addr, u32 insn);
int aarch64_insn_patch_text(void *addrs[], u32 insns[], int cnt);

75
arch/arm64/kernel/patching.c
View File

@ -105,6 +105,81 @@ noinstr int aarch64_insn_write_literal_u64(void *addr, u64 val)
return ret;
}
typedef void text_poke_f(void *dst, void *src, size_t patched, size_t len);
static void *__text_poke(text_poke_f func, void *addr, void *src, size_t len)
{
unsigned long flags;
size_t patched = 0;
size_t size;
void *waddr;
void *ptr;
raw_spin_lock_irqsave(&patch_lock, flags);
while (patched < len) {
ptr = addr + patched;
size = min_t(size_t, PAGE_SIZE - offset_in_page(ptr),
len - patched);
waddr = patch_map(ptr, FIX_TEXT_POKE0);
func(waddr, src, patched, size);
patch_unmap(FIX_TEXT_POKE0);
patched += size;
}
raw_spin_unlock_irqrestore(&patch_lock, flags);
flush_icache_range((uintptr_t)addr, (uintptr_t)addr + len);
return addr;
}
static void text_poke_memcpy(void *dst, void *src, size_t patched, size_t len)
{
copy_to_kernel_nofault(dst, src + patched, len);
}
static void text_poke_memset(void *dst, void *src, size_t patched, size_t len)
{
u32 c = *(u32 *)src;
memset32(dst, c, len / 4);
}
/**
* aarch64_insn_copy - Copy instructions into (an unused part of) RX memory
* @dst: address to modify
* @src: source of the copy
* @len: length to copy
*
* Useful for JITs to dump new code blocks into unused regions of RX memory.
*/
noinstr void *aarch64_insn_copy(void *dst, void *src, size_t len)
{
/* A64 instructions must be word aligned */
if ((uintptr_t)dst & 0x3)
return NULL;
return __text_poke(text_poke_memcpy, dst, src, len);
}
/**
* aarch64_insn_set - memset for RX memory regions.
* @dst: address to modify
* @insn: value to set
* @len: length of memory region.
*
* Useful for JITs to fill regions of RX memory with illegal instructions.
*/
noinstr void *aarch64_insn_set(void *dst, u32 insn, size_t len)
{
if ((uintptr_t)dst & 0x3)
return NULL;
return __text_poke(text_poke_memset, dst, &insn, len);
}
int __kprobes aarch64_insn_patch_text_nosync(void *addr, u32 insn)
{
u32 *tp = addr;

26
arch/arm64/kernel/stacktrace.c
View File

@ -7,6 +7,7 @@
#include <linux/kernel.h>
#include <linux/efi.h>
#include <linux/export.h>
#include <linux/filter.h>
#include <linux/ftrace.h>
#include <linux/kprobes.h>
#include <linux/sched.h>
@ -266,6 +267,31 @@ noinline noinstr void arch_stack_walk(stack_trace_consume_fn consume_entry,
kunwind_stack_walk(arch_kunwind_consume_entry, &data, task, regs);
}
struct bpf_unwind_consume_entry_data {
bool (*consume_entry)(void *cookie, u64 ip, u64 sp, u64 fp);
void *cookie;
};
static bool
arch_bpf_unwind_consume_entry(const struct kunwind_state *state, void *cookie)
{
struct bpf_unwind_consume_entry_data *data = cookie;
return data->consume_entry(data->cookie, state->common.pc, 0,
state->common.fp);
}
noinline noinstr void arch_bpf_stack_walk(bool (*consume_entry)(void *cookie, u64 ip, u64 sp,
u64 fp), void *cookie)
{
struct bpf_unwind_consume_entry_data data = {
.consume_entry = consume_entry,
.cookie = cookie,
};
kunwind_stack_walk(arch_bpf_unwind_consume_entry, &data, current, NULL);
}
static bool dump_backtrace_entry(void *arg, unsigned long where)
{
char *loglvl = arg;

224
arch/arm64/net/bpf_jit_comp.c
View File

@ -76,6 +76,7 @@ struct jit_ctx {
int *offset;
int exentry_idx;
__le32 *image;
__le32 *ro_image;
u32 stack_size;
int fpb_offset;
};
@ -205,6 +206,14 @@ static void jit_fill_hole(void *area, unsigned int size)
*ptr++ = cpu_to_le32(AARCH64_BREAK_FAULT);
}
int bpf_arch_text_invalidate(void *dst, size_t len)
{
if (!aarch64_insn_set(dst, AARCH64_BREAK_FAULT, len))
return -EINVAL;
return 0;
}
static inline int epilogue_offset(const struct jit_ctx *ctx)
{
int to = ctx->epilogue_offset;
@ -285,7 +294,8 @@ static bool is_lsi_offset(int offset, int scale)
/* Tail call offset to jump into */
#define PROLOGUE_OFFSET (BTI_INSNS + 2 + PAC_INSNS + 8)
static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf,
bool is_exception_cb)
{
const struct bpf_prog *prog = ctx->prog;
const bool is_main_prog = !bpf_is_subprog(prog);
@ -333,19 +343,34 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
emit(A64_MOV(1, A64_R(9), A64_LR), ctx);
emit(A64_NOP, ctx);
/* Sign lr */
if (IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL))
emit(A64_PACIASP, ctx);
if (!is_exception_cb) {
/* Sign lr */
if (IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL))
emit(A64_PACIASP, ctx);
/* Save FP and LR registers to stay align with ARM64 AAPCS */
emit(A64_PUSH(A64_FP, A64_LR, A64_SP), ctx);
emit(A64_MOV(1, A64_FP, A64_SP), ctx);
/* Save FP and LR registers to stay align with ARM64 AAPCS */
emit(A64_PUSH(A64_FP, A64_LR, A64_SP), ctx);
emit(A64_MOV(1, A64_FP, A64_SP), ctx);
/* Save callee-saved registers */
emit(A64_PUSH(r6, r7, A64_SP), ctx);
emit(A64_PUSH(r8, r9, A64_SP), ctx);
emit(A64_PUSH(fp, tcc, A64_SP), ctx);
emit(A64_PUSH(fpb, A64_R(28), A64_SP), ctx);
/* Save callee-saved registers */
emit(A64_PUSH(r6, r7, A64_SP), ctx);
emit(A64_PUSH(r8, r9, A64_SP), ctx);
emit(A64_PUSH(fp, tcc, A64_SP), ctx);
emit(A64_PUSH(fpb, A64_R(28), A64_SP), ctx);
} else {
/*
* Exception callback receives FP of Main Program as third
* parameter
*/
emit(A64_MOV(1, A64_FP, A64_R(2)), ctx);
/*
* Main Program already pushed the frame record and the
* callee-saved registers. The exception callback will not push
* anything and re-use the main program's stack.
*
* 10 registers are on the stack
*/
emit(A64_SUB_I(1, A64_SP, A64_FP, 80), ctx);
}
/* Set up BPF prog stack base register */
emit(A64_MOV(1, fp, A64_SP), ctx);
@ -365,6 +390,20 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
emit_bti(A64_BTI_J, ctx);
}
/*
* Program acting as exception boundary should save all ARM64
* Callee-saved registers as the exception callback needs to recover
* all ARM64 Callee-saved registers in its epilogue.
*/
if (prog->aux->exception_boundary) {
/*
* As we are pushing two more registers, BPF_FP should be moved
* 16 bytes
*/
emit(A64_SUB_I(1, fp, fp, 16), ctx);
emit(A64_PUSH(A64_R(23), A64_R(24), A64_SP), ctx);
}
emit(A64_SUB_I(1, fpb, fp, ctx->fpb_offset), ctx);
/* Stack must be multiples of 16B */
@ -653,7 +692,7 @@ static void build_plt(struct jit_ctx *ctx)
plt->target = (u64)&dummy_tramp;
}
static void build_epilogue(struct jit_ctx *ctx)
static void build_epilogue(struct jit_ctx *ctx, bool is_exception_cb)
{
const u8 r0 = bpf2a64[BPF_REG_0];
const u8 r6 = bpf2a64[BPF_REG_6];
@ -666,6 +705,15 @@ static void build_epilogue(struct jit_ctx *ctx)
/* We're done with BPF stack */
emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_size), ctx);
/*
* Program acting as exception boundary pushes R23 and R24 in addition
* to BPF callee-saved registers. Exception callback uses the boundary
* program's stack frame, so recover these extra registers in the above
* two cases.
*/
if (ctx->prog->aux->exception_boundary || is_exception_cb)
emit(A64_POP(A64_R(23), A64_R(24), A64_SP), ctx);
/* Restore x27 and x28 */
emit(A64_POP(fpb, A64_R(28), A64_SP), ctx);
/* Restore fs (x25) and x26 */
@ -707,7 +755,8 @@ static int add_exception_handler(const struct bpf_insn *insn,
struct jit_ctx *ctx,
int dst_reg)
{
off_t offset;
off_t ins_offset;
off_t fixup_offset;
unsigned long pc;
struct exception_table_entry *ex;
@ -724,12 +773,17 @@ static int add_exception_handler(const struct bpf_insn *insn,
return -EINVAL;
ex = &ctx->prog->aux->extable[ctx->exentry_idx];
pc = (unsigned long)&ctx->image[ctx->idx - 1];
pc = (unsigned long)&ctx->ro_image[ctx->idx - 1];
offset = pc - (long)&ex->insn;
if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN))
/*
* This is the relative offset of the instruction that may fault from
* the exception table itself. This will be written to the exception
* table and if this instruction faults, the destination register will
* be set to '0' and the execution will jump to the next instruction.
*/
ins_offset = pc - (long)&ex->insn;
if (WARN_ON_ONCE(ins_offset >= 0 || ins_offset < INT_MIN))
return -ERANGE;
ex->insn = offset;
/*
* Since the extable follows the program, the fixup offset is always
@ -738,12 +792,25 @@ static int add_exception_handler(const struct bpf_insn *insn,
* bits. We don't need to worry about buildtime or runtime sort
* modifying the upper bits because the table is already sorted, and
* isn't part of the main exception table.
*
* The fixup_offset is set to the next instruction from the instruction
* that may fault. The execution will jump to this after handling the
* fault.
*/
offset = (long)&ex->fixup - (pc + AARCH64_INSN_SIZE);
if (!FIELD_FIT(BPF_FIXUP_OFFSET_MASK, offset))
fixup_offset = (long)&ex->fixup - (pc + AARCH64_INSN_SIZE);
if (!FIELD_FIT(BPF_FIXUP_OFFSET_MASK, fixup_offset))
return -ERANGE;
ex->fixup = FIELD_PREP(BPF_FIXUP_OFFSET_MASK, offset) |
/*
* The offsets above have been calculated using the RO buffer but we
* need to use the R/W buffer for writes.
* switch ex to rw buffer for writing.
*/
ex = (void *)ctx->image + ((void *)ex - (void *)ctx->ro_image);
ex->insn = ins_offset;
ex->fixup = FIELD_PREP(BPF_FIXUP_OFFSET_MASK, fixup_offset) |
FIELD_PREP(BPF_FIXUP_REG_MASK, dst_reg);
ex->type = EX_TYPE_BPF;
@ -1511,7 +1578,8 @@ static inline void bpf_flush_icache(void *start, void *end)
struct arm64_jit_data {
struct bpf_binary_header *header;
u8 *image;
u8 *ro_image;
struct bpf_binary_header *ro_header;
struct jit_ctx ctx;
};
@ -1520,12 +1588,14 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
int image_size, prog_size, extable_size, extable_align, extable_offset;
struct bpf_prog *tmp, *orig_prog = prog;
struct bpf_binary_header *header;
struct bpf_binary_header *ro_header;
struct arm64_jit_data *jit_data;
bool was_classic = bpf_prog_was_classic(prog);
bool tmp_blinded = false;
bool extra_pass = false;
struct jit_ctx ctx;
u8 *image_ptr;
u8 *ro_image_ptr;
if (!prog->jit_requested)
return orig_prog;
@ -1552,8 +1622,11 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
}
if (jit_data->ctx.offset) {
ctx = jit_data->ctx;
image_ptr = jit_data->image;
ro_image_ptr = jit_data->ro_image;
ro_header = jit_data->ro_header;
header = jit_data->header;
image_ptr = (void *)header + ((void *)ro_image_ptr
- (void *)ro_header);
extra_pass = true;
prog_size = sizeof(u32) * ctx.idx;
goto skip_init_ctx;
@ -1575,7 +1648,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
* BPF line info needs ctx->offset[i] to be the offset of
* instruction[i] in jited image, so build prologue first.
*/
if (build_prologue(&ctx, was_classic)) {
if (build_prologue(&ctx, was_classic, prog->aux->exception_cb)) {
prog = orig_prog;
goto out_off;
}
@ -1586,7 +1659,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
}
ctx.epilogue_offset = ctx.idx;
build_epilogue(&ctx);
build_epilogue(&ctx, prog->aux->exception_cb);
build_plt(&ctx);
extable_align = __alignof__(struct exception_table_entry);
@ -1598,63 +1671,81 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
/* also allocate space for plt target */
extable_offset = round_up(prog_size + PLT_TARGET_SIZE, extable_align);
image_size = extable_offset + extable_size;
header = bpf_jit_binary_alloc(image_size, &image_ptr,
sizeof(u32), jit_fill_hole);
if (header == NULL) {
ro_header = bpf_jit_binary_pack_alloc(image_size, &ro_image_ptr,
sizeof(u32), &header, &image_ptr,
jit_fill_hole);
if (!ro_header) {
prog = orig_prog;
goto out_off;
}
/* 2. Now, the actual pass. */
/*
* Use the image(RW) for writing the JITed instructions. But also save
* the ro_image(RX) for calculating the offsets in the image. The RW
* image will be later copied to the RX image from where the program
* will run. The bpf_jit_binary_pack_finalize() will do this copy in the
* final step.
*/
ctx.image = (__le32 *)image_ptr;
ctx.ro_image = (__le32 *)ro_image_ptr;
if (extable_size)
prog->aux->extable = (void *)image_ptr + extable_offset;
prog->aux->extable = (void *)ro_image_ptr + extable_offset;
skip_init_ctx:
ctx.idx = 0;
ctx.exentry_idx = 0;
build_prologue(&ctx, was_classic);
build_prologue(&ctx, was_classic, prog->aux->exception_cb);
if (build_body(&ctx, extra_pass)) {
bpf_jit_binary_free(header);
prog = orig_prog;
goto out_off;
goto out_free_hdr;
}
build_epilogue(&ctx);
build_epilogue(&ctx, prog->aux->exception_cb);
build_plt(&ctx);
/* 3. Extra pass to validate JITed code. */
if (validate_ctx(&ctx)) {
bpf_jit_binary_free(header);
prog = orig_prog;
goto out_off;
goto out_free_hdr;
}
/* And we're done. */
if (bpf_jit_enable > 1)
bpf_jit_dump(prog->len, prog_size, 2, ctx.image);
bpf_flush_icache(header, ctx.image + ctx.idx);
if (!prog->is_func || extra_pass) {
if (extra_pass && ctx.idx != jit_data->ctx.idx) {
pr_err_once("multi-func JIT bug %d != %d\n",
ctx.idx, jit_data->ctx.idx);
bpf_jit_binary_free(header);
prog->bpf_func = NULL;
prog->jited = 0;
prog->jited_len = 0;
goto out_free_hdr;
}
if (WARN_ON(bpf_jit_binary_pack_finalize(prog, ro_header,
header))) {
/* ro_header has been freed */
ro_header = NULL;
prog = orig_prog;
goto out_off;
}
bpf_jit_binary_lock_ro(header);
/*
* The instructions have now been copied to the ROX region from
* where they will execute. Now the data cache has to be cleaned to
* the PoU and the I-cache has to be invalidated for the VAs.
*/
bpf_flush_icache(ro_header, ctx.ro_image + ctx.idx);
} else {
jit_data->ctx = ctx;
jit_data->image = image_ptr;
jit_data->ro_image = ro_image_ptr;
jit_data->header = header;
jit_data->ro_header = ro_header;
}
prog->bpf_func = (void *)ctx.image;
prog->bpf_func = (void *)ctx.ro_image;
prog->jited = 1;
prog->jited_len = prog_size;
@ -1675,6 +1766,14 @@ out:
bpf_jit_prog_release_other(prog, prog == orig_prog ?
tmp : orig_prog);
return prog;
out_free_hdr:
if (header) {
bpf_arch_text_copy(&ro_header->size, &header->size,
sizeof(header->size));
bpf_jit_binary_pack_free(ro_header, header);
}
goto out_off;
}
bool bpf_jit_supports_kfunc_call(void)
@ -1682,6 +1781,13 @@ bool bpf_jit_supports_kfunc_call(void)
return true;
}
void *bpf_arch_text_copy(void *dst, void *src, size_t len)
{
if (!aarch64_insn_copy(dst, src, len))
return ERR_PTR(-EINVAL);
return dst;
}
u64 bpf_jit_alloc_exec_limit(void)
{
return VMALLOC_END - VMALLOC_START;
@ -2310,3 +2416,37 @@ bool bpf_jit_supports_ptr_xchg(void)
{
return true;
}
bool bpf_jit_supports_exceptions(void)
{
/* We unwind through both kernel frames starting from within bpf_throw
* call and BPF frames. Therefore we require FP unwinder to be enabled
* to walk kernel frames and reach BPF frames in the stack trace.
* ARM64 kernel is aways compiled with CONFIG_FRAME_POINTER=y
*/
return true;
}
void bpf_jit_free(struct bpf_prog *prog)
{
if (prog->jited) {
struct arm64_jit_data *jit_data = prog->aux->jit_data;
struct bpf_binary_header *hdr;
/*
* If we fail the final pass of JIT (from jit_subprogs),
* the program may not be finalized yet. Call finalize here
* before freeing it.
*/
if (jit_data) {
bpf_arch_text_copy(&jit_data->ro_header->size, &jit_data->header->size,
sizeof(jit_data->header->size));
kfree(jit_data);
}
hdr = bpf_jit_binary_pack_hdr(prog);
bpf_jit_binary_pack_free(hdr, NULL);
WARN_ON_ONCE(!bpf_prog_kallsyms_verify_off(prog));
}
bpf_prog_unlock_free(prog);
}

134
arch/riscv/net/bpf_jit.h
View File

@ -18,6 +18,11 @@ static inline bool rvc_enabled(void)
return IS_ENABLED(CONFIG_RISCV_ISA_C);
}
static inline bool rvzbb_enabled(void)
{
return IS_ENABLED(CONFIG_RISCV_ISA_ZBB) && riscv_has_extension_likely(RISCV_ISA_EXT_ZBB);
}
enum {
RV_REG_ZERO = 0, /* The constant value 0 */
RV_REG_RA = 1, /* Return address */
@ -730,6 +735,33 @@ static inline u16 rvc_swsp(u32 imm8, u8 rs2)
return rv_css_insn(0x6, imm, rs2, 0x2);
}
/* RVZBB instrutions. */
static inline u32 rvzbb_sextb(u8 rd, u8 rs1)
{
return rv_i_insn(0x604, rs1, 1, rd, 0x13);
}
static inline u32 rvzbb_sexth(u8 rd, u8 rs1)
{
return rv_i_insn(0x605, rs1, 1, rd, 0x13);
}
static inline u32 rvzbb_zexth(u8 rd, u8 rs)
{
if (IS_ENABLED(CONFIG_64BIT))
return rv_i_insn(0x80, rs, 4, rd, 0x3b);
return rv_i_insn(0x80, rs, 4, rd, 0x33);
}
static inline u32 rvzbb_rev8(u8 rd, u8 rs)
{
if (IS_ENABLED(CONFIG_64BIT))
return rv_i_insn(0x6b8, rs, 5, rd, 0x13);
return rv_i_insn(0x698, rs, 5, rd, 0x13);
}
/*
* RV64-only instructions.
*
@ -1087,6 +1119,108 @@ static inline void emit_subw(u8 rd, u8 rs1, u8 rs2, struct rv_jit_context *ctx)
emit(rv_subw(rd, rs1, rs2), ctx);
}
static inline void emit_sextb(u8 rd, u8 rs, struct rv_jit_context *ctx)
{
if (rvzbb_enabled()) {
emit(rvzbb_sextb(rd, rs), ctx);
return;
}
emit_slli(rd, rs, 56, ctx);
emit_srai(rd, rd, 56, ctx);
}
static inline void emit_sexth(u8 rd, u8 rs, struct rv_jit_context *ctx)
{
if (rvzbb_enabled()) {
emit(rvzbb_sexth(rd, rs), ctx);
return;
}
emit_slli(rd, rs, 48, ctx);
emit_srai(rd, rd, 48, ctx);
}
static inline void emit_sextw(u8 rd, u8 rs, struct rv_jit_context *ctx)
{
emit_addiw(rd, rs, 0, ctx);
}
static inline void emit_zexth(u8 rd, u8 rs, struct rv_jit_context *ctx)
{
if (rvzbb_enabled()) {
emit(rvzbb_zexth(rd, rs), ctx);
return;
}
emit_slli(rd, rs, 48, ctx);
emit_srli(rd, rd, 48, ctx);
}
static inline void emit_zextw(u8 rd, u8 rs, struct rv_jit_context *ctx)
{
emit_slli(rd, rs, 32, ctx);
emit_srli(rd, rd, 32, ctx);
}
static inline void emit_bswap(u8 rd, s32 imm, struct rv_jit_context *ctx)
{
if (rvzbb_enabled()) {
int bits = 64 - imm;
emit(rvzbb_rev8(rd, rd), ctx);
if (bits)
emit_srli(rd, rd, bits, ctx);
return;
}
emit_li(RV_REG_T2, 0, ctx);
emit_andi(RV_REG_T1, rd, 0xff, ctx);
emit_add(RV_REG_T2, RV_REG_T2, RV_REG_T1, ctx);
emit_slli(RV_REG_T2, RV_REG_T2, 8, ctx);
emit_srli(rd, rd, 8, ctx);
if (imm == 16)
goto out_be;
emit_andi(RV_REG_T1, rd, 0xff, ctx);
emit_add(RV_REG_T2, RV_REG_T2, RV_REG_T1, ctx);
emit_slli(RV_REG_T2, RV_REG_T2, 8, ctx);
emit_srli(rd, rd, 8, ctx);
emit_andi(RV_REG_T1, rd, 0xff, ctx);
emit_add(RV_REG_T2, RV_REG_T2, RV_REG_T1, ctx);
emit_slli(RV_REG_T2, RV_REG_T2, 8, ctx);
emit_srli(rd, rd, 8, ctx);
if (imm == 32)
goto out_be;
emit_andi(RV_REG_T1, rd, 0xff, ctx);
emit_add(RV_REG_T2, RV_REG_T2, RV_REG_T1, ctx);
emit_slli(RV_REG_T2, RV_REG_T2, 8, ctx);
emit_srli(rd, rd, 8, ctx);
emit_andi(RV_REG_T1, rd, 0xff, ctx);
emit_add(RV_REG_T2, RV_REG_T2, RV_REG_T1, ctx);
emit_slli(RV_REG_T2, RV_REG_T2, 8, ctx);
emit_srli(rd, rd, 8, ctx);
emit_andi(RV_REG_T1, rd, 0xff, ctx);
emit_add(RV_REG_T2, RV_REG_T2, RV_REG_T1, ctx);
emit_slli(RV_REG_T2, RV_REG_T2, 8, ctx);
emit_srli(rd, rd, 8, ctx);
emit_andi(RV_REG_T1, rd, 0xff, ctx);
emit_add(RV_REG_T2, RV_REG_T2, RV_REG_T1, ctx);
emit_slli(RV_REG_T2, RV_REG_T2, 8, ctx);
emit_srli(rd, rd, 8, ctx);
out_be:
emit_andi(RV_REG_T1, rd, 0xff, ctx);
emit_add(RV_REG_T2, RV_REG_T2, RV_REG_T1, ctx);
emit_mv(rd, RV_REG_T2, ctx);
}
#endif /* __riscv_xlen == 64 */
void bpf_jit_build_prologue(struct rv_jit_context *ctx);

215
arch/riscv/net/bpf_jit_comp64.c
View File

@ -141,6 +141,19 @@ static bool in_auipc_jalr_range(s64 val)
val < ((1L << 31) - (1L << 11));
}
/* Modify rd pointer to alternate reg to avoid corrupting original reg */
static void emit_sextw_alt(u8 *rd, u8 ra, struct rv_jit_context *ctx)
{
emit_sextw(ra, *rd, ctx);
*rd = ra;
}
static void emit_zextw_alt(u8 *rd, u8 ra, struct rv_jit_context *ctx)
{
emit_zextw(ra, *rd, ctx);
*rd = ra;
}
/* Emit fixed-length instructions for address */
static int emit_addr(u8 rd, u64 addr, bool extra_pass, struct rv_jit_context *ctx)
{
@ -326,12 +339,6 @@ static void emit_branch(u8 cond, u8 rd, u8 rs, int rvoff,
emit(rv_jalr(RV_REG_ZERO, RV_REG_T1, lower), ctx);
}
static void emit_zext_32(u8 reg, struct rv_jit_context *ctx)
{
emit_slli(reg, reg, 32, ctx);
emit_srli(reg, reg, 32, ctx);
}
static int emit_bpf_tail_call(int insn, struct rv_jit_context *ctx)
{
int tc_ninsn, off, start_insn = ctx->ninsns;
@ -346,7 +353,7 @@ static int emit_bpf_tail_call(int insn, struct rv_jit_context *ctx)
*/
tc_ninsn = insn ? ctx->offset[insn] - ctx->offset[insn - 1] :
ctx->offset[0];
emit_zext_32(RV_REG_A2, ctx);
emit_zextw(RV_REG_A2, RV_REG_A2, ctx);
off = offsetof(struct bpf_array, map.max_entries);
if (is_12b_check(off, insn))
@ -405,38 +412,6 @@ static void init_regs(u8 *rd, u8 *rs, const struct bpf_insn *insn,
*rs = bpf_to_rv_reg(insn->src_reg, ctx);
}
static void emit_zext_32_rd_rs(u8 *rd, u8 *rs, struct rv_jit_context *ctx)
{
emit_mv(RV_REG_T2, *rd, ctx);
emit_zext_32(RV_REG_T2, ctx);
emit_mv(RV_REG_T1, *rs, ctx);
emit_zext_32(RV_REG_T1, ctx);
*rd = RV_REG_T2;
*rs = RV_REG_T1;
}
static void emit_sext_32_rd_rs(u8 *rd, u8 *rs, struct rv_jit_context *ctx)
{
emit_addiw(RV_REG_T2, *rd, 0, ctx);
emit_addiw(RV_REG_T1, *rs, 0, ctx);
*rd = RV_REG_T2;
*rs = RV_REG_T1;
}
static void emit_zext_32_rd_t1(u8 *rd, struct rv_jit_context *ctx)
{
emit_mv(RV_REG_T2, *rd, ctx);
emit_zext_32(RV_REG_T2, ctx);
emit_zext_32(RV_REG_T1, ctx);
*rd = RV_REG_T2;
}
static void emit_sext_32_rd(u8 *rd, struct rv_jit_context *ctx)
{
emit_addiw(RV_REG_T2, *rd, 0, ctx);
*rd = RV_REG_T2;
}
static int emit_jump_and_link(u8 rd, s64 rvoff, bool fixed_addr,
struct rv_jit_context *ctx)
{
@ -519,32 +494,32 @@ static void emit_atomic(u8 rd, u8 rs, s16 off, s32 imm, bool is64,
emit(is64 ? rv_amoadd_d(rs, rs, rd, 0, 0) :
rv_amoadd_w(rs, rs, rd, 0, 0), ctx);
if (!is64)
emit_zext_32(rs, ctx);
emit_zextw(rs, rs, ctx);
break;
case BPF_AND | BPF_FETCH:
emit(is64 ? rv_amoand_d(rs, rs, rd, 0, 0) :
rv_amoand_w(rs, rs, rd, 0, 0), ctx);
if (!is64)
emit_zext_32(rs, ctx);
emit_zextw(rs, rs, ctx);
break;
case BPF_OR | BPF_FETCH:
emit(is64 ? rv_amoor_d(rs, rs, rd, 0, 0) :
rv_amoor_w(rs, rs, rd, 0, 0), ctx);
if (!is64)
emit_zext_32(rs, ctx);
emit_zextw(rs, rs, ctx);
break;
case BPF_XOR | BPF_FETCH:
emit(is64 ? rv_amoxor_d(rs, rs, rd, 0, 0) :
rv_amoxor_w(rs, rs, rd, 0, 0), ctx);
if (!is64)
emit_zext_32(rs, ctx);
emit_zextw(rs, rs, ctx);
break;
/* src_reg = atomic_xchg(dst_reg + off16, src_reg); */
case BPF_XCHG:
emit(is64 ? rv_amoswap_d(rs, rs, rd, 0, 0) :
rv_amoswap_w(rs, rs, rd, 0, 0), ctx);
if (!is64)
emit_zext_32(rs, ctx);
emit_zextw(rs, rs, ctx);
break;
/* r0 = atomic_cmpxchg(dst_reg + off16, r0, src_reg); */
case BPF_CMPXCHG:
@ -1091,7 +1066,7 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
case BPF_ALU64 | BPF_MOV | BPF_X:
if (imm == 1) {
/* Special mov32 for zext */
emit_zext_32(rd, ctx);
emit_zextw(rd, rd, ctx);
break;
}
switch (insn->off) {
@ -1099,16 +1074,17 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
emit_mv(rd, rs, ctx);
break;
case 8:
emit_sextb(rd, rs, ctx);
break;
case 16:
emit_slli(RV_REG_T1, rs, 64 - insn->off, ctx);
emit_srai(rd, RV_REG_T1, 64 - insn->off, ctx);
emit_sexth(rd, rs, ctx);
break;
case 32:
emit_addiw(rd, rs, 0, ctx);
emit_sextw(rd, rs, ctx);
break;
}
if (!is64 && !aux->verifier_zext)
emit_zext_32(rd, ctx);
emit_zextw(rd, rd, ctx);
break;
/* dst = dst OP src */
@ -1116,7 +1092,7 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
case BPF_ALU64 | BPF_ADD | BPF_X:
emit_add(rd, rd, rs, ctx);
if (!is64 && !aux->verifier_zext)
emit_zext_32(rd, ctx);
emit_zextw(rd, rd, ctx);
break;
case BPF_ALU | BPF_SUB | BPF_X:
case BPF_ALU64 | BPF_SUB | BPF_X:
@ -1126,31 +1102,31 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
emit_subw(rd, rd, rs, ctx);
if (!is64 && !aux->verifier_zext)
emit_zext_32(rd, ctx);
emit_zextw(rd, rd, ctx);
break;
case BPF_ALU | BPF_AND | BPF_X:
case BPF_ALU64 | BPF_AND | BPF_X:
emit_and(rd, rd, rs, ctx);
if (!is64 && !aux->verifier_zext)
emit_zext_32(rd, ctx);
emit_zextw(rd, rd, ctx);
break;
case BPF_ALU | BPF_OR | BPF_X:
case BPF_ALU64 | BPF_OR | BPF_X:
emit_or(rd, rd, rs, ctx);
if (!is64 && !aux->verifier_zext)
emit_zext_32(rd, ctx);
emit_zextw(rd, rd, ctx);
break;
case BPF_ALU | BPF_XOR | BPF_X:
case BPF_ALU64 | BPF_XOR | BPF_X:
emit_xor(rd, rd, rs, ctx);
if (!is64 && !aux->verifier_zext)
emit_zext_32(rd, ctx);
emit_zextw(rd, rd, ctx);
break;
case BPF_ALU | BPF_MUL | BPF_X:
case BPF_ALU64 | BPF_MUL | BPF_X:
emit(is64 ? rv_mul(rd, rd, rs) : rv_mulw(rd, rd, rs), ctx);
if (!is64 && !aux->verifier_zext)
emit_zext_32(rd, ctx);
emit_zextw(rd, rd, ctx);
break;
case BPF_ALU | BPF_DIV | BPF_X:
case BPF_ALU64 | BPF_DIV | BPF_X:
@ -1159,7 +1135,7 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
else
emit(is64 ? rv_divu(rd, rd, rs) : rv_divuw(rd, rd, rs), ctx);
if (!is64 && !aux->verifier_zext)
emit_zext_32(rd, ctx);
emit_zextw(rd, rd, ctx);
break;
case BPF_ALU | BPF_MOD | BPF_X:
case BPF_ALU64 | BPF_MOD | BPF_X:
@ -1168,25 +1144,25 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
else
emit(is64 ? rv_remu(rd, rd, rs) : rv_remuw(rd, rd, rs), ctx);
if (!is64 && !aux->verifier_zext)
emit_zext_32(rd, ctx);
emit_zextw(rd, rd, ctx);
break;
case BPF_ALU | BPF_LSH | BPF_X:
case BPF_ALU64 | BPF_LSH | BPF_X:
emit(is64 ? rv_sll(rd, rd, rs) : rv_sllw(rd, rd, rs), ctx);
if (!is64 && !aux->verifier_zext)
emit_zext_32(rd, ctx);
emit_zextw(rd, rd, ctx);
break;
case BPF_ALU | BPF_RSH | BPF_X:
case BPF_ALU64 | BPF_RSH | BPF_X:
emit(is64 ? rv_srl(rd, rd, rs) : rv_srlw(rd, rd, rs), ctx);
if (!is64 && !aux->verifier_zext)
emit_zext_32(rd, ctx);
emit_zextw(rd, rd, ctx);
break;
case BPF_ALU | BPF_ARSH | BPF_X:
case BPF_ALU64 | BPF_ARSH | BPF_X:
emit(is64 ? rv_sra(rd, rd, rs) : rv_sraw(rd, rd, rs), ctx);
if (!is64 && !aux->verifier_zext)
emit_zext_32(rd, ctx);
emit_zextw(rd, rd, ctx);
break;
/* dst = -dst */
@ -1194,73 +1170,27 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
case BPF_ALU64 | BPF_NEG:
emit_sub(rd, RV_REG_ZERO, rd, ctx);
if (!is64 && !aux->verifier_zext)
emit_zext_32(rd, ctx);
emit_zextw(rd, rd, ctx);
break;
/* dst = BSWAP##imm(dst) */
case BPF_ALU | BPF_END | BPF_FROM_LE:
switch (imm) {
case 16:
emit_slli(rd, rd, 48, ctx);
emit_srli(rd, rd, 48, ctx);
emit_zexth(rd, rd, ctx);
break;
case 32:
if (!aux->verifier_zext)
emit_zext_32(rd, ctx);
emit_zextw(rd, rd, ctx);
break;
case 64:
/* Do nothing */
break;
}
break;
case BPF_ALU | BPF_END | BPF_FROM_BE:
case BPF_ALU64 | BPF_END | BPF_FROM_LE:
emit_li(RV_REG_T2, 0, ctx);
emit_andi(RV_REG_T1, rd, 0xff, ctx);
emit_add(RV_REG_T2, RV_REG_T2, RV_REG_T1, ctx);
emit_slli(RV_REG_T2, RV_REG_T2, 8, ctx);
emit_srli(rd, rd, 8, ctx);
if (imm == 16)
goto out_be;
emit_andi(RV_REG_T1, rd, 0xff, ctx);
emit_add(RV_REG_T2, RV_REG_T2, RV_REG_T1, ctx);
emit_slli(RV_REG_T2, RV_REG_T2, 8, ctx);
emit_srli(rd, rd, 8, ctx);
emit_andi(RV_REG_T1, rd, 0xff, ctx);
emit_add(RV_REG_T2, RV_REG_T2, RV_REG_T1, ctx);
emit_slli(RV_REG_T2, RV_REG_T2, 8, ctx);
emit_srli(rd, rd, 8, ctx);
if (imm == 32)
goto out_be;
emit_andi(RV_REG_T1, rd, 0xff, ctx);
emit_add(RV_REG_T2, RV_REG_T2, RV_REG_T1, ctx);
emit_slli(RV_REG_T2, RV_REG_T2, 8, ctx);
emit_srli(rd, rd, 8, ctx);
emit_andi(RV_REG_T1, rd, 0xff, ctx);
emit_add(RV_REG_T2, RV_REG_T2, RV_REG_T1, ctx);
emit_slli(RV_REG_T2, RV_REG_T2, 8, ctx);
emit_srli(rd, rd, 8, ctx);
emit_andi(RV_REG_T1, rd, 0xff, ctx);
emit_add(RV_REG_T2, RV_REG_T2, RV_REG_T1, ctx);
emit_slli(RV_REG_T2, RV_REG_T2, 8, ctx);
emit_srli(rd, rd, 8, ctx);
emit_andi(RV_REG_T1, rd, 0xff, ctx);
emit_add(RV_REG_T2, RV_REG_T2, RV_REG_T1, ctx);
emit_slli(RV_REG_T2, RV_REG_T2, 8, ctx);
emit_srli(rd, rd, 8, ctx);
out_be:
emit_andi(RV_REG_T1, rd, 0xff, ctx);
emit_add(RV_REG_T2, RV_REG_T2, RV_REG_T1, ctx);
emit_mv(rd, RV_REG_T2, ctx);
emit_bswap(rd, imm, ctx);
break;
/* dst = imm */
@ -1268,7 +1198,7 @@ out_be:
case BPF_ALU64 | BPF_MOV | BPF_K:
emit_imm(rd, imm, ctx);
if (!is64 && !aux->verifier_zext)
emit_zext_32(rd, ctx);
emit_zextw(rd, rd, ctx);
break;
/* dst = dst OP imm */
@ -1281,7 +1211,7 @@ out_be:
emit_add(rd, rd, RV_REG_T1, ctx);
}
if (!is64 && !aux->verifier_zext)
emit_zext_32(rd, ctx);
emit_zextw(rd, rd, ctx);
break;
case BPF_ALU | BPF_SUB | BPF_K:
case BPF_ALU64 | BPF_SUB | BPF_K:
@ -1292,7 +1222,7 @@ out_be:
emit_sub(rd, rd, RV_REG_T1, ctx);
}
if (!is64 && !aux->verifier_zext)
emit_zext_32(rd, ctx);
emit_zextw(rd, rd, ctx);
break;
case BPF_ALU | BPF_AND | BPF_K:
case BPF_ALU64 | BPF_AND | BPF_K:
@ -1303,7 +1233,7 @@ out_be:
emit_and(rd, rd, RV_REG_T1, ctx);
}
if (!is64 && !aux->verifier_zext)
emit_zext_32(rd, ctx);
emit_zextw(rd, rd, ctx);
break;
case BPF_ALU | BPF_OR | BPF_K:
case BPF_ALU64 | BPF_OR | BPF_K:
@ -1314,7 +1244,7 @@ out_be:
emit_or(rd, rd, RV_REG_T1, ctx);
}
if (!is64 && !aux->verifier_zext)
emit_zext_32(rd, ctx);
emit_zextw(rd, rd, ctx);
break;
case BPF_ALU | BPF_XOR | BPF_K:
case BPF_ALU64 | BPF_XOR | BPF_K:
@ -1325,7 +1255,7 @@ out_be:
emit_xor(rd, rd, RV_REG_T1, ctx);
}
if (!is64 && !aux->verifier_zext)
emit_zext_32(rd, ctx);
emit_zextw(rd, rd, ctx);
break;
case BPF_ALU | BPF_MUL | BPF_K:
case BPF_ALU64 | BPF_MUL | BPF_K:
@ -1333,7 +1263,7 @@ out_be:
emit(is64 ? rv_mul(rd, rd, RV_REG_T1) :
rv_mulw(rd, rd, RV_REG_T1), ctx);
if (!is64 && !aux->verifier_zext)
emit_zext_32(rd, ctx);
emit_zextw(rd, rd, ctx);
break;
case BPF_ALU | BPF_DIV | BPF_K:
case BPF_ALU64 | BPF_DIV | BPF_K:
@ -1345,7 +1275,7 @@ out_be:
emit(is64 ? rv_divu(rd, rd, RV_REG_T1) :
rv_divuw(rd, rd, RV_REG_T1), ctx);
if (!is64 && !aux->verifier_zext)
emit_zext_32(rd, ctx);
emit_zextw(rd, rd, ctx);
break;
case BPF_ALU | BPF_MOD | BPF_K:
case BPF_ALU64 | BPF_MOD | BPF_K:
@ -1357,14 +1287,14 @@ out_be:
emit(is64 ? rv_remu(rd, rd, RV_REG_T1) :
rv_remuw(rd, rd, RV_REG_T1), ctx);
if (!is64 && !aux->verifier_zext)
emit_zext_32(rd, ctx);
emit_zextw(rd, rd, ctx);
break;
case BPF_ALU | BPF_LSH | BPF_K:
case BPF_ALU64 | BPF_LSH | BPF_K:
emit_slli(rd, rd, imm, ctx);
if (!is64 && !aux->verifier_zext)
emit_zext_32(rd, ctx);
emit_zextw(rd, rd, ctx);
break;
case BPF_ALU | BPF_RSH | BPF_K:
case BPF_ALU64 | BPF_RSH | BPF_K:
@ -1374,7 +1304,7 @@ out_be:
emit(rv_srliw(rd, rd, imm), ctx);
if (!is64 && !aux->verifier_zext)
emit_zext_32(rd, ctx);
emit_zextw(rd, rd, ctx);
break;
case BPF_ALU | BPF_ARSH | BPF_K:
case BPF_ALU64 | BPF_ARSH | BPF_K:
@ -1384,7 +1314,7 @@ out_be:
emit(rv_sraiw(rd, rd, imm), ctx);
if (!is64 && !aux->verifier_zext)
emit_zext_32(rd, ctx);
emit_zextw(rd, rd, ctx);
break;
/* JUMP off */
@ -1425,10 +1355,13 @@ out_be:
rvoff = rv_offset(i, off, ctx);
if (!is64) {
s = ctx->ninsns;
if (is_signed_bpf_cond(BPF_OP(code)))
emit_sext_32_rd_rs(&rd, &rs, ctx);
else
emit_zext_32_rd_rs(&rd, &rs, ctx);
if (is_signed_bpf_cond(BPF_OP(code))) {
emit_sextw_alt(&rs, RV_REG_T1, ctx);
emit_sextw_alt(&rd, RV_REG_T2, ctx);
} else {
emit_zextw_alt(&rs, RV_REG_T1, ctx);
emit_zextw_alt(&rd, RV_REG_T2, ctx);
}
e = ctx->ninsns;
/* Adjust for extra insns */
@ -1439,8 +1372,7 @@ out_be:
/* Adjust for and */
rvoff -= 4;
emit_and(RV_REG_T1, rd, rs, ctx);
emit_branch(BPF_JNE, RV_REG_T1, RV_REG_ZERO, rvoff,
ctx);
emit_branch(BPF_JNE, RV_REG_T1, RV_REG_ZERO, rvoff, ctx);
} else {
emit_branch(BPF_OP(code), rd, rs, rvoff, ctx);
}
@ -1469,18 +1401,18 @@ out_be:
case BPF_JMP32 | BPF_JSLE | BPF_K:
rvoff = rv_offset(i, off, ctx);
s = ctx->ninsns;
if (imm) {
if (imm)
emit_imm(RV_REG_T1, imm, ctx);
rs = RV_REG_T1;
} else {
/* If imm is 0, simply use zero register. */
rs = RV_REG_ZERO;
}
rs = imm ? RV_REG_T1 : RV_REG_ZERO;
if (!is64) {
if (is_signed_bpf_cond(BPF_OP(code)))
emit_sext_32_rd(&rd, ctx);
else
emit_zext_32_rd_t1(&rd, ctx);
if (is_signed_bpf_cond(BPF_OP(code))) {
emit_sextw_alt(&rd, RV_REG_T2, ctx);
/* rs has been sign extended */
} else {
emit_zextw_alt(&rd, RV_REG_T2, ctx);
if (imm)
emit_zextw(rs, rs, ctx);
}
}
e = ctx->ninsns;
@ -1504,7 +1436,7 @@ out_be:
* as t1 is used only in comparison against zero.
*/
if (!is64 && imm < 0)
emit_addiw(RV_REG_T1, RV_REG_T1, 0, ctx);
emit_sextw(RV_REG_T1, RV_REG_T1, ctx);
e = ctx->ninsns;
rvoff -= ninsns_rvoff(e - s);
emit_branch(BPF_JNE, RV_REG_T1, RV_REG_ZERO, rvoff, ctx);
@ -1874,3 +1806,8 @@ bool bpf_jit_supports_kfunc_call(void)
{
return true;
}
bool bpf_jit_supports_ptr_xchg(void)
{
return true;
}

8
drivers/hid/bpf/hid_bpf_dispatch.c
View File

@ -176,9 +176,9 @@ __bpf_kfunc_end_defs();
* The following set contains all functions we agree BPF programs
* can use.
*/
BTF_SET8_START(hid_bpf_kfunc_ids)
BTF_KFUNCS_START(hid_bpf_kfunc_ids)
BTF_ID_FLAGS(func, hid_bpf_get_data, KF_RET_NULL)
BTF_SET8_END(hid_bpf_kfunc_ids)
BTF_KFUNCS_END(hid_bpf_kfunc_ids)
static const struct btf_kfunc_id_set hid_bpf_kfunc_set = {
.owner = THIS_MODULE,
@ -487,12 +487,12 @@ static const struct btf_kfunc_id_set hid_bpf_fmodret_set = {
};
/* for syscall HID-BPF */
BTF_SET8_START(hid_bpf_syscall_kfunc_ids)
BTF_KFUNCS_START(hid_bpf_syscall_kfunc_ids)
BTF_ID_FLAGS(func, hid_bpf_attach_prog)
BTF_ID_FLAGS(func, hid_bpf_allocate_context, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, hid_bpf_release_context, KF_RELEASE)
BTF_ID_FLAGS(func, hid_bpf_hw_request)
BTF_SET8_END(hid_bpf_syscall_kfunc_ids)
BTF_KFUNCS_END(hid_bpf_syscall_kfunc_ids)
static const struct btf_kfunc_id_set hid_bpf_syscall_kfunc_set = {
.owner = THIS_MODULE,

4
fs/verity/measure.c
View File

@ -159,9 +159,9 @@ __bpf_kfunc int bpf_get_fsverity_digest(struct file *file, struct bpf_dynptr_ker
__bpf_kfunc_end_defs();
BTF_SET8_START(fsverity_set_ids)
BTF_KFUNCS_START(fsverity_set_ids)
BTF_ID_FLAGS(func, bpf_get_fsverity_digest, KF_TRUSTED_ARGS)
BTF_SET8_END(fsverity_set_ids)
BTF_KFUNCS_END(fsverity_set_ids)
static int bpf_get_fsverity_digest_filter(const struct bpf_prog *prog, u32 kfunc_id)
{

3
include/linux/bpf-cgroup.h
View File

@ -196,7 +196,8 @@ static inline bool cgroup_bpf_sock_enabled(struct sock *sk,
({ \
int __ret = 0; \
if (cgroup_bpf_enabled(CGROUP_INET_INGRESS) && \
cgroup_bpf_sock_enabled(sk, CGROUP_INET_INGRESS)) \
cgroup_bpf_sock_enabled(sk, CGROUP_INET_INGRESS) && sk && \
sk_fullsock(sk)) \
__ret = __cgroup_bpf_run_filter_skb(sk, skb, \
CGROUP_INET_INGRESS); \
\

35
include/linux/bpf.h
View File

@ -251,10 +251,7 @@ struct bpf_list_node_kern {
} __attribute__((aligned(8)));
struct bpf_map {
/* The first two cachelines with read-mostly members of which some
* are also accessed in fast-path (e.g. ops, max_entries).
*/
const struct bpf_map_ops *ops ____cacheline_aligned;
const struct bpf_map_ops *ops;
struct bpf_map *inner_map_meta;
#ifdef CONFIG_SECURITY
void *security;
@ -276,17 +273,14 @@ struct bpf_map {
struct obj_cgroup *objcg;
#endif
char name[BPF_OBJ_NAME_LEN];
/* The 3rd and 4th cacheline with misc members to avoid false sharing
* particularly with refcounting.
*/
atomic64_t refcnt ____cacheline_aligned;
struct mutex freeze_mutex;
atomic64_t refcnt;
atomic64_t usercnt;
/* rcu is used before freeing and work is only used during freeing */
union {
struct work_struct work;
struct rcu_head rcu;
};
struct mutex freeze_mutex;
atomic64_t writecnt;
/* 'Ownership' of program-containing map is claimed by the first program
* that is going to use this map or by the first program which FD is
@ -1189,7 +1183,6 @@ struct bpf_trampoline {
int progs_cnt[BPF_TRAMP_MAX];
/* Executable image of trampoline */
struct bpf_tramp_image *cur_image;
struct module *mod;
};
struct bpf_attach_target_info {
@ -1416,6 +1409,7 @@ struct bpf_jit_poke_descriptor {
struct bpf_ctx_arg_aux {
u32 offset;
enum bpf_reg_type reg_type;
struct btf *btf;
u32 btf_id;
};
@ -1709,6 +1703,19 @@ struct bpf_struct_ops {
struct btf_func_model func_models[BPF_STRUCT_OPS_MAX_NR_MEMBERS];
};
/* Every member of a struct_ops type has an instance even a member is not
* an operator (function pointer). The "info" field will be assigned to
* prog->aux->ctx_arg_info of BPF struct_ops programs to provide the
* argument information required by the verifier to verify the program.
*
* btf_ctx_access() will lookup prog->aux->ctx_arg_info to find the
* corresponding entry for an given argument.
*/
struct bpf_struct_ops_arg_info {
struct bpf_ctx_arg_aux *info;
u32 cnt;
};
struct bpf_struct_ops_desc {
struct bpf_struct_ops *st_ops;
@ -1716,6 +1723,9 @@ struct bpf_struct_ops_desc {
const struct btf_type *value_type;
u32 type_id;
u32 value_id;
/* Collection of argument information for each member */
struct bpf_struct_ops_arg_info *arg_info;
};
enum bpf_struct_ops_state {
@ -1790,6 +1800,7 @@ int bpf_struct_ops_desc_init(struct bpf_struct_ops_desc *st_ops_desc,
struct btf *btf,
struct bpf_verifier_log *log);
void bpf_map_struct_ops_info_fill(struct bpf_map_info *info, struct bpf_map *map);
void bpf_struct_ops_desc_release(struct bpf_struct_ops_desc *st_ops_desc);
#else
#define register_bpf_struct_ops(st_ops, type) ({ (void *)(st_ops); 0; })
static inline bool bpf_try_module_get(const void *data, struct module *owner)
@ -1814,6 +1825,10 @@ static inline void bpf_map_struct_ops_info_fill(struct bpf_map_info *info, struc
{
}
static inline void bpf_struct_ops_desc_release(struct bpf_struct_ops_desc *st_ops_desc)
{
}
#endif
#if defined(CONFIG_CGROUP_BPF) && defined(CONFIG_BPF_LSM)

30
include/linux/bpf_local_storage.h
View File

@ -129,10 +129,36 @@ bpf_local_storage_map_alloc(union bpf_attr *attr,
struct bpf_local_storage_cache *cache,
bool bpf_ma);
struct bpf_local_storage_data *
void __bpf_local_storage_insert_cache(struct bpf_local_storage *local_storage,
struct bpf_local_storage_map *smap,
struct bpf_local_storage_elem *selem);
/* If cacheit_lockit is false, this lookup function is lockless */
static inline struct bpf_local_storage_data *
bpf_local_storage_lookup(struct bpf_local_storage *local_storage,
struct bpf_local_storage_map *smap,
bool cacheit_lockit);
bool cacheit_lockit)
{
struct bpf_local_storage_data *sdata;
struct bpf_local_storage_elem *selem;
/* Fast path (cache hit) */
sdata = rcu_dereference_check(local_storage->cache[smap->cache_idx],
bpf_rcu_lock_held());
if (sdata && rcu_access_pointer(sdata->smap) == smap)
return sdata;
/* Slow path (cache miss) */
hlist_for_each_entry_rcu(selem, &local_storage->list, snode,
rcu_read_lock_trace_held())
if (rcu_access_pointer(SDATA(selem)->smap) == smap)
break;
if (!selem)
return NULL;
if (cacheit_lockit)
__bpf_local_storage_insert_cache(local_storage, smap, selem);
return SDATA(selem);
}
void bpf_local_storage_destroy(struct bpf_local_storage *local_storage);

10
include/linux/bpf_verifier.h
View File

@ -610,6 +610,7 @@ struct bpf_subprog_arg_info {
enum bpf_arg_type arg_type;
union {
u32 mem_size;
u32 btf_id;
};
};
@ -918,6 +919,15 @@ static inline void mark_verifier_state_scratched(struct bpf_verifier_env *env)
env->scratched_stack_slots = ~0ULL;
}
static inline bool bpf_stack_narrow_access_ok(int off, int fill_size, int spill_size)
{
#ifdef __BIG_ENDIAN
off -= spill_size - fill_size;
#endif
return !(off % BPF_REG_SIZE);
}
const char *reg_type_str(struct bpf_verifier_env *env, enum bpf_reg_type type);
const char *dynptr_type_str(enum bpf_dynptr_type type);
const char *iter_type_str(const struct btf *btf, u32 btf_id);

23
include/linux/btf.h
View File

@ -495,6 +495,12 @@ static inline void *btf_id_set8_contains(const struct btf_id_set8 *set, u32 id)
return bsearch(&id, set->pairs, set->cnt, sizeof(set->pairs[0]), btf_id_cmp_func);
}
bool btf_param_match_suffix(const struct btf *btf,
const struct btf_param *arg,
const char *suffix);
int btf_ctx_arg_offset(const struct btf *btf, const struct btf_type *func_proto,
u32 arg_no);
struct bpf_verifier_log;
#if defined(CONFIG_BPF_JIT) && defined(CONFIG_BPF_SYSCALL)
@ -525,10 +531,9 @@ s32 btf_find_dtor_kfunc(struct btf *btf, u32 btf_id);
int register_btf_id_dtor_kfuncs(const struct btf_id_dtor_kfunc *dtors, u32 add_cnt,
struct module *owner);
struct btf_struct_meta *btf_find_struct_meta(const struct btf *btf, u32 btf_id);
const struct btf_type *
btf_get_prog_ctx_type(struct bpf_verifier_log *log, const struct btf *btf,
const struct btf_type *t, enum bpf_prog_type prog_type,
int arg);
bool btf_is_prog_ctx_type(struct bpf_verifier_log *log, const struct btf *btf,
const struct btf_type *t, enum bpf_prog_type prog_type,
int arg);
int get_kern_ctx_btf_id(struct bpf_verifier_log *log, enum bpf_prog_type prog_type);
bool btf_types_are_same(const struct btf *btf1, u32 id1,
const struct btf *btf2, u32 id2);
@ -568,12 +573,12 @@ static inline struct btf_struct_meta *btf_find_struct_meta(const struct btf *btf
{
return NULL;
}
static inline const struct btf_member *
btf_get_prog_ctx_type(struct bpf_verifier_log *log, const struct btf *btf,
const struct btf_type *t, enum bpf_prog_type prog_type,
int arg)
static inline bool
btf_is_prog_ctx_type(struct bpf_verifier_log *log, const struct btf *btf,
const struct btf_type *t, enum bpf_prog_type prog_type,
int arg)
{
return NULL;
return false;
}
static inline int get_kern_ctx_btf_id(struct bpf_verifier_log *log,
enum bpf_prog_type prog_type) {

21
include/linux/btf_ids.h
View File

@ -8,6 +8,9 @@ struct btf_id_set {
u32 ids[];
};
/* This flag implies BTF_SET8 holds kfunc(s) */
#define BTF_SET8_KFUNCS (1 << 0)
struct btf_id_set8 {
u32 cnt;
u32 flags;
@ -21,6 +24,7 @@ struct btf_id_set8 {
#include <linux/compiler.h> /* for __PASTE */
#include <linux/compiler_attributes.h> /* for __maybe_unused */
#include <linux/stringify.h>
/*
* Following macros help to define lists of BTF IDs placed
@ -183,17 +187,18 @@ extern struct btf_id_set name;
* .word (1 << 3) | (1 << 1) | (1 << 2)
*
*/
#define __BTF_SET8_START(name, scope) \
#define __BTF_SET8_START(name, scope, flags) \
__BTF_ID_LIST(name, local) \
asm( \
".pushsection " BTF_IDS_SECTION ",\"a\"; \n" \
"." #scope " __BTF_ID__set8__" #name "; \n" \
"__BTF_ID__set8__" #name ":; \n" \
".zero 8 \n" \
".zero 4 \n" \
".long " __stringify(flags) "\n" \
".popsection; \n");
#define BTF_SET8_START(name) \
__BTF_ID_LIST(name, local) \
__BTF_SET8_START(name, local)
__BTF_SET8_START(name, local, 0)
#define BTF_SET8_END(name) \
asm( \
@ -202,6 +207,12 @@ asm( \
".popsection; \n"); \
extern struct btf_id_set8 name;
#define BTF_KFUNCS_START(name) \
__BTF_SET8_START(name, local, BTF_SET8_KFUNCS)
#define BTF_KFUNCS_END(name) \
BTF_SET8_END(name)
#else
#define BTF_ID_LIST(name) static u32 __maybe_unused name[64];
@ -216,6 +227,8 @@ extern struct btf_id_set8 name;
#define BTF_SET_END(name)
#define BTF_SET8_START(name) static struct btf_id_set8 __maybe_unused name = { 0 };
#define BTF_SET8_END(name)
#define BTF_KFUNCS_START(name) static struct btf_id_set8 __maybe_unused name = { .flags = BTF_SET8_KFUNCS };
#define BTF_KFUNCS_END(name)
#endif /* CONFIG_DEBUG_INFO_BTF */

21
include/linux/filter.h
View File

@ -547,24 +547,27 @@ static inline bool insn_is_zext(const struct bpf_insn *insn)
__BPF_MAP(n, __BPF_DECL_ARGS, __BPF_N, u64, __ur_1, u64, __ur_2, \
u64, __ur_3, u64, __ur_4, u64, __ur_5)
#define BPF_CALL_x(x, name, ...) \
#define BPF_CALL_x(x, attr, name, ...) \
static __always_inline \
u64 ____##name(__BPF_MAP(x, __BPF_DECL_ARGS, __BPF_V, __VA_ARGS__)); \
typedef u64 (*btf_##name)(__BPF_MAP(x, __BPF_DECL_ARGS, __BPF_V, __VA_ARGS__)); \
u64 name(__BPF_REG(x, __BPF_DECL_REGS, __BPF_N, __VA_ARGS__)); \
u64 name(__BPF_REG(x, __BPF_DECL_REGS, __BPF_N, __VA_ARGS__)) \
attr u64 name(__BPF_REG(x, __BPF_DECL_REGS, __BPF_N, __VA_ARGS__)); \
attr u64 name(__BPF_REG(x, __BPF_DECL_REGS, __BPF_N, __VA_ARGS__)) \
{ \
return ((btf_##name)____##name)(__BPF_MAP(x,__BPF_CAST,__BPF_N,__VA_ARGS__));\
} \
static __always_inline \
u64 ____##name(__BPF_MAP(x, __BPF_DECL_ARGS, __BPF_V, __VA_ARGS__))
#define BPF_CALL_0(name, ...) BPF_CALL_x(0, name, __VA_ARGS__)
#define BPF_CALL_1(name, ...) BPF_CALL_x(1, name, __VA_ARGS__)
#define BPF_CALL_2(name, ...) BPF_CALL_x(2, name, __VA_ARGS__)
#define BPF_CALL_3(name, ...) BPF_CALL_x(3, name, __VA_ARGS__)
#define BPF_CALL_4(name, ...) BPF_CALL_x(4, name, __VA_ARGS__)
#define BPF_CALL_5(name, ...) BPF_CALL_x(5, name, __VA_ARGS__)
#define __NOATTR
#define BPF_CALL_0(name, ...) BPF_CALL_x(0, __NOATTR, name, __VA_ARGS__)
#define BPF_CALL_1(name, ...) BPF_CALL_x(1, __NOATTR, name, __VA_ARGS__)
#define BPF_CALL_2(name, ...) BPF_CALL_x(2, __NOATTR, name, __VA_ARGS__)
#define BPF_CALL_3(name, ...) BPF_CALL_x(3, __NOATTR, name, __VA_ARGS__)
#define BPF_CALL_4(name, ...) BPF_CALL_x(4, __NOATTR, name, __VA_ARGS__)
#define BPF_CALL_5(name, ...) BPF_CALL_x(5, __NOATTR, name, __VA_ARGS__)
#define NOTRACE_BPF_CALL_1(name, ...) BPF_CALL_x(1, notrace, name, __VA_ARGS__)
#define bpf_ctx_range(TYPE, MEMBER) \
offsetof(TYPE, MEMBER) ... offsetofend(TYPE, MEMBER) - 1

25
include/uapi/linux/bpf.h
View File

@ -77,12 +77,29 @@ struct bpf_insn {
__s32 imm; /* signed immediate constant */
};
/* Key of an a BPF_MAP_TYPE_LPM_TRIE entry */
/* Deprecated: use struct bpf_lpm_trie_key_u8 (when the "data" member is needed for
* byte access) or struct bpf_lpm_trie_key_hdr (when using an alternative type for
* the trailing flexible array member) instead.
*/
struct bpf_lpm_trie_key {
__u32 prefixlen; /* up to 32 for AF_INET, 128 for AF_INET6 */
__u8 data[0]; /* Arbitrary size */
};
/* Header for bpf_lpm_trie_key structs */
struct bpf_lpm_trie_key_hdr {
__u32 prefixlen;
};
/* Key of an a BPF_MAP_TYPE_LPM_TRIE entry, with trailing byte array. */
struct bpf_lpm_trie_key_u8 {
union {
struct bpf_lpm_trie_key_hdr hdr;
__u32 prefixlen;
};
__u8 data[]; /* Arbitrary size */
};
struct bpf_cgroup_storage_key {
__u64 cgroup_inode_id; /* cgroup inode id */
__u32 attach_type; /* program attach type (enum bpf_attach_type) */
@ -617,7 +634,11 @@ union bpf_iter_link_info {
* to NULL to begin the batched operation. After each subsequent
* **BPF_MAP_LOOKUP_BATCH**, the caller should pass the resultant
* *out_batch* as the *in_batch* for the next operation to
* continue iteration from the current point.
* continue iteration from the current point. Both *in_batch* and
* *out_batch* must point to memory large enough to hold a key,
* except for maps of type **BPF_MAP_TYPE_{HASH, PERCPU_HASH,
* LRU_HASH, LRU_PERCPU_HASH}**, for which batch parameters
* must be at least 4 bytes wide regardless of key size.
*
* The *keys* and *values* are output parameters which must point
* to memory large enough to hold *count* items based on the key

5
init/Kconfig
View File

@ -1466,11 +1466,6 @@ config SYSCTL_ARCH_UNALIGN_ALLOW
config HAVE_PCSPKR_PLATFORM
bool
# interpreter that classic socket filters depend on
config BPF
bool
select CRYPTO_LIB_SHA1
menuconfig EXPERT
bool "Configure standard kernel features (expert users)"
# Unhide debug options, to make the on-by-default options visible

1
kernel/bpf/Kconfig
View File

@ -3,6 +3,7 @@
# BPF interpreter that, for example, classic socket filters depend on.
config BPF
bool
select CRYPTO_LIB_SHA1
# Used by archs to tell that they support BPF JIT compiler plus which
# flavour. Only one of the two can be selected for a specific arch since

52
kernel/bpf/bpf_local_storage.c
View File

@ -414,47 +414,21 @@ void bpf_selem_unlink(struct bpf_local_storage_elem *selem, bool reuse_now)
bpf_selem_unlink_storage(selem, reuse_now);
}
/* If cacheit_lockit is false, this lookup function is lockless */
struct bpf_local_storage_data *
bpf_local_storage_lookup(struct bpf_local_storage *local_storage,
struct bpf_local_storage_map *smap,
bool cacheit_lockit)
void __bpf_local_storage_insert_cache(struct bpf_local_storage *local_storage,
struct bpf_local_storage_map *smap,
struct bpf_local_storage_elem *selem)
{
struct bpf_local_storage_data *sdata;
struct bpf_local_storage_elem *selem;
unsigned long flags;
/* Fast path (cache hit) */
sdata = rcu_dereference_check(local_storage->cache[smap->cache_idx],
bpf_rcu_lock_held());
if (sdata && rcu_access_pointer(sdata->smap) == smap)
return sdata;
/* Slow path (cache miss) */
hlist_for_each_entry_rcu(selem, &local_storage->list, snode,
rcu_read_lock_trace_held())
if (rcu_access_pointer(SDATA(selem)->smap) == smap)
break;
if (!selem)
return NULL;
sdata = SDATA(selem);
if (cacheit_lockit) {
unsigned long flags;
/* spinlock is needed to avoid racing with the
* parallel delete. Otherwise, publishing an already
* deleted sdata to the cache will become a use-after-free
* problem in the next bpf_local_storage_lookup().
*/
raw_spin_lock_irqsave(&local_storage->lock, flags);
if (selem_linked_to_storage(selem))
rcu_assign_pointer(local_storage->cache[smap->cache_idx],
sdata);
raw_spin_unlock_irqrestore(&local_storage->lock, flags);
}
return sdata;
/* spinlock is needed to avoid racing with the
* parallel delete. Otherwise, publishing an already
* deleted sdata to the cache will become a use-after-free
* problem in the next bpf_local_storage_lookup().
*/
raw_spin_lock_irqsave(&local_storage->lock, flags);
if (selem_linked_to_storage(selem))
rcu_assign_pointer(local_storage->cache[smap->cache_idx], SDATA(selem));
raw_spin_unlock_irqrestore(&local_storage->lock, flags);
}
static int check_flags(const struct bpf_local_storage_data *old_sdata,

6
kernel/bpf/bpf_lsm.c
View File

@ -282,10 +282,6 @@ BTF_ID(func, bpf_lsm_file_lock)
BTF_ID(func, bpf_lsm_file_open)
BTF_ID(func, bpf_lsm_file_receive)
#ifdef CONFIG_SECURITY_NETWORK
BTF_ID(func, bpf_lsm_inet_conn_established)
#endif /* CONFIG_SECURITY_NETWORK */
BTF_ID(func, bpf_lsm_inode_create)
BTF_ID(func, bpf_lsm_inode_free_security)
BTF_ID(func, bpf_lsm_inode_getattr)
@ -336,6 +332,8 @@ BTF_ID(func, bpf_lsm_sb_umount)
BTF_ID(func, bpf_lsm_settime)
#ifdef CONFIG_SECURITY_NETWORK
BTF_ID(func, bpf_lsm_inet_conn_established)
BTF_ID(func, bpf_lsm_socket_accept)
BTF_ID(func, bpf_lsm_socket_bind)
BTF_ID(func, bpf_lsm_socket_connect)

227
kernel/bpf/bpf_struct_ops.c
View File

@ -116,17 +116,183 @@ static bool is_valid_value_type(struct btf *btf, s32 value_id,
return true;
}
#define MAYBE_NULL_SUFFIX "__nullable"
#define MAX_STUB_NAME 128
/* Return the type info of a stub function, if it exists.
*
* The name of a stub function is made up of the name of the struct_ops and
* the name of the function pointer member, separated by "__". For example,
* if the struct_ops type is named "foo_ops" and the function pointer
* member is named "bar", the stub function name would be "foo_ops__bar".
*/
static const struct btf_type *
find_stub_func_proto(const struct btf *btf, const char *st_op_name,
const char *member_name)
{
char stub_func_name[MAX_STUB_NAME];
const struct btf_type *func_type;
s32 btf_id;
int cp;
cp = snprintf(stub_func_name, MAX_STUB_NAME, "%s__%s",
st_op_name, member_name);
if (cp >= MAX_STUB_NAME) {
pr_warn("Stub function name too long\n");
return NULL;
}
btf_id = btf_find_by_name_kind(btf, stub_func_name, BTF_KIND_FUNC);
if (btf_id < 0)
return NULL;
func_type = btf_type_by_id(btf, btf_id);
if (!func_type)
return NULL;
return btf_type_by_id(btf, func_type->type); /* FUNC_PROTO */
}
/* Prepare argument info for every nullable argument of a member of a
* struct_ops type.
*
* Initialize a struct bpf_struct_ops_arg_info according to type info of
* the arguments of a stub function. (Check kCFI for more information about
* stub functions.)
*
* Each member in the struct_ops type has a struct bpf_struct_ops_arg_info
* to provide an array of struct bpf_ctx_arg_aux, which in turn provides
* the information that used by the verifier to check the arguments of the
* BPF struct_ops program assigned to the member. Here, we only care about
* the arguments that are marked as __nullable.
*
* The array of struct bpf_ctx_arg_aux is eventually assigned to
* prog->aux->ctx_arg_info of BPF struct_ops programs and passed to the
* verifier. (See check_struct_ops_btf_id())
*
* arg_info->info will be the list of struct bpf_ctx_arg_aux if success. If
* fails, it will be kept untouched.
*/
static int prepare_arg_info(struct btf *btf,
const char *st_ops_name,
const char *member_name,
const struct btf_type *func_proto,
struct bpf_struct_ops_arg_info *arg_info)
{
const struct btf_type *stub_func_proto, *pointed_type;
const struct btf_param *stub_args, *args;
struct bpf_ctx_arg_aux *info, *info_buf;
u32 nargs, arg_no, info_cnt = 0;
u32 arg_btf_id;
int offset;
stub_func_proto = find_stub_func_proto(btf, st_ops_name, member_name);
if (!stub_func_proto)
return 0;
/* Check if the number of arguments of the stub function is the same
* as the number of arguments of the function pointer.
*/
nargs = btf_type_vlen(func_proto);
if (nargs != btf_type_vlen(stub_func_proto)) {
pr_warn("the number of arguments of the stub function %s__%s does not match the number of arguments of the member %s of struct %s\n",
st_ops_name, member_name, member_name, st_ops_name);
return -EINVAL;
}
if (!nargs)
return 0;
args = btf_params(func_proto);
stub_args = btf_params(stub_func_proto);
info_buf = kcalloc(nargs, sizeof(*info_buf), GFP_KERNEL);
if (!info_buf)
return -ENOMEM;
/* Prepare info for every nullable argument */
info = info_buf;
for (arg_no = 0; arg_no < nargs; arg_no++) {
/* Skip arguments that is not suffixed with
* "__nullable".
*/
if (!btf_param_match_suffix(btf, &stub_args[arg_no],
MAYBE_NULL_SUFFIX))
continue;
/* Should be a pointer to struct */
pointed_type = btf_type_resolve_ptr(btf,
args[arg_no].type,
&arg_btf_id);
if (!pointed_type ||
!btf_type_is_struct(pointed_type)) {
pr_warn("stub function %s__%s has %s tagging to an unsupported type\n",
st_ops_name, member_name, MAYBE_NULL_SUFFIX);
goto err_out;
}
offset = btf_ctx_arg_offset(btf, func_proto, arg_no);
if (offset < 0) {
pr_warn("stub function %s__%s has an invalid trampoline ctx offset for arg#%u\n",
st_ops_name, member_name, arg_no);
goto err_out;
}
if (args[arg_no].type != stub_args[arg_no].type) {
pr_warn("arg#%u type in stub function %s__%s does not match with its original func_proto\n",
arg_no, st_ops_name, member_name);
goto err_out;
}
/* Fill the information of the new argument */
info->reg_type =
PTR_TRUSTED | PTR_TO_BTF_ID | PTR_MAYBE_NULL;
info->btf_id = arg_btf_id;
info->btf = btf;
info->offset = offset;
info++;
info_cnt++;
}
if (info_cnt) {
arg_info->info = info_buf;
arg_info->cnt = info_cnt;
} else {
kfree(info_buf);
}
return 0;
err_out:
kfree(info_buf);
return -EINVAL;
}
/* Clean up the arg_info in a struct bpf_struct_ops_desc. */
void bpf_struct_ops_desc_release(struct bpf_struct_ops_desc *st_ops_desc)
{
struct bpf_struct_ops_arg_info *arg_info;
int i;
arg_info = st_ops_desc->arg_info;
for (i = 0; i < btf_type_vlen(st_ops_desc->type); i++)
kfree(arg_info[i].info);
kfree(arg_info);
}
int bpf_struct_ops_desc_init(struct bpf_struct_ops_desc *st_ops_desc,
struct btf *btf,
struct bpf_verifier_log *log)
{
struct bpf_struct_ops *st_ops = st_ops_desc->st_ops;
struct bpf_struct_ops_arg_info *arg_info;
const struct btf_member *member;
const struct btf_type *t;
s32 type_id, value_id;
char value_name[128];
const char *mname;
int i;
int i, err;
if (strlen(st_ops->name) + VALUE_PREFIX_LEN >=
sizeof(value_name)) {
@ -136,6 +302,11 @@ int bpf_struct_ops_desc_init(struct bpf_struct_ops_desc *st_ops_desc,
}
sprintf(value_name, "%s%s", VALUE_PREFIX, st_ops->name);
if (!st_ops->cfi_stubs) {
pr_warn("struct_ops for %s has no cfi_stubs\n", st_ops->name);
return -EINVAL;
}
type_id = btf_find_by_name_kind(btf, st_ops->name,
BTF_KIND_STRUCT);
if (type_id < 0) {
@ -160,6 +331,17 @@ int bpf_struct_ops_desc_init(struct bpf_struct_ops_desc *st_ops_desc,
if (!is_valid_value_type(btf, value_id, t, value_name))
return -EINVAL;
arg_info = kcalloc(btf_type_vlen(t), sizeof(*arg_info),
GFP_KERNEL);
if (!arg_info)
return -ENOMEM;
st_ops_desc->arg_info = arg_info;
st_ops_desc->type = t;
st_ops_desc->type_id = type_id;
st_ops_desc->value_id = value_id;
st_ops_desc->value_type = btf_type_by_id(btf, value_id);
for_each_member(i, t, member) {
const struct btf_type *func_proto;
@ -167,43 +349,52 @@ int bpf_struct_ops_desc_init(struct bpf_struct_ops_desc *st_ops_desc,
if (!*mname) {
pr_warn("anon member in struct %s is not supported\n",
st_ops->name);
return -EOPNOTSUPP;
err = -EOPNOTSUPP;
goto errout;
}
if (__btf_member_bitfield_size(t, member)) {
pr_warn("bit field member %s in struct %s is not supported\n",
mname, st_ops->name);
return -EOPNOTSUPP;
err = -EOPNOTSUPP;
goto errout;
}
func_proto = btf_type_resolve_func_ptr(btf,
member->type,
NULL);
if (func_proto &&
btf_distill_func_proto(log, btf,
if (!func_proto)
continue;
if (btf_distill_func_proto(log, btf,
func_proto, mname,
&st_ops->func_models[i])) {
pr_warn("Error in parsing func ptr %s in struct %s\n",
mname, st_ops->name);
return -EINVAL;
err = -EINVAL;
goto errout;
}
err = prepare_arg_info(btf, st_ops->name, mname,
func_proto,
arg_info + i);
if (err)
goto errout;
}
if (i == btf_type_vlen(t)) {
if (st_ops->init(btf)) {
pr_warn("Error in init bpf_struct_ops %s\n",
st_ops->name);
return -EINVAL;
} else {
st_ops_desc->type_id = type_id;
st_ops_desc->type = t;
st_ops_desc->value_id = value_id;
st_ops_desc->value_type = btf_type_by_id(btf,
value_id);
}
if (st_ops->init(btf)) {
pr_warn("Error in init bpf_struct_ops %s\n",
st_ops->name);
err = -EINVAL;
goto errout;
}
return 0;
errout:
bpf_struct_ops_desc_release(st_ops_desc);
return err;
}
static int bpf_struct_ops_map_get_next_key(struct bpf_map *map, void *key,

275
kernel/bpf/btf.c
View File

@ -1699,6 +1699,13 @@ static void btf_free_struct_meta_tab(struct btf *btf)
static void btf_free_struct_ops_tab(struct btf *btf)
{
struct btf_struct_ops_tab *tab = btf->struct_ops_tab;
u32 i;
if (!tab)
return;
for (i = 0; i < tab->cnt; i++)
bpf_struct_ops_desc_release(&tab->ops[i]);
kfree(tab);
btf->struct_ops_tab = NULL;
@ -5687,15 +5694,29 @@ static int find_kern_ctx_type_id(enum bpf_prog_type prog_type)
return ctx_type->type;
}
const struct btf_type *
btf_get_prog_ctx_type(struct bpf_verifier_log *log, const struct btf *btf,
const struct btf_type *t, enum bpf_prog_type prog_type,
int arg)
bool btf_is_prog_ctx_type(struct bpf_verifier_log *log, const struct btf *btf,
const struct btf_type *t, enum bpf_prog_type prog_type,
int arg)
{
const struct btf_type *ctx_type;
const char *tname, *ctx_tname;
t = btf_type_by_id(btf, t->type);
/* KPROBE programs allow bpf_user_pt_regs_t typedef, which we need to
* check before we skip all the typedef below.
*/
if (prog_type == BPF_PROG_TYPE_KPROBE) {
while (btf_type_is_modifier(t) && !btf_type_is_typedef(t))
t = btf_type_by_id(btf, t->type);
if (btf_type_is_typedef(t)) {
tname = btf_name_by_offset(btf, t->name_off);
if (tname && strcmp(tname, "bpf_user_pt_regs_t") == 0)
return true;
}
}
while (btf_type_is_modifier(t))
t = btf_type_by_id(btf, t->type);
if (!btf_type_is_struct(t)) {
@ -5704,27 +5725,30 @@ btf_get_prog_ctx_type(struct bpf_verifier_log *log, const struct btf *btf,
* is not supported yet.
* BPF_PROG_TYPE_RAW_TRACEPOINT is fine.
*/
return NULL;
return false;
}
tname = btf_name_by_offset(btf, t->name_off);
if (!tname) {
bpf_log(log, "arg#%d struct doesn't have a name\n", arg);
return NULL;
return false;
}
ctx_type = find_canonical_prog_ctx_type(prog_type);
if (!ctx_type) {
bpf_log(log, "btf_vmlinux is malformed\n");
/* should not happen */
return NULL;
return false;
}
again:
ctx_tname = btf_name_by_offset(btf_vmlinux, ctx_type->name_off);
if (!ctx_tname) {
/* should not happen */
bpf_log(log, "Please fix kernel include/linux/bpf_types.h\n");
return NULL;
return false;
}
/* program types without named context types work only with arg:ctx tag */
if (ctx_tname[0] == '\0')
return false;
/* only compare that prog's ctx type name is the same as
* kernel expects. No need to compare field by field.
* It's ok for bpf prog to do:
@ -5733,20 +5757,20 @@ again:
* { // no fields of skb are ever used }
*/
if (strcmp(ctx_tname, "__sk_buff") == 0 && strcmp(tname, "sk_buff") == 0)
return ctx_type;
return true;
if (strcmp(ctx_tname, "xdp_md") == 0 && strcmp(tname, "xdp_buff") == 0)
return ctx_type;
return true;
if (strcmp(ctx_tname, tname)) {
/* bpf_user_pt_regs_t is a typedef, so resolve it to
* underlying struct and check name again
*/
if (!btf_type_is_modifier(ctx_type))
return NULL;
return false;
while (btf_type_is_modifier(ctx_type))
ctx_type = btf_type_by_id(btf_vmlinux, ctx_type->type);
goto again;
}
return ctx_type;
return true;
}
/* forward declarations for arch-specific underlying types of
@ -5898,7 +5922,7 @@ static int btf_translate_to_vmlinux(struct bpf_verifier_log *log,
enum bpf_prog_type prog_type,
int arg)
{
if (!btf_get_prog_ctx_type(log, btf, t, prog_type, arg))
if (!btf_is_prog_ctx_type(log, btf, t, prog_type, arg))
return -ENOENT;
return find_kern_ctx_type_id(prog_type);
}
@ -6130,6 +6154,26 @@ static bool prog_args_trusted(const struct bpf_prog *prog)
}
}
int btf_ctx_arg_offset(const struct btf *btf, const struct btf_type *func_proto,
u32 arg_no)
{
const struct btf_param *args;
const struct btf_type *t;
int off = 0, i;
u32 sz;
args = btf_params(func_proto);
for (i = 0; i < arg_no; i++) {
t = btf_type_by_id(btf, args[i].type);
t = btf_resolve_size(btf, t, &sz);
if (IS_ERR(t))
return PTR_ERR(t);
off += roundup(sz, 8);
}
return off;
}
bool btf_ctx_access(int off, int size, enum bpf_access_type type,
const struct bpf_prog *prog,
struct bpf_insn_access_aux *info)
@ -6266,7 +6310,7 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,
}
info->reg_type = ctx_arg_info->reg_type;
info->btf = btf_vmlinux;
info->btf = ctx_arg_info->btf ? : btf_vmlinux;
info->btf_id = ctx_arg_info->btf_id;
return true;
}
@ -6985,9 +7029,78 @@ static bool btf_is_dynptr_ptr(const struct btf *btf, const struct btf_type *t)
return false;
}
struct bpf_cand_cache {
const char *name;
u32 name_len;
u16 kind;
u16 cnt;
struct {
const struct btf *btf;
u32 id;
} cands[];
};
static DEFINE_MUTEX(cand_cache_mutex);
static struct bpf_cand_cache *
bpf_core_find_cands(struct bpf_core_ctx *ctx, u32 local_type_id);
static int btf_get_ptr_to_btf_id(struct bpf_verifier_log *log, int arg_idx,
const struct btf *btf, const struct btf_type *t)
{
struct bpf_cand_cache *cc;
struct bpf_core_ctx ctx = {
.btf = btf,
.log = log,
};
u32 kern_type_id, type_id;
int err = 0;
/* skip PTR and modifiers */
type_id = t->type;
t = btf_type_by_id(btf, t->type);
while (btf_type_is_modifier(t)) {
type_id = t->type;
t = btf_type_by_id(btf, t->type);
}
mutex_lock(&cand_cache_mutex);
cc = bpf_core_find_cands(&ctx, type_id);
if (IS_ERR(cc)) {
err = PTR_ERR(cc);
bpf_log(log, "arg#%d reference type('%s %s') candidate matching error: %d\n",
arg_idx, btf_type_str(t), __btf_name_by_offset(btf, t->name_off),
err);
goto cand_cache_unlock;
}
if (cc->cnt != 1) {
bpf_log(log, "arg#%d reference type('%s %s') %s\n",
arg_idx, btf_type_str(t), __btf_name_by_offset(btf, t->name_off),
cc->cnt == 0 ? "has no matches" : "is ambiguous");
err = cc->cnt == 0 ? -ENOENT : -ESRCH;
goto cand_cache_unlock;
}
if (btf_is_module(cc->cands[0].btf)) {
bpf_log(log, "arg#%d reference type('%s %s') points to kernel module type (unsupported)\n",
arg_idx, btf_type_str(t), __btf_name_by_offset(btf, t->name_off));
err = -EOPNOTSUPP;
goto cand_cache_unlock;
}
kern_type_id = cc->cands[0].id;
cand_cache_unlock:
mutex_unlock(&cand_cache_mutex);
if (err)
return err;
return kern_type_id;
}
enum btf_arg_tag {
ARG_TAG_CTX = 0x1,
ARG_TAG_NONNULL = 0x2,
ARG_TAG_TRUSTED = 0x4,
ARG_TAG_NULLABLE = 0x8,
};
/* Process BTF of a function to produce high-level expectation of function
@ -7053,6 +7166,8 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
args = (const struct btf_param *)(t + 1);
nargs = btf_type_vlen(t);
if (nargs > MAX_BPF_FUNC_REG_ARGS) {
if (!is_global)
return -EINVAL;
bpf_log(log, "Global function %s() with %d > %d args. Buggy compiler.\n",
tname, nargs, MAX_BPF_FUNC_REG_ARGS);
return -EINVAL;
@ -7062,6 +7177,8 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
while (btf_type_is_modifier(t))
t = btf_type_by_id(btf, t->type);
if (!btf_type_is_int(t) && !btf_is_any_enum(t)) {
if (!is_global)
return -EINVAL;
bpf_log(log,
"Global function %s() doesn't return scalar. Only those are supported.\n",
tname);
@ -7089,8 +7206,12 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
if (strcmp(tag, "ctx") == 0) {
tags |= ARG_TAG_CTX;
} else if (strcmp(tag, "trusted") == 0) {
tags |= ARG_TAG_TRUSTED;
} else if (strcmp(tag, "nonnull") == 0) {
tags |= ARG_TAG_NONNULL;
} else if (strcmp(tag, "nullable") == 0) {
tags |= ARG_TAG_NULLABLE;
} else {
bpf_log(log, "arg#%d has unsupported set of tags\n", i);
return -EOPNOTSUPP;
@ -7107,11 +7228,15 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
if (!btf_type_is_ptr(t))
goto skip_pointer;
if ((tags & ARG_TAG_CTX) || btf_get_prog_ctx_type(log, btf, t, prog_type, i)) {
if ((tags & ARG_TAG_CTX) || btf_is_prog_ctx_type(log, btf, t, prog_type, i)) {
if (tags & ~ARG_TAG_CTX) {
bpf_log(log, "arg#%d has invalid combination of tags\n", i);
return -EINVAL;
}
if ((tags & ARG_TAG_CTX) &&
btf_validate_prog_ctx_type(log, btf, t, i, prog_type,
prog->expected_attach_type))
return -EINVAL;
sub->args[i].arg_type = ARG_PTR_TO_CTX;
continue;
}
@ -7123,9 +7248,32 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
sub->args[i].arg_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY;
continue;
}
if (tags & ARG_TAG_TRUSTED) {
int kern_type_id;
if (tags & ARG_TAG_NONNULL) {
bpf_log(log, "arg#%d has invalid combination of tags\n", i);
return -EINVAL;
}
kern_type_id = btf_get_ptr_to_btf_id(log, i, btf, t);
if (kern_type_id < 0)
return kern_type_id;
sub->args[i].arg_type = ARG_PTR_TO_BTF_ID | PTR_TRUSTED;
if (tags & ARG_TAG_NULLABLE)
sub->args[i].arg_type |= PTR_MAYBE_NULL;
sub->args[i].btf_id = kern_type_id;
continue;
}
if (is_global) { /* generic user data pointer */
u32 mem_size;
if (tags & ARG_TAG_NULLABLE) {
bpf_log(log, "arg#%d has invalid combination of tags\n", i);
return -EINVAL;
}
t = btf_type_skip_modifiers(btf, t->type, NULL);
ref_t = btf_resolve_size(btf, t, &mem_size);
if (IS_ERR(ref_t)) {
@ -7151,28 +7299,13 @@ skip_pointer:
sub->args[i].arg_type = ARG_ANYTHING;
continue;
}
if (!is_global)
return -EINVAL;
bpf_log(log, "Arg#%d type %s in %s() is not supported yet.\n",
i, btf_type_str(t), tname);
return -EINVAL;
}
for (i = 0; i < nargs; i++) {
const char *tag;
if (sub->args[i].arg_type != ARG_PTR_TO_CTX)
continue;
/* check if arg has "arg:ctx" tag */
t = btf_type_by_id(btf, args[i].type);
tag = btf_find_decl_tag_value(btf, fn_t, i, "arg:");
if (IS_ERR_OR_NULL(tag) || strcmp(tag, "ctx") != 0)
continue;
if (btf_validate_prog_ctx_type(log, btf, t, i, prog_type,
prog->expected_attach_type))
return -EINVAL;
}
sub->arg_cnt = nargs;
sub->args_cached = true;
@ -7649,6 +7782,17 @@ static struct btf *btf_get_module_btf(const struct module *module)
return btf;
}
static int check_btf_kconfigs(const struct module *module, const char *feature)
{
if (!module && IS_ENABLED(CONFIG_DEBUG_INFO_BTF)) {
pr_err("missing vmlinux BTF, cannot register %s\n", feature);
return -ENOENT;
}
if (module && IS_ENABLED(CONFIG_DEBUG_INFO_BTF_MODULES))
pr_warn("missing module BTF, cannot register %s\n", feature);
return 0;
}
BPF_CALL_4(bpf_btf_find_by_name_kind, char *, name, int, name_sz, u32, kind, int, flags)
{
struct btf *btf = NULL;
@ -8009,15 +8153,8 @@ static int __register_btf_kfunc_id_set(enum btf_kfunc_hook hook,
int ret, i;
btf = btf_get_module_btf(kset->owner);
if (!btf) {
if (!kset->owner && IS_ENABLED(CONFIG_DEBUG_INFO_BTF)) {
pr_err("missing vmlinux BTF, cannot register kfuncs\n");
return -ENOENT;
}
if (kset->owner && IS_ENABLED(CONFIG_DEBUG_INFO_BTF_MODULES))
pr_warn("missing module BTF, cannot register kfuncs\n");
return 0;
}
if (!btf)
return check_btf_kconfigs(kset->owner, "kfunc");
if (IS_ERR(btf))
return PTR_ERR(btf);
@ -8041,6 +8178,14 @@ int register_btf_kfunc_id_set(enum bpf_prog_type prog_type,
{
enum btf_kfunc_hook hook;
/* All kfuncs need to be tagged as such in BTF.
* WARN() for initcall registrations that do not check errors.
*/
if (!(kset->set->flags & BTF_SET8_KFUNCS)) {
WARN_ON(!kset->owner);
return -EINVAL;
}
hook = bpf_prog_type_to_kfunc_hook(prog_type);
return __register_btf_kfunc_id_set(hook, kset);
}
@ -8117,17 +8262,8 @@ int register_btf_id_dtor_kfuncs(const struct btf_id_dtor_kfunc *dtors, u32 add_c
int ret;
btf = btf_get_module_btf(owner);
if (!btf) {
if (!owner && IS_ENABLED(CONFIG_DEBUG_INFO_BTF)) {
pr_err("missing vmlinux BTF, cannot register dtor kfuncs\n");
return -ENOENT;
}
if (owner && IS_ENABLED(CONFIG_DEBUG_INFO_BTF_MODULES)) {
pr_err("missing module BTF, cannot register dtor kfuncs\n");
return -ENOENT;
}
return 0;
}
if (!btf)
return check_btf_kconfigs(owner, "dtor kfuncs");
if (IS_ERR(btf))
return PTR_ERR(btf);
@ -8242,17 +8378,6 @@ size_t bpf_core_essential_name_len(const char *name)
return n;
}
struct bpf_cand_cache {
const char *name;
u32 name_len;
u16 kind;
u16 cnt;
struct {
const struct btf *btf;
u32 id;
} cands[];
};
static void bpf_free_cands(struct bpf_cand_cache *cands)
{
if (!cands->cnt)
@ -8273,8 +8398,6 @@ static struct bpf_cand_cache *vmlinux_cand_cache[VMLINUX_CAND_CACHE_SIZE];
#define MODULE_CAND_CACHE_SIZE 31
static struct bpf_cand_cache *module_cand_cache[MODULE_CAND_CACHE_SIZE];
static DEFINE_MUTEX(cand_cache_mutex);
static void __print_cand_cache(struct bpf_verifier_log *log,
struct bpf_cand_cache **cache,
int cache_size)
@ -8803,7 +8926,9 @@ int __register_bpf_struct_ops(struct bpf_struct_ops *st_ops)
btf = btf_get_module_btf(st_ops->owner);
if (!btf)
return -EINVAL;
return check_btf_kconfigs(st_ops->owner, "struct_ops");
if (IS_ERR(btf))
return PTR_ERR(btf);
log = kzalloc(sizeof(*log), GFP_KERNEL | __GFP_NOWARN);
if (!log) {
@ -8823,3 +8948,21 @@ errout:
}
EXPORT_SYMBOL_GPL(__register_bpf_struct_ops);
#endif
bool btf_param_match_suffix(const struct btf *btf,
const struct btf_param *arg,
const char *suffix)
{
int suffix_len = strlen(suffix), len;
const char *param_name;
/* In the future, this can be ported to use BTF tagging */
param_name = btf_name_by_offset(btf, arg->name_off);
if (str_is_empty(param_name))
return false;
len = strlen(param_name);
if (len <= suffix_len)
return false;
param_name += len - suffix_len;
return !strncmp(param_name, suffix, suffix_len);
}

3
kernel/bpf/cgroup.c
View File

@ -1364,9 +1364,6 @@ int __cgroup_bpf_run_filter_skb(struct sock *sk,
struct cgroup *cgrp;
int ret;
if (!sk || !sk_fullsock(sk))
return 0;
if (sk->sk_family != AF_INET && sk->sk_family != AF_INET6)
return 0;

4
kernel/bpf/cpumask.c
View File

@ -424,7 +424,7 @@ __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask)
__bpf_kfunc_end_defs();
BTF_SET8_START(cpumask_kfunc_btf_ids)
BTF_KFUNCS_START(cpumask_kfunc_btf_ids)
BTF_ID_FLAGS(func, bpf_cpumask_create, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_cpumask_release, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_cpumask_acquire, KF_ACQUIRE | KF_TRUSTED_ARGS)
@ -450,7 +450,7 @@ BTF_ID_FLAGS(func, bpf_cpumask_copy, KF_RCU)
BTF_ID_FLAGS(func, bpf_cpumask_any_distribute, KF_RCU)
BTF_ID_FLAGS(func, bpf_cpumask_any_and_distribute, KF_RCU)
BTF_ID_FLAGS(func, bpf_cpumask_weight, KF_RCU)
BTF_SET8_END(cpumask_kfunc_btf_ids)
BTF_KFUNCS_END(cpumask_kfunc_btf_ids)
static const struct btf_kfunc_id_set cpumask_kfunc_set = {
.owner = THIS_MODULE,

16
kernel/bpf/helpers.c
View File

@ -334,7 +334,7 @@ static inline void __bpf_spin_lock_irqsave(struct bpf_spin_lock *lock)
__this_cpu_write(irqsave_flags, flags);
}
notrace BPF_CALL_1(bpf_spin_lock, struct bpf_spin_lock *, lock)
NOTRACE_BPF_CALL_1(bpf_spin_lock, struct bpf_spin_lock *, lock)
{
__bpf_spin_lock_irqsave(lock);
return 0;
@ -357,7 +357,7 @@ static inline void __bpf_spin_unlock_irqrestore(struct bpf_spin_lock *lock)
local_irq_restore(flags);
}
notrace BPF_CALL_1(bpf_spin_unlock, struct bpf_spin_lock *, lock)
NOTRACE_BPF_CALL_1(bpf_spin_unlock, struct bpf_spin_lock *, lock)
{
__bpf_spin_unlock_irqrestore(lock);
return 0;
@ -2487,9 +2487,9 @@ __bpf_kfunc void *bpf_cast_to_kern_ctx(void *obj)
return obj;
}
__bpf_kfunc void *bpf_rdonly_cast(void *obj__ign, u32 btf_id__k)
__bpf_kfunc void *bpf_rdonly_cast(const void *obj__ign, u32 btf_id__k)
{
return obj__ign;
return (void *)obj__ign;
}
__bpf_kfunc void bpf_rcu_read_lock(void)
@ -2547,7 +2547,7 @@ __bpf_kfunc void bpf_throw(u64 cookie)
__bpf_kfunc_end_defs();
BTF_SET8_START(generic_btf_ids)
BTF_KFUNCS_START(generic_btf_ids)
#ifdef CONFIG_KEXEC_CORE
BTF_ID_FLAGS(func, crash_kexec, KF_DESTRUCTIVE)
#endif
@ -2576,7 +2576,7 @@ BTF_ID_FLAGS(func, bpf_task_get_cgroup1, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
#endif
BTF_ID_FLAGS(func, bpf_task_from_pid, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_throw)
BTF_SET8_END(generic_btf_ids)
BTF_KFUNCS_END(generic_btf_ids)
static const struct btf_kfunc_id_set generic_kfunc_set = {
.owner = THIS_MODULE,
@ -2592,7 +2592,7 @@ BTF_ID(struct, cgroup)
BTF_ID(func, bpf_cgroup_release_dtor)
#endif
BTF_SET8_START(common_btf_ids)
BTF_KFUNCS_START(common_btf_ids)
BTF_ID_FLAGS(func, bpf_cast_to_kern_ctx)
BTF_ID_FLAGS(func, bpf_rdonly_cast)
BTF_ID_FLAGS(func, bpf_rcu_read_lock)
@ -2621,7 +2621,7 @@ BTF_ID_FLAGS(func, bpf_dynptr_is_null)
BTF_ID_FLAGS(func, bpf_dynptr_is_rdonly)
BTF_ID_FLAGS(func, bpf_dynptr_size)
BTF_ID_FLAGS(func, bpf_dynptr_clone)
BTF_SET8_END(common_btf_ids)
BTF_KFUNCS_END(common_btf_ids)
static const struct btf_kfunc_id_set common_kfunc_set = {
.owner = THIS_MODULE,

62
kernel/bpf/log.c
View File

@ -9,6 +9,7 @@
#include <linux/bpf.h>
#include <linux/bpf_verifier.h>
#include <linux/math64.h>
#include <linux/string.h>
#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
@ -333,7 +334,8 @@ find_linfo(const struct bpf_verifier_env *env, u32 insn_off)
{
const struct bpf_line_info *linfo;
const struct bpf_prog *prog;
u32 i, nr_linfo;
u32 nr_linfo;
int l, r, m;
prog = env->prog;
nr_linfo = prog->aux->nr_linfo;
@ -342,11 +344,30 @@ find_linfo(const struct bpf_verifier_env *env, u32 insn_off)
return NULL;
linfo = prog->aux->linfo;
for (i = 1; i < nr_linfo; i++)
if (insn_off < linfo[i].insn_off)
break;
/* Loop invariant: linfo[l].insn_off <= insns_off.
* linfo[0].insn_off == 0 which always satisfies above condition.
* Binary search is searching for rightmost linfo entry that satisfies
* the above invariant, giving us the desired record that covers given
* instruction offset.
*/
l = 0;
r = nr_linfo - 1;
while (l < r) {
/* (r - l + 1) / 2 means we break a tie to the right, so if:
* l=1, r=2, linfo[l].insn_off <= insn_off, linfo[r].insn_off > insn_off,
* then m=2, we see that linfo[m].insn_off > insn_off, and so
* r becomes 1 and we exit the loop with correct l==1.
* If the tie was broken to the left, m=1 would end us up in
* an endless loop where l and m stay at 1 and r stays at 2.
*/
m = l + (r - l + 1) / 2;
if (linfo[m].insn_off <= insn_off)
l = m;
else
r = m - 1;
}
return &linfo[i - 1];
return &linfo[l];
}
static const char *ltrim(const char *s)
@ -361,13 +382,28 @@ __printf(3, 4) void verbose_linfo(struct bpf_verifier_env *env,
u32 insn_off,
const char *prefix_fmt, ...)
{
const struct bpf_line_info *linfo;
const struct bpf_line_info *linfo, *prev_linfo;
const struct btf *btf;
const char *s, *fname;
if (!bpf_verifier_log_needed(&env->log))
return;
prev_linfo = env->prev_linfo;
linfo = find_linfo(env, insn_off);
if (!linfo || linfo == env->prev_linfo)
if (!linfo || linfo == prev_linfo)
return;
/* It often happens that two separate linfo records point to the same
* source code line, but have differing column numbers. Given verifier
* log doesn't emit column information, from user perspective we just
* end up emitting the same source code line twice unnecessarily.
* So instead check that previous and current linfo record point to
* the same file (file_name_offs match) and the same line number, and
* avoid emitting duplicated source code line in such case.
*/
if (prev_linfo && linfo->file_name_off == prev_linfo->file_name_off &&
BPF_LINE_INFO_LINE_NUM(linfo->line_col) == BPF_LINE_INFO_LINE_NUM(prev_linfo->line_col))
return;
if (prefix_fmt) {
@ -378,9 +414,15 @@ __printf(3, 4) void verbose_linfo(struct bpf_verifier_env *env,
va_end(args);
}
verbose(env, "%s\n",
ltrim(btf_name_by_offset(env->prog->aux->btf,
linfo->line_off)));
btf = env->prog->aux->btf;
s = ltrim(btf_name_by_offset(btf, linfo->line_off));
verbose(env, "%s", s); /* source code line */
s = btf_name_by_offset(btf, linfo->file_name_off);
/* leave only file name */
fname = strrchr(s, '/');
fname = fname ? fname + 1 : s;
verbose(env, " @ %s:%u\n", fname, BPF_LINE_INFO_LINE_NUM(linfo->line_col));
env->prev_linfo = linfo;
}

20
kernel/bpf/lpm_trie.c
View File

@ -164,13 +164,13 @@ static inline int extract_bit(const u8 *data, size_t index)
*/
static size_t longest_prefix_match(const struct lpm_trie *trie,
const struct lpm_trie_node *node,
const struct bpf_lpm_trie_key *key)
const struct bpf_lpm_trie_key_u8 *key)
{
u32 limit = min(node->prefixlen, key->prefixlen);
u32 prefixlen = 0, i = 0;
BUILD_BUG_ON(offsetof(struct lpm_trie_node, data) % sizeof(u32));
BUILD_BUG_ON(offsetof(struct bpf_lpm_trie_key, data) % sizeof(u32));
BUILD_BUG_ON(offsetof(struct bpf_lpm_trie_key_u8, data) % sizeof(u32));
#if defined(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) && defined(CONFIG_64BIT)
@ -229,7 +229,7 @@ static void *trie_lookup_elem(struct bpf_map *map, void *_key)
{
struct lpm_trie *trie = container_of(map, struct lpm_trie, map);
struct lpm_trie_node *node, *found = NULL;
struct bpf_lpm_trie_key *key = _key;
struct bpf_lpm_trie_key_u8 *key = _key;
if (key->prefixlen > trie->max_prefixlen)
return NULL;
@ -309,7 +309,7 @@ static long trie_update_elem(struct bpf_map *map,
struct lpm_trie *trie = container_of(map, struct lpm_trie, map);
struct lpm_trie_node *node, *im_node = NULL, *new_node = NULL;
struct lpm_trie_node __rcu **slot;
struct bpf_lpm_trie_key *key = _key;
struct bpf_lpm_trie_key_u8 *key = _key;
unsigned long irq_flags;
unsigned int next_bit;
size_t matchlen = 0;
@ -437,7 +437,7 @@ out:
static long trie_delete_elem(struct bpf_map *map, void *_key)
{
struct lpm_trie *trie = container_of(map, struct lpm_trie, map);
struct bpf_lpm_trie_key *key = _key;
struct bpf_lpm_trie_key_u8 *key = _key;
struct lpm_trie_node __rcu **trim, **trim2;
struct lpm_trie_node *node, *parent;
unsigned long irq_flags;
@ -536,7 +536,7 @@ out:
sizeof(struct lpm_trie_node))
#define LPM_VAL_SIZE_MIN 1
#define LPM_KEY_SIZE(X) (sizeof(struct bpf_lpm_trie_key) + (X))
#define LPM_KEY_SIZE(X) (sizeof(struct bpf_lpm_trie_key_u8) + (X))
#define LPM_KEY_SIZE_MAX LPM_KEY_SIZE(LPM_DATA_SIZE_MAX)
#define LPM_KEY_SIZE_MIN LPM_KEY_SIZE(LPM_DATA_SIZE_MIN)
@ -565,7 +565,7 @@ static struct bpf_map *trie_alloc(union bpf_attr *attr)
/* copy mandatory map attributes */
bpf_map_init_from_attr(&trie->map, attr);
trie->data_size = attr->key_size -
offsetof(struct bpf_lpm_trie_key, data);
offsetof(struct bpf_lpm_trie_key_u8, data);
trie->max_prefixlen = trie->data_size * 8;
spin_lock_init(&trie->lock);
@ -616,7 +616,7 @@ static int trie_get_next_key(struct bpf_map *map, void *_key, void *_next_key)
{
struct lpm_trie_node *node, *next_node = NULL, *parent, *search_root;
struct lpm_trie *trie = container_of(map, struct lpm_trie, map);
struct bpf_lpm_trie_key *key = _key, *next_key = _next_key;
struct bpf_lpm_trie_key_u8 *key = _key, *next_key = _next_key;
struct lpm_trie_node **node_stack = NULL;
int err = 0, stack_ptr = -1;
unsigned int next_bit;
@ -703,7 +703,7 @@ find_leftmost:
}
do_copy:
next_key->prefixlen = next_node->prefixlen;
memcpy((void *)next_key + offsetof(struct bpf_lpm_trie_key, data),
memcpy((void *)next_key + offsetof(struct bpf_lpm_trie_key_u8, data),
next_node->data, trie->data_size);
free_stack:
kfree(node_stack);
@ -715,7 +715,7 @@ static int trie_check_btf(const struct bpf_map *map,
const struct btf_type *key_type,
const struct btf_type *value_type)
{
/* Keys must have struct bpf_lpm_trie_key embedded. */
/* Keys must have struct bpf_lpm_trie_key_u8 embedded. */
return BTF_INFO_KIND(key_type->info) != BTF_KIND_STRUCT ?
-EINVAL : 0;
}

4
kernel/bpf/map_iter.c
View File

@ -213,9 +213,9 @@ __bpf_kfunc s64 bpf_map_sum_elem_count(const struct bpf_map *map)
__bpf_kfunc_end_defs();
BTF_SET8_START(bpf_map_iter_kfunc_ids)
BTF_KFUNCS_START(bpf_map_iter_kfunc_ids)
BTF_ID_FLAGS(func, bpf_map_sum_elem_count, KF_TRUSTED_ARGS)
BTF_SET8_END(bpf_map_iter_kfunc_ids)
BTF_KFUNCS_END(bpf_map_iter_kfunc_ids)
static const struct btf_kfunc_id_set bpf_map_iter_kfunc_set = {
.owner = THIS_MODULE,

16
kernel/bpf/token.c
View File

@ -72,28 +72,28 @@ static void bpf_token_show_fdinfo(struct seq_file *m, struct file *filp)
u64 mask;
BUILD_BUG_ON(__MAX_BPF_CMD >= 64);
mask = (1ULL << __MAX_BPF_CMD) - 1;
mask = BIT_ULL(__MAX_BPF_CMD) - 1;
if ((token->allowed_cmds & mask) == mask)
seq_printf(m, "allowed_cmds:\tany\n");
else
seq_printf(m, "allowed_cmds:\t0x%llx\n", token->allowed_cmds);
BUILD_BUG_ON(__MAX_BPF_MAP_TYPE >= 64);
mask = (1ULL << __MAX_BPF_MAP_TYPE) - 1;
mask = BIT_ULL(__MAX_BPF_MAP_TYPE) - 1;
if ((token->allowed_maps & mask) == mask)
seq_printf(m, "allowed_maps:\tany\n");
else
seq_printf(m, "allowed_maps:\t0x%llx\n", token->allowed_maps);
BUILD_BUG_ON(__MAX_BPF_PROG_TYPE >= 64);
mask = (1ULL << __MAX_BPF_PROG_TYPE) - 1;
mask = BIT_ULL(__MAX_BPF_PROG_TYPE) - 1;
if ((token->allowed_progs & mask) == mask)
seq_printf(m, "allowed_progs:\tany\n");
else
seq_printf(m, "allowed_progs:\t0x%llx\n", token->allowed_progs);
BUILD_BUG_ON(__MAX_BPF_ATTACH_TYPE >= 64);
mask = (1ULL << __MAX_BPF_ATTACH_TYPE) - 1;
mask = BIT_ULL(__MAX_BPF_ATTACH_TYPE) - 1;
if ((token->allowed_attachs & mask) == mask)
seq_printf(m, "allowed_attachs:\tany\n");
else
@ -253,7 +253,7 @@ bool bpf_token_allow_cmd(const struct bpf_token *token, enum bpf_cmd cmd)
{
if (!token)
return false;
if (!(token->allowed_cmds & (1ULL << cmd)))
if (!(token->allowed_cmds & BIT_ULL(cmd)))
return false;
return security_bpf_token_cmd(token, cmd) == 0;
}
@ -263,7 +263,7 @@ bool bpf_token_allow_map_type(const struct bpf_token *token, enum bpf_map_type t
if (!token || type >= __MAX_BPF_MAP_TYPE)
return false;
return token->allowed_maps & (1ULL << type);
return token->allowed_maps & BIT_ULL(type);
}
bool bpf_token_allow_prog_type(const struct bpf_token *token,
@ -273,6 +273,6 @@ bool bpf_token_allow_prog_type(const struct bpf_token *token,
if (!token || prog_type >= __MAX_BPF_PROG_TYPE || attach_type >= __MAX_BPF_ATTACH_TYPE)
return false;
return (token->allowed_progs & (1ULL << prog_type)) &&
(token->allowed_attachs & (1ULL << attach_type));
return (token->allowed_progs & BIT_ULL(prog_type)) &&
(token->allowed_attachs & BIT_ULL(attach_type));
}

236
kernel/bpf/verifier.c
View File

@ -528,6 +528,11 @@ static bool is_sync_callback_calling_insn(struct bpf_insn *insn)
(bpf_pseudo_kfunc_call(insn) && is_sync_callback_calling_kfunc(insn->imm));
}
static bool is_async_callback_calling_insn(struct bpf_insn *insn)
{
return bpf_helper_call(insn) && is_async_callback_calling_function(insn->imm);
}
static bool is_storage_get_function(enum bpf_func_id func_id)
{
return func_id == BPF_FUNC_sk_storage_get ||
@ -1155,6 +1160,12 @@ static bool is_spilled_scalar_reg(const struct bpf_stack_state *stack)
stack->spilled_ptr.type == SCALAR_VALUE;
}
static bool is_spilled_scalar_reg64(const struct bpf_stack_state *stack)
{
return stack->slot_type[0] == STACK_SPILL &&
stack->spilled_ptr.type == SCALAR_VALUE;
}
/* Mark stack slot as STACK_MISC, unless it is already STACK_INVALID, in which
* case they are equivalent, or it's STACK_ZERO, in which case we preserve
* more precise STACK_ZERO.
@ -2264,8 +2275,7 @@ static void __reg_assign_32_into_64(struct bpf_reg_state *reg)
}
/* Mark a register as having a completely unknown (scalar) value. */
static void __mark_reg_unknown(const struct bpf_verifier_env *env,
struct bpf_reg_state *reg)
static void __mark_reg_unknown_imprecise(struct bpf_reg_state *reg)
{
/*
* Clear type, off, and union(map_ptr, range) and
@ -2277,10 +2287,20 @@ static void __mark_reg_unknown(const struct bpf_verifier_env *env,
reg->ref_obj_id = 0;
reg->var_off = tnum_unknown;
reg->frameno = 0;
reg->precise = !env->bpf_capable;
reg->precise = false;
__mark_reg_unbounded(reg);
}
/* Mark a register as having a completely unknown (scalar) value,
* initialize .precise as true when not bpf capable.
*/
static void __mark_reg_unknown(const struct bpf_verifier_env *env,
struct bpf_reg_state *reg)
{
__mark_reg_unknown_imprecise(reg);
reg->precise = !env->bpf_capable;
}
static void mark_reg_unknown(struct bpf_verifier_env *env,
struct bpf_reg_state *regs, u32 regno)
{
@ -4380,20 +4400,6 @@ static u64 reg_const_value(struct bpf_reg_state *reg, bool subreg32)
return subreg32 ? tnum_subreg(reg->var_off).value : reg->var_off.value;
}
static bool __is_scalar_unbounded(struct bpf_reg_state *reg)
{
return tnum_is_unknown(reg->var_off) &&
reg->smin_value == S64_MIN && reg->smax_value == S64_MAX &&
reg->umin_value == 0 && reg->umax_value == U64_MAX &&
reg->s32_min_value == S32_MIN && reg->s32_max_value == S32_MAX &&
reg->u32_min_value == 0 && reg->u32_max_value == U32_MAX;
}
static bool register_is_bounded(struct bpf_reg_state *reg)
{
return reg->type == SCALAR_VALUE && !__is_scalar_unbounded(reg);
}
static bool __is_pointer_value(bool allow_ptr_leaks,
const struct bpf_reg_state *reg)
{
@ -4504,7 +4510,7 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,
return err;
mark_stack_slot_scratched(env, spi);
if (reg && !(off % BPF_REG_SIZE) && register_is_bounded(reg) && env->bpf_capable) {
if (reg && !(off % BPF_REG_SIZE) && reg->type == SCALAR_VALUE && env->bpf_capable) {
bool reg_value_fits;
reg_value_fits = get_reg_width(reg) <= BITS_PER_BYTE * size;
@ -4792,7 +4798,8 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
if (dst_regno < 0)
return 0;
if (!(off % BPF_REG_SIZE) && size == spill_size) {
if (size <= spill_size &&
bpf_stack_narrow_access_ok(off, size, spill_size)) {
/* The earlier check_reg_arg() has decided the
* subreg_def for this insn. Save it first.
*/
@ -4800,6 +4807,12 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
copy_register_state(&state->regs[dst_regno], reg);
state->regs[dst_regno].subreg_def = subreg_def;
/* Break the relation on a narrowing fill.
* coerce_reg_to_size will adjust the boundaries.
*/
if (get_reg_width(reg) > size * BITS_PER_BYTE)
state->regs[dst_regno].id = 0;
} else {
int spill_cnt = 0, zero_cnt = 0;
@ -5247,6 +5260,11 @@ bad_type:
return -EINVAL;
}
static bool in_sleepable(struct bpf_verifier_env *env)
{
return env->prog->aux->sleepable;
}
/* The non-sleepable programs and sleepable programs with explicit bpf_rcu_read_lock()
* can dereference RCU protected pointers and result is PTR_TRUSTED.
*/
@ -5254,7 +5272,7 @@ static bool in_rcu_cs(struct bpf_verifier_env *env)
{
return env->cur_state->active_rcu_lock ||
env->cur_state->active_lock.ptr ||
!env->prog->aux->sleepable;
!in_sleepable(env);
}
/* Once GCC supports btf_type_tag the following mechanism will be replaced with tag check */
@ -5806,6 +5824,17 @@ static int check_ptr_alignment(struct bpf_verifier_env *env,
strict);
}
static int round_up_stack_depth(struct bpf_verifier_env *env, int stack_depth)
{
if (env->prog->jit_requested)
return round_up(stack_depth, 16);
/* round up to 32-bytes, since this is granularity
* of interpreter stack size
*/
return round_up(max_t(u32, stack_depth, 1), 32);
}
/* starting from main bpf function walk all instructions of the function
* and recursively walk all callees that given function can call.
* Ignore jump and exit insns.
@ -5849,10 +5878,7 @@ process_func:
depth);
return -EACCES;
}
/* round up to 32-bytes, since this is granularity
* of interpreter stack size
*/
depth += round_up(max_t(u32, subprog[idx].stack_depth, 1), 32);
depth += round_up_stack_depth(env, subprog[idx].stack_depth);
if (depth > MAX_BPF_STACK) {
verbose(env, "combined stack size of %d calls is %d. Too large\n",
frame + 1, depth);
@ -5946,7 +5972,7 @@ continue_func:
*/
if (frame == 0)
return 0;
depth -= round_up(max_t(u32, subprog[idx].stack_depth, 1), 32);
depth -= round_up_stack_depth(env, subprog[idx].stack_depth);
frame--;
i = ret_insn[frame];
idx = ret_prog[frame];
@ -6077,10 +6103,10 @@ static void coerce_reg_to_size(struct bpf_reg_state *reg, int size)
* values are also truncated so we push 64-bit bounds into
* 32-bit bounds. Above were truncated < 32-bits already.
*/
if (size < 4) {
if (size < 4)
__mark_reg32_unbounded(reg);
reg_bounds_sync(reg);
}
reg_bounds_sync(reg);
}
static void set_sext64_default_val(struct bpf_reg_state *reg, int size)
@ -8236,6 +8262,7 @@ found:
switch ((int)reg->type) {
case PTR_TO_BTF_ID:
case PTR_TO_BTF_ID | PTR_TRUSTED:
case PTR_TO_BTF_ID | PTR_TRUSTED | PTR_MAYBE_NULL:
case PTR_TO_BTF_ID | MEM_RCU:
case PTR_TO_BTF_ID | PTR_MAYBE_NULL:
case PTR_TO_BTF_ID | PTR_MAYBE_NULL | MEM_RCU:
@ -9338,6 +9365,18 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog,
ret = process_dynptr_func(env, regno, -1, arg->arg_type, 0);
if (ret)
return ret;
} else if (base_type(arg->arg_type) == ARG_PTR_TO_BTF_ID) {
struct bpf_call_arg_meta meta;
int err;
if (register_is_null(reg) && type_may_be_null(arg->arg_type))
continue;
memset(&meta, 0, sizeof(meta)); /* leave func_id as zero */
err = check_reg_type(env, regno, arg->arg_type, &arg->btf_id, &meta);
err = err ?: check_func_arg_reg_off(env, reg, regno, arg->arg_type);
if (err)
return err;
} else {
bpf_log(log, "verifier bug: unrecognized arg#%d type %d\n",
i, arg->arg_type);
@ -9413,9 +9452,7 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins
return -EFAULT;
}
if (insn->code == (BPF_JMP | BPF_CALL) &&
insn->src_reg == 0 &&
insn->imm == BPF_FUNC_timer_set_callback) {
if (is_async_callback_calling_insn(insn)) {
struct bpf_verifier_state *async_cb;
/* there is no real recursion here. timer callbacks are async */
@ -9474,6 +9511,13 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
if (subprog_is_global(env, subprog)) {
const char *sub_name = subprog_name(env, subprog);
/* Only global subprogs cannot be called with a lock held. */
if (env->cur_state->active_lock.ptr) {
verbose(env, "global function calls are not allowed while holding a lock,\n"
"use static function instead\n");
return -EINVAL;
}
if (err) {
verbose(env, "Caller passes invalid args into func#%d ('%s')\n",
subprog, sub_name);
@ -10130,7 +10174,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
return -EINVAL;
}
if (!env->prog->aux->sleepable && fn->might_sleep) {
if (!in_sleepable(env) && fn->might_sleep) {
verbose(env, "helper call might sleep in a non-sleepable prog\n");
return -EINVAL;
}
@ -10160,7 +10204,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
return -EINVAL;
}
if (env->prog->aux->sleepable && is_storage_get_function(func_id))
if (in_sleepable(env) && is_storage_get_function(func_id))
env->insn_aux_data[insn_idx].storage_get_func_atomic = true;
}
@ -10656,24 +10700,6 @@ static bool is_kfunc_rcu_protected(struct bpf_kfunc_call_arg_meta *meta)
return meta->kfunc_flags & KF_RCU_PROTECTED;
}
static bool __kfunc_param_match_suffix(const struct btf *btf,
const struct btf_param *arg,
const char *suffix)
{
int suffix_len = strlen(suffix), len;
const char *param_name;
/* In the future, this can be ported to use BTF tagging */
param_name = btf_name_by_offset(btf, arg->name_off);
if (str_is_empty(param_name))
return false;
len = strlen(param_name);
if (len < suffix_len)
return false;
param_name += len - suffix_len;
return !strncmp(param_name, suffix, suffix_len);
}
static bool is_kfunc_arg_mem_size(const struct btf *btf,
const struct btf_param *arg,
const struct bpf_reg_state *reg)
@ -10684,7 +10710,7 @@ static bool is_kfunc_arg_mem_size(const struct btf *btf,
if (!btf_type_is_scalar(t) || reg->type != SCALAR_VALUE)
return false;
return __kfunc_param_match_suffix(btf, arg, "__sz");
return btf_param_match_suffix(btf, arg, "__sz");
}
static bool is_kfunc_arg_const_mem_size(const struct btf *btf,
@ -10697,47 +10723,47 @@ static bool is_kfunc_arg_const_mem_size(const struct btf *btf,
if (!btf_type_is_scalar(t) || reg->type != SCALAR_VALUE)
return false;
return __kfunc_param_match_suffix(btf, arg, "__szk");
return btf_param_match_suffix(btf, arg, "__szk");
}
static bool is_kfunc_arg_optional(const struct btf *btf, const struct btf_param *arg)
{
return __kfunc_param_match_suffix(btf, arg, "__opt");
return btf_param_match_suffix(btf, arg, "__opt");
}
static bool is_kfunc_arg_constant(const struct btf *btf, const struct btf_param *arg)
{
return __kfunc_param_match_suffix(btf, arg, "__k");
return btf_param_match_suffix(btf, arg, "__k");
}
static bool is_kfunc_arg_ignore(const struct btf *btf, const struct btf_param *arg)
{
return __kfunc_param_match_suffix(btf, arg, "__ign");
return btf_param_match_suffix(btf, arg, "__ign");
}
static bool is_kfunc_arg_alloc_obj(const struct btf *btf, const struct btf_param *arg)
{
return __kfunc_param_match_suffix(btf, arg, "__alloc");
return btf_param_match_suffix(btf, arg, "__alloc");
}
static bool is_kfunc_arg_uninit(const struct btf *btf, const struct btf_param *arg)
{
return __kfunc_param_match_suffix(btf, arg, "__uninit");
return btf_param_match_suffix(btf, arg, "__uninit");
}
static bool is_kfunc_arg_refcounted_kptr(const struct btf *btf, const struct btf_param *arg)
{
return __kfunc_param_match_suffix(btf, arg, "__refcounted_kptr");
return btf_param_match_suffix(btf, arg, "__refcounted_kptr");
}
static bool is_kfunc_arg_nullable(const struct btf *btf, const struct btf_param *arg)
{
return __kfunc_param_match_suffix(btf, arg, "__nullable");
return btf_param_match_suffix(btf, arg, "__nullable");
}
static bool is_kfunc_arg_const_str(const struct btf *btf, const struct btf_param *arg)
{
return __kfunc_param_match_suffix(btf, arg, "__str");
return btf_param_match_suffix(btf, arg, "__str");
}
static bool is_kfunc_arg_scalar_with_name(const struct btf *btf,
@ -11007,7 +11033,7 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,
* type to our caller. When a set of conditions hold in the BTF type of
* arguments, we resolve it to a known kfunc_ptr_arg_type.
*/
if (btf_get_prog_ctx_type(&env->log, meta->btf, t, resolve_prog_type(env->prog), argno))
if (btf_is_prog_ctx_type(&env->log, meta->btf, t, resolve_prog_type(env->prog), argno))
return KF_ARG_PTR_TO_CTX;
if (is_kfunc_arg_alloc_obj(meta->btf, &args[argno]))
@ -11519,7 +11545,7 @@ static bool check_css_task_iter_allowlist(struct bpf_verifier_env *env)
return true;
fallthrough;
default:
return env->prog->aux->sleepable;
return in_sleepable(env);
}
}
@ -12040,7 +12066,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
}
sleepable = is_kfunc_sleepable(&meta);
if (sleepable && !env->prog->aux->sleepable) {
if (sleepable && !in_sleepable(env)) {
verbose(env, "program must be sleepable to call sleepable kfunc %s\n", func_name);
return -EACCES;
}
@ -15567,7 +15593,7 @@ static int visit_insn(int t, struct bpf_verifier_env *env)
return DONE_EXPLORING;
case BPF_CALL:
if (insn->src_reg == 0 && insn->imm == BPF_FUNC_timer_set_callback)
if (is_async_callback_calling_insn(insn))
/* Mark this call insn as a prune point to trigger
* is_state_visited() check before call itself is
* processed by __check_func_call(). Otherwise new
@ -16483,6 +16509,43 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,
}
}
static struct bpf_reg_state unbound_reg;
static __init int unbound_reg_init(void)
{
__mark_reg_unknown_imprecise(&unbound_reg);
unbound_reg.live |= REG_LIVE_READ;
return 0;
}
late_initcall(unbound_reg_init);
static bool is_stack_all_misc(struct bpf_verifier_env *env,
struct bpf_stack_state *stack)
{
u32 i;
for (i = 0; i < ARRAY_SIZE(stack->slot_type); ++i) {
if ((stack->slot_type[i] == STACK_MISC) ||
(stack->slot_type[i] == STACK_INVALID && env->allow_uninit_stack))
continue;
return false;
}
return true;
}
static struct bpf_reg_state *scalar_reg_for_stack(struct bpf_verifier_env *env,
struct bpf_stack_state *stack)
{
if (is_spilled_scalar_reg64(stack))
return &stack->spilled_ptr;
if (is_stack_all_misc(env, stack))
return &unbound_reg;
return NULL;
}
static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
struct bpf_func_state *cur, struct bpf_idmap *idmap, bool exact)
{
@ -16521,6 +16584,20 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
if (i >= cur->allocated_stack)
return false;
/* 64-bit scalar spill vs all slots MISC and vice versa.
* Load from all slots MISC produces unbound scalar.
* Construct a fake register for such stack and call
* regsafe() to ensure scalar ids are compared.
*/
old_reg = scalar_reg_for_stack(env, &old->stack[spi]);
cur_reg = scalar_reg_for_stack(env, &cur->stack[spi]);
if (old_reg && cur_reg) {
if (!regsafe(env, old_reg, cur_reg, idmap, exact))
return false;
i += BPF_REG_SIZE - 1;
continue;
}
/* if old state was safe with misc data in the stack
* it will be safe with zero-initialized stack.
* The opposite is not true
@ -17574,7 +17651,6 @@ static int do_check(struct bpf_verifier_env *env)
if (env->cur_state->active_lock.ptr) {
if ((insn->src_reg == BPF_REG_0 && insn->imm != BPF_FUNC_spin_unlock) ||
(insn->src_reg == BPF_PSEUDO_CALL) ||
(insn->src_reg == BPF_PSEUDO_KFUNC_CALL &&
(insn->off != 0 || !is_bpf_graph_api_kfunc(insn->imm)))) {
verbose(env, "function calls are not allowed while holding a lock\n");
@ -17622,14 +17698,12 @@ static int do_check(struct bpf_verifier_env *env)
return -EINVAL;
}
process_bpf_exit_full:
if (env->cur_state->active_lock.ptr &&
!in_rbtree_lock_required_cb(env)) {
if (env->cur_state->active_lock.ptr && !env->cur_state->curframe) {
verbose(env, "bpf_spin_unlock is missing\n");
return -EINVAL;
}
if (env->cur_state->active_rcu_lock &&
!in_rbtree_lock_required_cb(env)) {
if (env->cur_state->active_rcu_lock && !env->cur_state->curframe) {
verbose(env, "bpf_rcu_read_unlock is missing\n");
return -EINVAL;
}
@ -17958,6 +18032,8 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env,
case BPF_MAP_TYPE_SK_STORAGE:
case BPF_MAP_TYPE_TASK_STORAGE:
case BPF_MAP_TYPE_CGRP_STORAGE:
case BPF_MAP_TYPE_QUEUE:
case BPF_MAP_TYPE_STACK:
break;
default:
verbose(env,
@ -19603,7 +19679,7 @@ static int do_misc_fixups(struct bpf_verifier_env *env)
}
if (is_storage_get_function(insn->imm)) {
if (!env->prog->aux->sleepable ||
if (!in_sleepable(env) ||
env->insn_aux_data[i + delta].storage_get_func_atomic)
insn_buf[0] = BPF_MOV64_IMM(BPF_REG_5, (__force __s32)GFP_ATOMIC);
else
@ -20139,6 +20215,18 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog)
mark_reg_known_zero(env, regs, i);
reg->mem_size = arg->mem_size;
reg->id = ++env->id_gen;
} else if (base_type(arg->arg_type) == ARG_PTR_TO_BTF_ID) {
reg->type = PTR_TO_BTF_ID;
if (arg->arg_type & PTR_MAYBE_NULL)
reg->type |= PTR_MAYBE_NULL;
if (arg->arg_type & PTR_UNTRUSTED)
reg->type |= PTR_UNTRUSTED;
if (arg->arg_type & PTR_TRUSTED)
reg->type |= PTR_TRUSTED;
mark_reg_known_zero(env, regs, i);
reg->btf = bpf_get_btf_vmlinux(); /* can't fail at this point */
reg->btf_id = arg->btf_id;
reg->id = ++env->id_gen;
} else {
WARN_ONCE(1, "BUG: unhandled arg#%d type %d\n",
i - BPF_REG_1, arg->arg_type);
@ -20351,6 +20439,12 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env)
}
}
/* btf_ctx_access() used this to provide argument type info */
prog->aux->ctx_arg_info =
st_ops_desc->arg_info[member_idx].info;
prog->aux->ctx_arg_info_size =
st_ops_desc->arg_info[member_idx].cnt;
prog->aux->attach_func_proto = func_proto;
prog->aux->attach_func_name = mname;
env->ops = st_ops->verifier_ops;

4
kernel/cgroup/rstat.c
View File

@ -562,10 +562,10 @@ void cgroup_base_stat_cputime_show(struct seq_file *seq)
}
/* Add bpf kfuncs for cgroup_rstat_updated() and cgroup_rstat_flush() */
BTF_SET8_START(bpf_rstat_kfunc_ids)
BTF_KFUNCS_START(bpf_rstat_kfunc_ids)
BTF_ID_FLAGS(func, cgroup_rstat_updated)
BTF_ID_FLAGS(func, cgroup_rstat_flush, KF_SLEEPABLE)
BTF_SET8_END(bpf_rstat_kfunc_ids)
BTF_KFUNCS_END(bpf_rstat_kfunc_ids)
static const struct btf_kfunc_id_set bpf_rstat_kfunc_set = {
.owner = THIS_MODULE,

6
kernel/events/core.c
View File

@ -9302,10 +9302,6 @@ void perf_event_bpf_event(struct bpf_prog *prog,
{
struct perf_bpf_event bpf_event;
if (type <= PERF_BPF_EVENT_UNKNOWN ||
type >= PERF_BPF_EVENT_MAX)
return;
switch (type) {
case PERF_BPF_EVENT_PROG_LOAD:
case PERF_BPF_EVENT_PROG_UNLOAD:
@ -9313,7 +9309,7 @@ void perf_event_bpf_event(struct bpf_prog *prog,
perf_event_bpf_emit_ksymbols(prog, type);
break;
default:
break;
return;
}
if (!atomic_read(&nr_bpf_events))

8
kernel/trace/bpf_trace.c
View File

@ -1412,14 +1412,14 @@ __bpf_kfunc int bpf_verify_pkcs7_signature(struct bpf_dynptr_kern *data_ptr,
__bpf_kfunc_end_defs();
BTF_SET8_START(key_sig_kfunc_set)
BTF_KFUNCS_START(key_sig_kfunc_set)
BTF_ID_FLAGS(func, bpf_lookup_user_key, KF_ACQUIRE | KF_RET_NULL | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_lookup_system_key, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_key_put, KF_RELEASE)
#ifdef CONFIG_SYSTEM_DATA_VERIFICATION
BTF_ID_FLAGS(func, bpf_verify_pkcs7_signature, KF_SLEEPABLE)
#endif
BTF_SET8_END(key_sig_kfunc_set)
BTF_KFUNCS_END(key_sig_kfunc_set)
static const struct btf_kfunc_id_set bpf_key_sig_kfunc_set = {
.owner = THIS_MODULE,
@ -1475,9 +1475,9 @@ __bpf_kfunc int bpf_get_file_xattr(struct file *file, const char *name__str,
__bpf_kfunc_end_defs();
BTF_SET8_START(fs_kfunc_set_ids)
BTF_KFUNCS_START(fs_kfunc_set_ids)
BTF_ID_FLAGS(func, bpf_get_file_xattr, KF_SLEEPABLE | KF_TRUSTED_ARGS)
BTF_SET8_END(fs_kfunc_set_ids)
BTF_KFUNCS_END(fs_kfunc_set_ids)
static int bpf_get_file_xattr_filter(const struct bpf_prog *prog, u32 kfunc_id)
{

8
net/bpf/test_run.c
View File

@ -617,21 +617,21 @@ CFI_NOSEAL(bpf_kfunc_call_memb_release_dtor);
__bpf_kfunc_end_defs();
BTF_SET8_START(bpf_test_modify_return_ids)
BTF_KFUNCS_START(bpf_test_modify_return_ids)
BTF_ID_FLAGS(func, bpf_modify_return_test)
BTF_ID_FLAGS(func, bpf_modify_return_test2)
BTF_ID_FLAGS(func, bpf_fentry_test1, KF_SLEEPABLE)
BTF_SET8_END(bpf_test_modify_return_ids)
BTF_KFUNCS_END(bpf_test_modify_return_ids)
static const struct btf_kfunc_id_set bpf_test_modify_return_set = {
.owner = THIS_MODULE,
.set = &bpf_test_modify_return_ids,
};
BTF_SET8_START(test_sk_check_kfunc_ids)
BTF_KFUNCS_START(test_sk_check_kfunc_ids)
BTF_ID_FLAGS(func, bpf_kfunc_call_test_release, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_kfunc_call_memb_release, KF_RELEASE)
BTF_SET8_END(test_sk_check_kfunc_ids)
BTF_KFUNCS_END(test_sk_check_kfunc_ids)
static void *bpf_test_init(const union bpf_attr *kattr, u32 user_size,
u32 size, u32 headroom, u32 tailroom)

20
net/core/filter.c
View File

@ -11982,21 +11982,21 @@ int bpf_dynptr_from_skb_rdonly(struct sk_buff *skb, u64 flags,
return 0;
}
BTF_SET8_START(bpf_kfunc_check_set_skb)
BTF_KFUNCS_START(bpf_kfunc_check_set_skb)
BTF_ID_FLAGS(func, bpf_dynptr_from_skb)
BTF_SET8_END(bpf_kfunc_check_set_skb)
BTF_KFUNCS_END(bpf_kfunc_check_set_skb)
BTF_SET8_START(bpf_kfunc_check_set_xdp)
BTF_KFUNCS_START(bpf_kfunc_check_set_xdp)
BTF_ID_FLAGS(func, bpf_dynptr_from_xdp)
BTF_SET8_END(bpf_kfunc_check_set_xdp)
BTF_KFUNCS_END(bpf_kfunc_check_set_xdp)
BTF_SET8_START(bpf_kfunc_check_set_sock_addr)
BTF_KFUNCS_START(bpf_kfunc_check_set_sock_addr)
BTF_ID_FLAGS(func, bpf_sock_addr_set_sun_path)
BTF_SET8_END(bpf_kfunc_check_set_sock_addr)
BTF_KFUNCS_END(bpf_kfunc_check_set_sock_addr)
BTF_SET8_START(bpf_kfunc_check_set_tcp_reqsk)
BTF_KFUNCS_START(bpf_kfunc_check_set_tcp_reqsk)
BTF_ID_FLAGS(func, bpf_sk_assign_tcp_reqsk, KF_TRUSTED_ARGS)
BTF_SET8_END(bpf_kfunc_check_set_tcp_reqsk)
BTF_KFUNCS_END(bpf_kfunc_check_set_tcp_reqsk)
static const struct btf_kfunc_id_set bpf_kfunc_set_skb = {
.owner = THIS_MODULE,
@ -12075,9 +12075,9 @@ __bpf_kfunc int bpf_sock_destroy(struct sock_common *sock)
__bpf_kfunc_end_defs();
BTF_SET8_START(bpf_sk_iter_kfunc_ids)
BTF_KFUNCS_START(bpf_sk_iter_kfunc_ids)
BTF_ID_FLAGS(func, bpf_sock_destroy, KF_TRUSTED_ARGS)
BTF_SET8_END(bpf_sk_iter_kfunc_ids)
BTF_KFUNCS_END(bpf_sk_iter_kfunc_ids)
static int tracing_iter_filter(const struct bpf_prog *prog, u32 kfunc_id)
{

4
net/core/xdp.c
View File

@ -771,11 +771,11 @@ __bpf_kfunc int bpf_xdp_metadata_rx_vlan_tag(const struct xdp_md *ctx,
__bpf_kfunc_end_defs();
BTF_SET8_START(xdp_metadata_kfunc_ids)
BTF_KFUNCS_START(xdp_metadata_kfunc_ids)
#define XDP_METADATA_KFUNC(_, __, name, ___) BTF_ID_FLAGS(func, name, KF_TRUSTED_ARGS)
XDP_METADATA_KFUNC_xxx
#undef XDP_METADATA_KFUNC
BTF_SET8_END(xdp_metadata_kfunc_ids)
BTF_KFUNCS_END(xdp_metadata_kfunc_ids)
static const struct btf_kfunc_id_set xdp_metadata_kfunc_set = {
.owner = THIS_MODULE,

4
net/ipv4/bpf_tcp_ca.c
View File

@ -201,13 +201,13 @@ bpf_tcp_ca_get_func_proto(enum bpf_func_id func_id,
}
}
BTF_SET8_START(bpf_tcp_ca_check_kfunc_ids)
BTF_KFUNCS_START(bpf_tcp_ca_check_kfunc_ids)
BTF_ID_FLAGS(func, tcp_reno_ssthresh)
BTF_ID_FLAGS(func, tcp_reno_cong_avoid)
BTF_ID_FLAGS(func, tcp_reno_undo_cwnd)
BTF_ID_FLAGS(func, tcp_slow_start)
BTF_ID_FLAGS(func, tcp_cong_avoid_ai)
BTF_SET8_END(bpf_tcp_ca_check_kfunc_ids)
BTF_KFUNCS_END(bpf_tcp_ca_check_kfunc_ids)
static const struct btf_kfunc_id_set bpf_tcp_ca_kfunc_set = {
.owner = THIS_MODULE,

4
net/ipv4/fou_bpf.c
View File

@ -100,10 +100,10 @@ __bpf_kfunc int bpf_skb_get_fou_encap(struct __sk_buff *skb_ctx,
__bpf_kfunc_end_defs();
BTF_SET8_START(fou_kfunc_set)
BTF_KFUNCS_START(fou_kfunc_set)
BTF_ID_FLAGS(func, bpf_skb_set_fou_encap)
BTF_ID_FLAGS(func, bpf_skb_get_fou_encap)
BTF_SET8_END(fou_kfunc_set)
BTF_KFUNCS_END(fou_kfunc_set)
static const struct btf_kfunc_id_set fou_bpf_kfunc_set = {
.owner = THIS_MODULE,

4
net/ipv4/tcp_bbr.c
View File

@ -1155,7 +1155,7 @@ static struct tcp_congestion_ops tcp_bbr_cong_ops __read_mostly = {
.set_state = bbr_set_state,
};
BTF_SET8_START(tcp_bbr_check_kfunc_ids)
BTF_KFUNCS_START(tcp_bbr_check_kfunc_ids)
#ifdef CONFIG_X86
#ifdef CONFIG_DYNAMIC_FTRACE
BTF_ID_FLAGS(func, bbr_init)
@ -1168,7 +1168,7 @@ BTF_ID_FLAGS(func, bbr_min_tso_segs)
BTF_ID_FLAGS(func, bbr_set_state)
#endif
#endif
BTF_SET8_END(tcp_bbr_check_kfunc_ids)
BTF_KFUNCS_END(tcp_bbr_check_kfunc_ids)
static const struct btf_kfunc_id_set tcp_bbr_kfunc_set = {
.owner = THIS_MODULE,

4
net/ipv4/tcp_cubic.c
View File

@ -485,7 +485,7 @@ static struct tcp_congestion_ops cubictcp __read_mostly = {
.name = "cubic",
};
BTF_SET8_START(tcp_cubic_check_kfunc_ids)
BTF_KFUNCS_START(tcp_cubic_check_kfunc_ids)
#ifdef CONFIG_X86
#ifdef CONFIG_DYNAMIC_FTRACE
BTF_ID_FLAGS(func, cubictcp_init)
@ -496,7 +496,7 @@ BTF_ID_FLAGS(func, cubictcp_cwnd_event)
BTF_ID_FLAGS(func, cubictcp_acked)
#endif
#endif
BTF_SET8_END(tcp_cubic_check_kfunc_ids)
BTF_KFUNCS_END(tcp_cubic_check_kfunc_ids)
static const struct btf_kfunc_id_set tcp_cubic_kfunc_set = {
.owner = THIS_MODULE,

4
net/ipv4/tcp_dctcp.c
View File

@ -260,7 +260,7 @@ static struct tcp_congestion_ops dctcp_reno __read_mostly = {
.name = "dctcp-reno",
};
BTF_SET8_START(tcp_dctcp_check_kfunc_ids)
BTF_KFUNCS_START(tcp_dctcp_check_kfunc_ids)
#ifdef CONFIG_X86
#ifdef CONFIG_DYNAMIC_FTRACE
BTF_ID_FLAGS(func, dctcp_init)
@ -271,7 +271,7 @@ BTF_ID_FLAGS(func, dctcp_cwnd_undo)
BTF_ID_FLAGS(func, dctcp_state)
#endif
#endif
BTF_SET8_END(tcp_dctcp_check_kfunc_ids)
BTF_KFUNCS_END(tcp_dctcp_check_kfunc_ids)
static const struct btf_kfunc_id_set tcp_dctcp_kfunc_set = {
.owner = THIS_MODULE,

4
net/netfilter/nf_conntrack_bpf.c
View File

@ -467,7 +467,7 @@ __bpf_kfunc int bpf_ct_change_status(struct nf_conn *nfct, u32 status)
__bpf_kfunc_end_defs();
BTF_SET8_START(nf_ct_kfunc_set)
BTF_KFUNCS_START(nf_ct_kfunc_set)
BTF_ID_FLAGS(func, bpf_xdp_ct_alloc, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_xdp_ct_lookup, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_skb_ct_alloc, KF_ACQUIRE | KF_RET_NULL)
@ -478,7 +478,7 @@ BTF_ID_FLAGS(func, bpf_ct_set_timeout, KF_TRUSTED_ARGS)
BTF_ID_FLAGS(func, bpf_ct_change_timeout, KF_TRUSTED_ARGS)
BTF_ID_FLAGS(func, bpf_ct_set_status, KF_TRUSTED_ARGS)
BTF_ID_FLAGS(func, bpf_ct_change_status, KF_TRUSTED_ARGS)
BTF_SET8_END(nf_ct_kfunc_set)
BTF_KFUNCS_END(nf_ct_kfunc_set)
static const struct btf_kfunc_id_set nf_conntrack_kfunc_set = {
.owner = THIS_MODULE,

4
net/netfilter/nf_nat_bpf.c
View File

@ -54,9 +54,9 @@ __bpf_kfunc int bpf_ct_set_nat_info(struct nf_conn___init *nfct,
__bpf_kfunc_end_defs();
BTF_SET8_START(nf_nat_kfunc_set)
BTF_KFUNCS_START(nf_nat_kfunc_set)
BTF_ID_FLAGS(func, bpf_ct_set_nat_info, KF_TRUSTED_ARGS)
BTF_SET8_END(nf_nat_kfunc_set)
BTF_KFUNCS_END(nf_nat_kfunc_set)
static const struct btf_kfunc_id_set nf_bpf_nat_kfunc_set = {
.owner = THIS_MODULE,

5
net/xdp/xsk.c
View File

@ -313,10 +313,13 @@ static bool xsk_is_bound(struct xdp_sock *xs)
static int xsk_rcv_check(struct xdp_sock *xs, struct xdp_buff *xdp, u32 len)
{
struct net_device *dev = xdp->rxq->dev;
u32 qid = xdp->rxq->queue_index;
if (!xsk_is_bound(xs))
return -ENXIO;
if (xs->dev != xdp->rxq->dev || xs->queue_id != xdp->rxq->queue_index)
if (!dev->_rx[qid].pool || xs->umem != dev->_rx[qid].pool->umem)
return -EINVAL;
if (len > xsk_pool_get_rx_frame_size(xs->pool) && !xs->sg) {

4
net/xfrm/xfrm_interface_bpf.c
View File

@ -93,10 +93,10 @@ __bpf_kfunc int bpf_skb_set_xfrm_info(struct __sk_buff *skb_ctx, const struct bp
__bpf_kfunc_end_defs();
BTF_SET8_START(xfrm_ifc_kfunc_set)
BTF_KFUNCS_START(xfrm_ifc_kfunc_set)
BTF_ID_FLAGS(func, bpf_skb_get_xfrm_info)
BTF_ID_FLAGS(func, bpf_skb_set_xfrm_info)
BTF_SET8_END(xfrm_ifc_kfunc_set)
BTF_KFUNCS_END(xfrm_ifc_kfunc_set)
static const struct btf_kfunc_id_set xfrm_interface_kfunc_set = {
.owner = THIS_MODULE,

4
net/xfrm/xfrm_state_bpf.c
View File

@ -117,10 +117,10 @@ __bpf_kfunc void bpf_xdp_xfrm_state_release(struct xfrm_state *x)
__bpf_kfunc_end_defs();
BTF_SET8_START(xfrm_state_kfunc_set)
BTF_KFUNCS_START(xfrm_state_kfunc_set)
BTF_ID_FLAGS(func, bpf_xdp_get_xfrm_state, KF_RET_NULL | KF_ACQUIRE)
BTF_ID_FLAGS(func, bpf_xdp_xfrm_state_release, KF_RELEASE)
BTF_SET8_END(xfrm_state_kfunc_set)
BTF_KFUNCS_END(xfrm_state_kfunc_set)
static const struct btf_kfunc_id_set xfrm_state_xdp_kfunc_set = {
.owner = THIS_MODULE,

2
samples/bpf/map_perf_test_user.c
View File

@ -370,7 +370,7 @@ static void run_perf_test(int tasks)
static void fill_lpm_trie(void)
{
struct bpf_lpm_trie_key *key;
struct bpf_lpm_trie_key_u8 *key;
unsigned long value = 0;
unsigned int i;
int r;

2
samples/bpf/xdp_router_ipv4_user.c
View File

@ -91,7 +91,7 @@ static int recv_msg(struct sockaddr_nl sock_addr, int sock)
static void read_route(struct nlmsghdr *nh, int nll)
{
char dsts[24], gws[24], ifs[16], dsts_len[24], metrics[24];
struct bpf_lpm_trie_key *prefix_key;
struct bpf_lpm_trie_key_u8 *prefix_key;
struct rtattr *rt_attr;
struct rtmsg *rt_msg;
int rtm_family;

2
scripts/bpf_doc.py
View File

@ -827,7 +827,7 @@ class PrinterHelpers(Printer):
print(' *{}{}'.format(' \t' if line else '', line))
print(' */')
print('static %s %s(*%s)(' % (self.map_type(proto['ret_type']),
print('static %s %s(* const %s)(' % (self.map_type(proto['ret_type']),
proto['ret_star'], proto['name']), end='')
comma = ''
for i, a in enumerate(proto['args']):

58
tools/bpf/bpftool/Documentation/bpftool-gen.rst
View File

@ -257,18 +257,48 @@ EXAMPLES
return 0;
}
This is example BPF application with two BPF programs and a mix of BPF maps
and global variables. Source code is split across two source code files.
**$ cat example3.bpf.c**
::
#include <linux/ptrace.h>
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
/* This header file is provided by the bpf_testmod module. */
#include "bpf_testmod.h"
int test_2_result = 0;
/* bpf_Testmod.ko calls this function, passing a "4"
* and testmod_map->data.
*/
SEC("struct_ops/test_2")
void BPF_PROG(test_2, int a, int b)
{
test_2_result = a + b;
}
SEC(".struct_ops")
struct bpf_testmod_ops testmod_map = {
.test_2 = (void *)test_2,
.data = 0x1,
};
This is example BPF application with three BPF programs and a mix of BPF
maps and global variables. Source code is split across three source code
files.
**$ clang --target=bpf -g example1.bpf.c -o example1.bpf.o**
**$ clang --target=bpf -g example2.bpf.c -o example2.bpf.o**
**$ bpftool gen object example.bpf.o example1.bpf.o example2.bpf.o**
**$ clang --target=bpf -g example3.bpf.c -o example3.bpf.o**
This set of commands compiles *example1.bpf.c* and *example2.bpf.c*
individually and then statically links respective object files into the final
BPF ELF object file *example.bpf.o*.
**$ bpftool gen object example.bpf.o example1.bpf.o example2.bpf.o example3.bpf.o**
This set of commands compiles *example1.bpf.c*, *example2.bpf.c* and
*example3.bpf.c* individually and then statically links respective object
files into the final BPF ELF object file *example.bpf.o*.
**$ bpftool gen skeleton example.bpf.o name example | tee example.skel.h**
@ -291,7 +321,15 @@ BPF ELF object file *example.bpf.o*.
struct bpf_map *data;
struct bpf_map *bss;
struct bpf_map *my_map;
struct bpf_map *testmod_map;
} maps;
struct {
struct example__testmod_map__bpf_testmod_ops {
const struct bpf_program *test_1;
const struct bpf_program *test_2;
int data;
} *testmod_map;
} struct_ops;
struct {
struct bpf_program *handle_sys_enter;
struct bpf_program *handle_sys_exit;
@ -304,6 +342,7 @@ BPF ELF object file *example.bpf.o*.
struct {
int x;
} data;
int test_2_result;
} *bss;
struct example__data {
_Bool global_flag;
@ -342,10 +381,16 @@ BPF ELF object file *example.bpf.o*.
skel->rodata->param1 = 128;
/* Change the value through the pointer of shadow type */
skel->struct_ops.testmod_map->data = 13;
err = example__load(skel);
if (err)
goto cleanup;
/* The result of the function test_2() */
printf("test_2_result: %d\n", skel->bss->test_2_result);
err = example__attach(skel);
if (err)
goto cleanup;
@ -372,6 +417,7 @@ BPF ELF object file *example.bpf.o*.
::
test_2_result: 17
my_map name: my_map
sys_enter prog FD: 8
my_static_var: 7

246
tools/bpf/bpftool/gen.c
View File

@ -7,6 +7,7 @@
#include <ctype.h>
#include <errno.h>
#include <fcntl.h>
#include <libgen.h>
#include <linux/err.h>
#include <stdbool.h>
#include <stdio.h>
@ -54,11 +55,27 @@ static bool str_has_suffix(const char *str, const char *suffix)
return true;
}
static const struct btf_type *
resolve_func_ptr(const struct btf *btf, __u32 id, __u32 *res_id)
{
const struct btf_type *t;
t = skip_mods_and_typedefs(btf, id, NULL);
if (!btf_is_ptr(t))
return NULL;
t = skip_mods_and_typedefs(btf, t->type, res_id);
return btf_is_func_proto(t) ? t : NULL;
}
static void get_obj_name(char *name, const char *file)
{
/* Using basename() GNU version which doesn't modify arg. */
strncpy(name, basename(file), MAX_OBJ_NAME_LEN - 1);
name[MAX_OBJ_NAME_LEN - 1] = '\0';
char file_copy[PATH_MAX];
/* Using basename() POSIX version to be more portable. */
strncpy(file_copy, file, PATH_MAX - 1)[PATH_MAX - 1] = '\0';
strncpy(name, basename(file_copy), MAX_OBJ_NAME_LEN - 1)[MAX_OBJ_NAME_LEN - 1] = '\0';
if (str_has_suffix(name, ".o"))
name[strlen(name) - 2] = '\0';
sanitize_identifier(name);
@ -906,6 +923,207 @@ codegen_progs_skeleton(struct bpf_object *obj, size_t prog_cnt, bool populate_li
}
}
static int walk_st_ops_shadow_vars(struct btf *btf, const char *ident,
const struct btf_type *map_type, __u32 map_type_id)
{
LIBBPF_OPTS(btf_dump_emit_type_decl_opts, opts, .indent_level = 3);
const struct btf_type *member_type;
__u32 offset, next_offset = 0;
const struct btf_member *m;
struct btf_dump *d = NULL;
const char *member_name;
__u32 member_type_id;
int i, err = 0, n;
int size;
d = btf_dump__new(btf, codegen_btf_dump_printf, NULL, NULL);
if (!d)
return -errno;
n = btf_vlen(map_type);
for (i = 0, m = btf_members(map_type); i < n; i++, m++) {
member_type = skip_mods_and_typedefs(btf, m->type, &member_type_id);
member_name = btf__name_by_offset(btf, m->name_off);
offset = m->offset / 8;
if (next_offset < offset)
printf("\t\t\tchar __padding_%d[%d];\n", i, offset - next_offset);
switch (btf_kind(member_type)) {
case BTF_KIND_INT:
case BTF_KIND_FLOAT:
case BTF_KIND_ENUM:
case BTF_KIND_ENUM64:
/* scalar type */
printf("\t\t\t");
opts.field_name = member_name;
err = btf_dump__emit_type_decl(d, member_type_id, &opts);
if (err) {
p_err("Failed to emit type declaration for %s: %d", member_name, err);
goto out;
}
printf(";\n");
size = btf__resolve_size(btf, member_type_id);
if (size < 0) {
p_err("Failed to resolve size of %s: %d\n", member_name, size);
err = size;
goto out;
}
next_offset = offset + size;
break;
case BTF_KIND_PTR:
if (resolve_func_ptr(btf, m->type, NULL)) {
/* Function pointer */
printf("\t\t\tstruct bpf_program *%s;\n", member_name);
next_offset = offset + sizeof(void *);
break;
}
/* All pointer types are unsupported except for
* function pointers.
*/
fallthrough;
default:
/* Unsupported types
*
* Types other than scalar types and function
* pointers are currently not supported in order to
* prevent conflicts in the generated code caused
* by multiple definitions. For instance, if the
* struct type FOO is used in a struct_ops map,
* bpftool has to generate definitions for FOO,
* which may result in conflicts if FOO is defined
* in different skeleton files.
*/
size = btf__resolve_size(btf, member_type_id);
if (size < 0) {
p_err("Failed to resolve size of %s: %d\n", member_name, size);
err = size;
goto out;
}
printf("\t\t\tchar __unsupported_%d[%d];\n", i, size);
next_offset = offset + size;
break;
}
}
/* Cannot fail since it must be a struct type */
size = btf__resolve_size(btf, map_type_id);
if (next_offset < (__u32)size)
printf("\t\t\tchar __padding_end[%d];\n", size - next_offset);
out:
btf_dump__free(d);
return err;
}
/* Generate the pointer of the shadow type for a struct_ops map.
*
* This function adds a pointer of the shadow type for a struct_ops map.
* The members of a struct_ops map can be exported through a pointer to a
* shadow type. The user can access these members through the pointer.
*
* A shadow type includes not all members, only members of some types.
* They are scalar types and function pointers. The function pointers are
* translated to the pointer of the struct bpf_program. The scalar types
* are translated to the original type without any modifiers.
*
* Unsupported types will be translated to a char array to occupy the same
* space as the original field, being renamed as __unsupported_*. The user
* should treat these fields as opaque data.
*/
static int gen_st_ops_shadow_type(const char *obj_name, struct btf *btf, const char *ident,
const struct bpf_map *map)
{
const struct btf_type *map_type;
const char *type_name;
__u32 map_type_id;
int err;
map_type_id = bpf_map__btf_value_type_id(map);
if (map_type_id == 0)
return -EINVAL;
map_type = btf__type_by_id(btf, map_type_id);
if (!map_type)
return -EINVAL;
type_name = btf__name_by_offset(btf, map_type->name_off);
printf("\t\tstruct %s__%s__%s {\n", obj_name, ident, type_name);
err = walk_st_ops_shadow_vars(btf, ident, map_type, map_type_id);
if (err)
return err;
printf("\t\t} *%s;\n", ident);
return 0;
}
static int gen_st_ops_shadow(const char *obj_name, struct btf *btf, struct bpf_object *obj)
{
int err, st_ops_cnt = 0;
struct bpf_map *map;
char ident[256];
if (!btf)
return 0;
/* Generate the pointers to shadow types of
* struct_ops maps.
*/
bpf_object__for_each_map(map, obj) {
if (bpf_map__type(map) != BPF_MAP_TYPE_STRUCT_OPS)
continue;
if (!get_map_ident(map, ident, sizeof(ident)))
continue;
if (st_ops_cnt == 0) /* first struct_ops map */
printf("\tstruct {\n");
st_ops_cnt++;
err = gen_st_ops_shadow_type(obj_name, btf, ident, map);
if (err)
return err;
}
if (st_ops_cnt)
printf("\t} struct_ops;\n");
return 0;
}
/* Generate the code to initialize the pointers of shadow types. */
static void gen_st_ops_shadow_init(struct btf *btf, struct bpf_object *obj)
{
struct bpf_map *map;
char ident[256];
if (!btf)
return;
/* Initialize the pointers to_ops shadow types of
* struct_ops maps.
*/
bpf_object__for_each_map(map, obj) {
if (bpf_map__type(map) != BPF_MAP_TYPE_STRUCT_OPS)
continue;
if (!get_map_ident(map, ident, sizeof(ident)))
continue;
codegen("\
\n\
obj->struct_ops.%1$s = bpf_map__initial_value(obj->maps.%1$s, NULL);\n\
\n\
", ident);
}
}
static int do_skeleton(int argc, char **argv)
{
char header_guard[MAX_OBJ_NAME_LEN + sizeof("__SKEL_H__")];
@ -1049,6 +1267,11 @@ static int do_skeleton(int argc, char **argv)
printf("\t} maps;\n");
}
btf = bpf_object__btf(obj);
err = gen_st_ops_shadow(obj_name, btf, obj);
if (err)
goto out;
if (prog_cnt) {
printf("\tstruct {\n");
bpf_object__for_each_program(prog, obj) {
@ -1072,7 +1295,6 @@ static int do_skeleton(int argc, char **argv)
printf("\t} links;\n");
}
btf = bpf_object__btf(obj);
if (btf) {
err = codegen_datasecs(obj, obj_name);
if (err)
@ -1130,6 +1352,12 @@ static int do_skeleton(int argc, char **argv)
if (err) \n\
goto err_out; \n\
\n\
", obj_name);
gen_st_ops_shadow_init(btf, obj);
codegen("\
\n\
return obj; \n\
err_out: \n\
%1$s__destroy(obj); \n\
@ -1439,6 +1667,10 @@ static int do_subskeleton(int argc, char **argv)
printf("\t} maps;\n");
}
err = gen_st_ops_shadow(obj_name, btf, obj);
if (err)
goto out;
if (prog_cnt) {
printf("\tstruct {\n");
bpf_object__for_each_program(prog, obj) {
@ -1550,6 +1782,12 @@ static int do_subskeleton(int argc, char **argv)
if (err) \n\
goto err; \n\
\n\
");
gen_st_ops_shadow_init(btf, obj);
codegen("\
\n\
return obj; \n\
err: \n\
%1$s__destroy(obj); \n\

70
tools/bpf/resolve_btfids/main.c
View File

@ -70,6 +70,7 @@
#include <sys/stat.h>
#include <fcntl.h>
#include <errno.h>
#include <linux/btf_ids.h>
#include <linux/rbtree.h>
#include <linux/zalloc.h>
#include <linux/err.h>
@ -78,7 +79,7 @@
#include <subcmd/parse-options.h>
#define BTF_IDS_SECTION ".BTF_ids"
#define BTF_ID "__BTF_ID__"
#define BTF_ID_PREFIX "__BTF_ID__"
#define BTF_STRUCT "struct"
#define BTF_UNION "union"
@ -89,6 +90,14 @@
#define ADDR_CNT 100
#if __BYTE_ORDER == __LITTLE_ENDIAN
# define ELFDATANATIVE ELFDATA2LSB
#elif __BYTE_ORDER == __BIG_ENDIAN
# define ELFDATANATIVE ELFDATA2MSB
#else
# error "Unknown machine endianness!"
#endif
struct btf_id {
struct rb_node rb_node;
char *name;
@ -116,6 +125,7 @@ struct object {
int idlist_shndx;
size_t strtabidx;
unsigned long idlist_addr;
int encoding;
} efile;
struct rb_root sets;
@ -161,7 +171,7 @@ static int eprintf(int level, int var, const char *fmt, ...)
static bool is_btf_id(const char *name)
{
return name && !strncmp(name, BTF_ID, sizeof(BTF_ID) - 1);
return name && !strncmp(name, BTF_ID_PREFIX, sizeof(BTF_ID_PREFIX) - 1);
}
static struct btf_id *btf_id__find(struct rb_root *root, const char *name)
@ -319,6 +329,7 @@ static int elf_collect(struct object *obj)
{
Elf_Scn *scn = NULL;
size_t shdrstrndx;
GElf_Ehdr ehdr;
int idx = 0;
Elf *elf;
int fd;
@ -350,6 +361,13 @@ static int elf_collect(struct object *obj)
return -1;
}
if (gelf_getehdr(obj->efile.elf, &ehdr) == NULL) {
pr_err("FAILED cannot get ELF header: %s\n",
elf_errmsg(-1));
return -1;
}
obj->efile.encoding = ehdr.e_ident[EI_DATA];
/*
* Scan all the elf sections and look for save data
* from .BTF_ids section and symbols.
@ -441,7 +459,7 @@ static int symbols_collect(struct object *obj)
* __BTF_ID__TYPE__vfs_truncate__0
* prefix = ^
*/
prefix = name + sizeof(BTF_ID) - 1;
prefix = name + sizeof(BTF_ID_PREFIX) - 1;
/* struct */
if (!strncmp(prefix, BTF_STRUCT, sizeof(BTF_STRUCT) - 1)) {
@ -649,19 +667,18 @@ static int cmp_id(const void *pa, const void *pb)
static int sets_patch(struct object *obj)
{
Elf_Data *data = obj->efile.idlist;
int *ptr = data->d_buf;
struct rb_node *next;
next = rb_first(&obj->sets);
while (next) {
unsigned long addr, idx;
struct btf_id_set8 *set8;
struct btf_id_set *set;
unsigned long addr, off;
struct btf_id *id;
int *base;
int cnt;
id = rb_entry(next, struct btf_id, rb_node);
addr = id->addr[0];
idx = addr - obj->efile.idlist_addr;
off = addr - obj->efile.idlist_addr;
/* sets are unique */
if (id->addr_cnt != 1) {
@ -670,14 +687,39 @@ static int sets_patch(struct object *obj)
return -1;
}
idx = idx / sizeof(int);
base = &ptr[idx] + (id->is_set8 ? 2 : 1);
cnt = ptr[idx];
if (id->is_set) {
set = data->d_buf + off;
qsort(set->ids, set->cnt, sizeof(set->ids[0]), cmp_id);
} else {
set8 = data->d_buf + off;
/*
* Make sure id is at the beginning of the pairs
* struct, otherwise the below qsort would not work.
*/
BUILD_BUG_ON(set8->pairs != &set8->pairs[0].id);
qsort(set8->pairs, set8->cnt, sizeof(set8->pairs[0]), cmp_id);
/*
* When ELF endianness does not match endianness of the
* host, libelf will do the translation when updating
* the ELF. This, however, corrupts SET8 flags which are
* already in the target endianness. So, let's bswap
* them to the host endianness and libelf will then
* correctly translate everything.
*/
if (obj->efile.encoding != ELFDATANATIVE) {
int i;
set8->flags = bswap_32(set8->flags);
for (i = 0; i < set8->cnt; i++) {
set8->pairs[i].flags =
bswap_32(set8->pairs[i].flags);
}
}
}
pr_debug("sorting addr %5lu: cnt %6d [%s]\n",
(idx + 1) * sizeof(int), cnt, id->name);
qsort(base, cnt, id->is_set8 ? sizeof(uint64_t) : sizeof(int), cmp_id);
off, id->is_set ? set->cnt : set8->cnt, id->name);
next = rb_next(next);
}

9
tools/include/linux/btf_ids.h
View File

@ -8,6 +8,15 @@ struct btf_id_set {
u32 ids[];
};
struct btf_id_set8 {
u32 cnt;
u32 flags;
struct {
u32 id;
u32 flags;
} pairs[];
};
#ifdef CONFIG_DEBUG_INFO_BTF
#include <linux/compiler.h> /* for __PASTE */

25
tools/include/uapi/linux/bpf.h
View File

@ -77,12 +77,29 @@ struct bpf_insn {
__s32 imm; /* signed immediate constant */
};
/* Key of an a BPF_MAP_TYPE_LPM_TRIE entry */
/* Deprecated: use struct bpf_lpm_trie_key_u8 (when the "data" member is needed for
* byte access) or struct bpf_lpm_trie_key_hdr (when using an alternative type for
* the trailing flexible array member) instead.
*/
struct bpf_lpm_trie_key {
__u32 prefixlen; /* up to 32 for AF_INET, 128 for AF_INET6 */
__u8 data[0]; /* Arbitrary size */
};
/* Header for bpf_lpm_trie_key structs */
struct bpf_lpm_trie_key_hdr {
__u32 prefixlen;
};
/* Key of an a BPF_MAP_TYPE_LPM_TRIE entry, with trailing byte array. */
struct bpf_lpm_trie_key_u8 {
union {
struct bpf_lpm_trie_key_hdr hdr;
__u32 prefixlen;
};
__u8 data[]; /* Arbitrary size */
};
struct bpf_cgroup_storage_key {
__u64 cgroup_inode_id; /* cgroup inode id */
__u32 attach_type; /* program attach type (enum bpf_attach_type) */
@ -617,7 +634,11 @@ union bpf_iter_link_info {
* to NULL to begin the batched operation. After each subsequent
* **BPF_MAP_LOOKUP_BATCH**, the caller should pass the resultant
* *out_batch* as the *in_batch* for the next operation to
* continue iteration from the current point.
* continue iteration from the current point. Both *in_batch* and
* *out_batch* must point to memory large enough to hold a key,
* except for maps of type **BPF_MAP_TYPE_{HASH, PERCPU_HASH,
* LRU_HASH, LRU_PERCPU_HASH}**, for which batch parameters
* must be at least 4 bytes wide regardless of key size.
*
* The *keys* and *values* are output parameters which must point
* to memory large enough to hold *count* items based on the key

41
tools/lib/bpf/bpf.h
View File

@ -35,7 +35,7 @@
extern "C" {
#endif
int libbpf_set_memlock_rlim(size_t memlock_bytes);
LIBBPF_API int libbpf_set_memlock_rlim(size_t memlock_bytes);
struct bpf_map_create_opts {
size_t sz; /* size of this struct for forward/backward compatibility */
@ -190,10 +190,14 @@ LIBBPF_API int bpf_map_delete_batch(int fd, const void *keys,
/**
* @brief **bpf_map_lookup_batch()** allows for batch lookup of BPF map elements.
*
* The parameter *in_batch* is the address of the first element in the batch to read.
* *out_batch* is an output parameter that should be passed as *in_batch* to subsequent
* calls to **bpf_map_lookup_batch()**. NULL can be passed for *in_batch* to indicate
* that the batched lookup starts from the beginning of the map.
* The parameter *in_batch* is the address of the first element in the batch to
* read. *out_batch* is an output parameter that should be passed as *in_batch*
* to subsequent calls to **bpf_map_lookup_batch()**. NULL can be passed for
* *in_batch* to indicate that the batched lookup starts from the beginning of
* the map. Both *in_batch* and *out_batch* must point to memory large enough to
* hold a single key, except for maps of type **BPF_MAP_TYPE_{HASH, PERCPU_HASH,
* LRU_HASH, LRU_PERCPU_HASH}**, for which the memory size must be at
* least 4 bytes wide regardless of key size.
*
* The *keys* and *values* are output parameters which must point to memory large enough to
* hold *count* items based on the key and value size of the map *map_fd*. The *keys*
@ -226,7 +230,10 @@ LIBBPF_API int bpf_map_lookup_batch(int fd, void *in_batch, void *out_batch,
*
* @param fd BPF map file descriptor
* @param in_batch address of the first element in batch to read, can pass NULL to
* get address of the first element in *out_batch*
* get address of the first element in *out_batch*. If not NULL, must be large
* enough to hold a key. For **BPF_MAP_TYPE_{HASH, PERCPU_HASH, LRU_HASH,
* LRU_PERCPU_HASH}**, the memory size must be at least 4 bytes wide regardless
* of key size.
* @param out_batch output parameter that should be passed to next call as *in_batch*
* @param keys pointer to an array of *count* keys
* @param values pointer to an array large enough for *count* values
@ -500,7 +507,10 @@ LIBBPF_API int bpf_obj_get_info_by_fd(int bpf_fd, void *info, __u32 *info_len);
* program corresponding to *prog_fd*.
*
* Populates up to *info_len* bytes of *info* and updates *info_len* with the
* actual number of bytes written to *info*.
* actual number of bytes written to *info*. Note that *info* should be
* zero-initialized or initialized as expected by the requested *info*
* type. Failing to (zero-)initialize *info* under certain circumstances can
* result in this helper returning an error.
*
* @param prog_fd BPF program file descriptor
* @param info pointer to **struct bpf_prog_info** that will be populated with
@ -517,7 +527,10 @@ LIBBPF_API int bpf_prog_get_info_by_fd(int prog_fd, struct bpf_prog_info *info,
* map corresponding to *map_fd*.
*
* Populates up to *info_len* bytes of *info* and updates *info_len* with the
* actual number of bytes written to *info*.
* actual number of bytes written to *info*. Note that *info* should be
* zero-initialized or initialized as expected by the requested *info*
* type. Failing to (zero-)initialize *info* under certain circumstances can
* result in this helper returning an error.
*
* @param map_fd BPF map file descriptor
* @param info pointer to **struct bpf_map_info** that will be populated with
@ -530,11 +543,14 @@ LIBBPF_API int bpf_prog_get_info_by_fd(int prog_fd, struct bpf_prog_info *info,
LIBBPF_API int bpf_map_get_info_by_fd(int map_fd, struct bpf_map_info *info, __u32 *info_len);
/**
* @brief **bpf_btf_get_info_by_fd()** obtains information about the
* @brief **bpf_btf_get_info_by_fd()** obtains information about the
* BTF object corresponding to *btf_fd*.
*
* Populates up to *info_len* bytes of *info* and updates *info_len* with the
* actual number of bytes written to *info*.
* actual number of bytes written to *info*. Note that *info* should be
* zero-initialized or initialized as expected by the requested *info*
* type. Failing to (zero-)initialize *info* under certain circumstances can
* result in this helper returning an error.
*
* @param btf_fd BTF object file descriptor
* @param info pointer to **struct bpf_btf_info** that will be populated with
@ -551,7 +567,10 @@ LIBBPF_API int bpf_btf_get_info_by_fd(int btf_fd, struct bpf_btf_info *info, __u
* link corresponding to *link_fd*.
*
* Populates up to *info_len* bytes of *info* and updates *info_len* with the
* actual number of bytes written to *info*.
* actual number of bytes written to *info*. Note that *info* should be
* zero-initialized or initialized as expected by the requested *info*
* type. Failing to (zero-)initialize *info* under certain circumstances can
* result in this helper returning an error.
*
* @param link_fd BPF link file descriptor
* @param info pointer to **struct bpf_link_info** that will be populated with

58
tools/lib/bpf/bpf_core_read.h
View File

@ -2,6 +2,8 @@
#ifndef __BPF_CORE_READ_H__
#define __BPF_CORE_READ_H__
#include <bpf/bpf_helpers.h>
/*
* enum bpf_field_info_kind is passed as a second argument into
* __builtin_preserve_field_info() built-in to get a specific aspect of
@ -44,7 +46,7 @@ enum bpf_enum_value_kind {
#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
#define __CORE_BITFIELD_PROBE_READ(dst, src, fld) \
bpf_probe_read_kernel( \
(void *)dst, \
(void *)dst, \
__CORE_RELO(src, fld, BYTE_SIZE), \
(const void *)src + __CORE_RELO(src, fld, BYTE_OFFSET))
#else
@ -143,8 +145,29 @@ enum bpf_enum_value_kind {
} \
})
/* Differentiator between compilers builtin implementations. This is a
* requirement due to the compiler parsing differences where GCC optimizes
* early in parsing those constructs of type pointers to the builtin specific
* type, resulting in not being possible to collect the required type
* information in the builtin expansion.
*/
#ifdef __clang__
#define ___bpf_typeof(type) ((typeof(type) *) 0)
#else
#define ___bpf_typeof1(type, NR) ({ \
extern typeof(type) *___concat(bpf_type_tmp_, NR); \
___concat(bpf_type_tmp_, NR); \
})
#define ___bpf_typeof(type) ___bpf_typeof1(type, __COUNTER__)
#endif
#ifdef __clang__
#define ___bpf_field_ref1(field) (field)
#define ___bpf_field_ref2(type, field) (((typeof(type) *)0)->field)
#define ___bpf_field_ref2(type, field) (___bpf_typeof(type)->field)
#else
#define ___bpf_field_ref1(field) (&(field))
#define ___bpf_field_ref2(type, field) (&(___bpf_typeof(type)->field))
#endif
#define ___bpf_field_ref(args...) \
___bpf_apply(___bpf_field_ref, ___bpf_narg(args))(args)
@ -194,7 +217,7 @@ enum bpf_enum_value_kind {
* BTF. Always succeeds.
*/
#define bpf_core_type_id_local(type) \
__builtin_btf_type_id(*(typeof(type) *)0, BPF_TYPE_ID_LOCAL)
__builtin_btf_type_id(*___bpf_typeof(type), BPF_TYPE_ID_LOCAL)
/*
* Convenience macro to get BTF type ID of a target kernel's type that matches
@ -204,7 +227,7 @@ enum bpf_enum_value_kind {
* - 0, if no matching type was found in a target kernel BTF.
*/
#define bpf_core_type_id_kernel(type) \
__builtin_btf_type_id(*(typeof(type) *)0, BPF_TYPE_ID_TARGET)
__builtin_btf_type_id(*___bpf_typeof(type), BPF_TYPE_ID_TARGET)
/*
* Convenience macro to check that provided named type
@ -214,7 +237,7 @@ enum bpf_enum_value_kind {
* 0, if no matching type is found.
*/
#define bpf_core_type_exists(type) \
__builtin_preserve_type_info(*(typeof(type) *)0, BPF_TYPE_EXISTS)
__builtin_preserve_type_info(*___bpf_typeof(type), BPF_TYPE_EXISTS)
/*
* Convenience macro to check that provided named type
@ -224,7 +247,7 @@ enum bpf_enum_value_kind {
* 0, if the type does not match any in the target kernel
*/
#define bpf_core_type_matches(type) \
__builtin_preserve_type_info(*(typeof(type) *)0, BPF_TYPE_MATCHES)
__builtin_preserve_type_info(*___bpf_typeof(type), BPF_TYPE_MATCHES)
/*
* Convenience macro to get the byte size of a provided named type
@ -234,7 +257,7 @@ enum bpf_enum_value_kind {
* 0, if no matching type is found.
*/
#define bpf_core_type_size(type) \
__builtin_preserve_type_info(*(typeof(type) *)0, BPF_TYPE_SIZE)
__builtin_preserve_type_info(*___bpf_typeof(type), BPF_TYPE_SIZE)
/*
* Convenience macro to check that provided enumerator value is defined in
@ -244,8 +267,13 @@ enum bpf_enum_value_kind {
* kernel's BTF;
* 0, if no matching enum and/or enum value within that enum is found.
*/
#ifdef __clang__
#define bpf_core_enum_value_exists(enum_type, enum_value) \
__builtin_preserve_enum_value(*(typeof(enum_type) *)enum_value, BPF_ENUMVAL_EXISTS)
#else
#define bpf_core_enum_value_exists(enum_type, enum_value) \
__builtin_preserve_enum_value(___bpf_typeof(enum_type), enum_value, BPF_ENUMVAL_EXISTS)
#endif
/*
* Convenience macro to get the integer value of an enumerator value in
@ -255,8 +283,13 @@ enum bpf_enum_value_kind {
* present in target kernel's BTF;
* 0, if no matching enum and/or enum value within that enum is found.
*/
#ifdef __clang__
#define bpf_core_enum_value(enum_type, enum_value) \
__builtin_preserve_enum_value(*(typeof(enum_type) *)enum_value, BPF_ENUMVAL_VALUE)
#else
#define bpf_core_enum_value(enum_type, enum_value) \
__builtin_preserve_enum_value(___bpf_typeof(enum_type), enum_value, BPF_ENUMVAL_VALUE)
#endif
/*
* bpf_core_read() abstracts away bpf_probe_read_kernel() call and captures
@ -292,6 +325,17 @@ enum bpf_enum_value_kind {
#define bpf_core_read_user_str(dst, sz, src) \
bpf_probe_read_user_str(dst, sz, (const void *)__builtin_preserve_access_index(src))
extern void *bpf_rdonly_cast(const void *obj, __u32 btf_id) __ksym __weak;
/*
* Cast provided pointer *ptr* into a pointer to a specified *type* in such
* a way that BPF verifier will become aware of associated kernel-side BTF
* type. This allows to access members of kernel types directly without the
* need to use BPF_CORE_READ() macros.
*/
#define bpf_core_cast(ptr, type) \
((typeof(type) *)bpf_rdonly_cast((ptr), bpf_core_type_id_kernel(type)))
#define ___concat(a, b) a ## b
#define ___apply(fn, n) ___concat(fn, n)
#define ___nth(_1, _2, _3, _4, _5, _6, _7, _8, _9, _10, __11, N, ...) N

2
tools/lib/bpf/bpf_helpers.h
View File

@ -190,6 +190,8 @@ enum libbpf_tristate {
#define __arg_ctx __attribute__((btf_decl_tag("arg:ctx")))
#define __arg_nonnull __attribute((btf_decl_tag("arg:nonnull")))
#define __arg_nullable __attribute((btf_decl_tag("arg:nullable")))
#define __arg_trusted __attribute((btf_decl_tag("arg:trusted")))
#ifndef ___bpf_concat
#define ___bpf_concat(a, b) a ## b

33
tools/lib/bpf/btf.c
View File

@ -1079,6 +1079,11 @@ struct btf *btf__new(const void *data, __u32 size)
return libbpf_ptr(btf_new(data, size, NULL));
}
struct btf *btf__new_split(const void *data, __u32 size, struct btf *base_btf)
{
return libbpf_ptr(btf_new(data, size, base_btf));
}
static struct btf *btf_parse_elf(const char *path, struct btf *base_btf,
struct btf_ext **btf_ext)
{
@ -3045,12 +3050,16 @@ done:
return btf_ext;
}
const void *btf_ext__get_raw_data(const struct btf_ext *btf_ext, __u32 *size)
const void *btf_ext__raw_data(const struct btf_ext *btf_ext, __u32 *size)
{
*size = btf_ext->data_size;
return btf_ext->data;
}
__attribute__((alias("btf_ext__raw_data")))
const void *btf_ext__get_raw_data(const struct btf_ext *btf_ext, __u32 *size);
struct btf_dedup;
static struct btf_dedup *btf_dedup_new(struct btf *btf, const struct btf_dedup_opts *opts);
@ -4932,10 +4941,9 @@ static int btf_dedup_remap_types(struct btf_dedup *d)
*/
struct btf *btf__load_vmlinux_btf(void)
{
const char *sysfs_btf_path = "/sys/kernel/btf/vmlinux";
/* fall back locations, trying to find vmlinux on disk */
const char *locations[] = {
/* try canonical vmlinux BTF through sysfs first */
"/sys/kernel/btf/vmlinux",
/* fall back to trying to find vmlinux on disk otherwise */
"/boot/vmlinux-%1$s",
"/lib/modules/%1$s/vmlinux-%1$s",
"/lib/modules/%1$s/build/vmlinux",
@ -4949,8 +4957,23 @@ struct btf *btf__load_vmlinux_btf(void)
struct btf *btf;
int i, err;
uname(&buf);
/* is canonical sysfs location accessible? */
if (faccessat(AT_FDCWD, sysfs_btf_path, F_OK, AT_EACCESS) < 0) {
pr_warn("kernel BTF is missing at '%s', was CONFIG_DEBUG_INFO_BTF enabled?\n",
sysfs_btf_path);
} else {
btf = btf__parse(sysfs_btf_path, NULL);
if (!btf) {
err = -errno;
pr_warn("failed to read kernel BTF from '%s': %d\n", sysfs_btf_path, err);
return libbpf_err_ptr(err);
}
pr_debug("loaded kernel BTF from '%s'\n", path);
return btf;
}
/* try fallback locations */
uname(&buf);
for (i = 0; i < ARRAY_SIZE(locations); i++) {
snprintf(path, PATH_MAX, locations[i], buf.release);

58
tools/lib/bpf/features.c
View File

@ -407,6 +407,61 @@ static int probe_kern_btf_enum64(int token_fd)
strs, sizeof(strs), token_fd));
}
static int probe_kern_arg_ctx_tag(int token_fd)
{
static const char strs[] = "\0a\0b\0arg:ctx\0";
const __u32 types[] = {
/* [1] INT */
BTF_TYPE_INT_ENC(1 /* "a" */, BTF_INT_SIGNED, 0, 32, 4),
/* [2] PTR -> VOID */
BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_PTR, 0, 0), 0),
/* [3] FUNC_PROTO `int(void *a)` */
BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_FUNC_PROTO, 0, 1), 1),
BTF_PARAM_ENC(1 /* "a" */, 2),
/* [4] FUNC 'a' -> FUNC_PROTO (main prog) */
BTF_TYPE_ENC(1 /* "a" */, BTF_INFO_ENC(BTF_KIND_FUNC, 0, BTF_FUNC_GLOBAL), 3),
/* [5] FUNC_PROTO `int(void *b __arg_ctx)` */
BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_FUNC_PROTO, 0, 1), 1),
BTF_PARAM_ENC(3 /* "b" */, 2),
/* [6] FUNC 'b' -> FUNC_PROTO (subprog) */
BTF_TYPE_ENC(3 /* "b" */, BTF_INFO_ENC(BTF_KIND_FUNC, 0, BTF_FUNC_GLOBAL), 5),
/* [7] DECL_TAG 'arg:ctx' -> func 'b' arg 'b' */
BTF_TYPE_DECL_TAG_ENC(5 /* "arg:ctx" */, 6, 0),
};
const struct bpf_insn insns[] = {
/* main prog */
BPF_CALL_REL(+1),
BPF_EXIT_INSN(),
/* global subprog */
BPF_EMIT_CALL(BPF_FUNC_get_func_ip), /* needs PTR_TO_CTX */
BPF_EXIT_INSN(),
};
const struct bpf_func_info_min func_infos[] = {
{ 0, 4 }, /* main prog -> FUNC 'a' */
{ 2, 6 }, /* subprog -> FUNC 'b' */
};
LIBBPF_OPTS(bpf_prog_load_opts, opts,
.token_fd = token_fd,
.prog_flags = token_fd ? BPF_F_TOKEN_FD : 0,
);
int prog_fd, btf_fd, insn_cnt = ARRAY_SIZE(insns);
btf_fd = libbpf__load_raw_btf((char *)types, sizeof(types), strs, sizeof(strs), token_fd);
if (btf_fd < 0)
return 0;
opts.prog_btf_fd = btf_fd;
opts.func_info = &func_infos;
opts.func_info_cnt = ARRAY_SIZE(func_infos);
opts.func_info_rec_size = sizeof(func_infos[0]);
prog_fd = bpf_prog_load(BPF_PROG_TYPE_KPROBE, "det_arg_ctx",
"GPL", insns, insn_cnt, &opts);
close(btf_fd);
return probe_fd(prog_fd);
}
typedef int (*feature_probe_fn)(int /* token_fd */);
static struct kern_feature_cache feature_cache;
@ -476,6 +531,9 @@ static struct kern_feature_desc {
[FEAT_UPROBE_MULTI_LINK] = {
"BPF multi-uprobe link support", probe_uprobe_multi_link,
},
[FEAT_ARG_CTX_TAG] = {
"kernel-side __arg_ctx tag", probe_kern_arg_ctx_tag,
},
};
bool feat_supported(struct kern_feature_cache *cache, enum kern_feature_id feat_id)

144
tools/lib/bpf/libbpf.c
View File

@ -33,6 +33,7 @@
#include <linux/filter.h>
#include <linux/limits.h>
#include <linux/perf_event.h>
#include <linux/bpf_perf_event.h>
#include <linux/ring_buffer.h>
#include <sys/epoll.h>
#include <sys/ioctl.h>
@ -1013,6 +1014,19 @@ static bool bpf_map__is_struct_ops(const struct bpf_map *map)
return map->def.type == BPF_MAP_TYPE_STRUCT_OPS;
}
static bool is_valid_st_ops_program(struct bpf_object *obj,
const struct bpf_program *prog)
{
int i;
for (i = 0; i < obj->nr_programs; i++) {
if (&obj->programs[i] == prog)
return prog->type == BPF_PROG_TYPE_STRUCT_OPS;
}
return false;
}
/* Init the map's fields that depend on kern_btf */
static int bpf_map__init_kern_struct_ops(struct bpf_map *map)
{
@ -1101,9 +1115,16 @@ static int bpf_map__init_kern_struct_ops(struct bpf_map *map)
if (btf_is_ptr(mtype)) {
struct bpf_program *prog;
prog = st_ops->progs[i];
/* Update the value from the shadow type */
prog = *(void **)mdata;
st_ops->progs[i] = prog;
if (!prog)
continue;
if (!is_valid_st_ops_program(obj, prog)) {
pr_warn("struct_ops init_kern %s: member %s is not a struct_ops program\n",
map->name, mname);
return -ENOTSUP;
}
kern_mtype = skip_mods_and_typedefs(kern_btf,
kern_mtype->type,
@ -1228,6 +1249,7 @@ static int init_struct_ops_maps(struct bpf_object *obj, const char *sec_name,
map->name = strdup(var_name);
if (!map->name)
return -ENOMEM;
map->btf_value_type_id = type_id;
map->def.type = BPF_MAP_TYPE_STRUCT_OPS;
map->def.key_size = sizeof(int);
@ -1524,11 +1546,20 @@ static Elf64_Sym *find_elf_var_sym(const struct bpf_object *obj, const char *nam
return ERR_PTR(-ENOENT);
}
/* Some versions of Android don't provide memfd_create() in their libc
* implementation, so avoid complications and just go straight to Linux
* syscall.
*/
static int sys_memfd_create(const char *name, unsigned flags)
{
return syscall(__NR_memfd_create, name, flags);
}
static int create_placeholder_fd(void)
{
int fd;
fd = ensure_good_fd(memfd_create("libbpf-placeholder-fd", MFD_CLOEXEC));
fd = ensure_good_fd(sys_memfd_create("libbpf-placeholder-fd", MFD_CLOEXEC));
if (fd < 0)
return -errno;
return fd;
@ -4660,7 +4691,7 @@ bpf_object__probe_loading(struct bpf_object *obj)
bool kernel_supports(const struct bpf_object *obj, enum kern_feature_id feat_id)
{
if (obj && obj->gen_loader)
if (obj->gen_loader)
/* To generate loader program assume the latest kernel
* to avoid doing extra prog_load, map_create syscalls.
*/
@ -4847,6 +4878,10 @@ static int bpf_object__create_map(struct bpf_object *obj, struct bpf_map *map, b
create_attr.btf_value_type_id = 0;
map->btf_key_type_id = 0;
map->btf_value_type_id = 0;
break;
case BPF_MAP_TYPE_STRUCT_OPS:
create_attr.btf_value_type_id = 0;
break;
default:
break;
}
@ -6339,6 +6374,14 @@ static struct {
/* all other program types don't have "named" context structs */
};
/* forward declarations for arch-specific underlying types of bpf_user_pt_regs_t typedef,
* for below __builtin_types_compatible_p() checks;
* with this approach we don't need any extra arch-specific #ifdef guards
*/
struct pt_regs;
struct user_pt_regs;
struct user_regs_struct;
static bool need_func_arg_type_fixup(const struct btf *btf, const struct bpf_program *prog,
const char *subprog_name, int arg_idx,
int arg_type_id, const char *ctx_name)
@ -6379,11 +6422,21 @@ static bool need_func_arg_type_fixup(const struct btf *btf, const struct bpf_pro
/* special cases */
switch (prog->type) {
case BPF_PROG_TYPE_KPROBE:
case BPF_PROG_TYPE_PERF_EVENT:
/* `struct pt_regs *` is expected, but we need to fix up */
if (btf_is_struct(t) && strcmp(tname, "pt_regs") == 0)
return true;
break;
case BPF_PROG_TYPE_PERF_EVENT:
if (__builtin_types_compatible_p(bpf_user_pt_regs_t, struct pt_regs) &&
btf_is_struct(t) && strcmp(tname, "pt_regs") == 0)
return true;
if (__builtin_types_compatible_p(bpf_user_pt_regs_t, struct user_pt_regs) &&
btf_is_struct(t) && strcmp(tname, "user_pt_regs") == 0)
return true;
if (__builtin_types_compatible_p(bpf_user_pt_regs_t, struct user_regs_struct) &&
btf_is_struct(t) && strcmp(tname, "user_regs_struct") == 0)
return true;
break;
case BPF_PROG_TYPE_RAW_TRACEPOINT:
case BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE:
/* allow u64* as ctx */
@ -6462,69 +6515,6 @@ static int clone_func_btf_info(struct btf *btf, int orig_fn_id, struct bpf_progr
return fn_id;
}
static int probe_kern_arg_ctx_tag(void)
{
/* To minimize merge conflicts with BPF token series that refactors
* feature detection code a lot, we don't integrate
* probe_kern_arg_ctx_tag() into kernel_supports() feature-detection
* framework yet, doing our own caching internally.
* This will be cleaned up a bit later when bpf/bpf-next trees settle.
*/
static int cached_result = -1;
static const char strs[] = "\0a\0b\0arg:ctx\0";
const __u32 types[] = {
/* [1] INT */
BTF_TYPE_INT_ENC(1 /* "a" */, BTF_INT_SIGNED, 0, 32, 4),
/* [2] PTR -> VOID */
BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_PTR, 0, 0), 0),
/* [3] FUNC_PROTO `int(void *a)` */
BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_FUNC_PROTO, 0, 1), 1),
BTF_PARAM_ENC(1 /* "a" */, 2),
/* [4] FUNC 'a' -> FUNC_PROTO (main prog) */
BTF_TYPE_ENC(1 /* "a" */, BTF_INFO_ENC(BTF_KIND_FUNC, 0, BTF_FUNC_GLOBAL), 3),
/* [5] FUNC_PROTO `int(void *b __arg_ctx)` */
BTF_TYPE_ENC(0, BTF_INFO_ENC(BTF_KIND_FUNC_PROTO, 0, 1), 1),
BTF_PARAM_ENC(3 /* "b" */, 2),
/* [6] FUNC 'b' -> FUNC_PROTO (subprog) */
BTF_TYPE_ENC(3 /* "b" */, BTF_INFO_ENC(BTF_KIND_FUNC, 0, BTF_FUNC_GLOBAL), 5),
/* [7] DECL_TAG 'arg:ctx' -> func 'b' arg 'b' */
BTF_TYPE_DECL_TAG_ENC(5 /* "arg:ctx" */, 6, 0),
};
const struct bpf_insn insns[] = {
/* main prog */
BPF_CALL_REL(+1),
BPF_EXIT_INSN(),
/* global subprog */
BPF_EMIT_CALL(BPF_FUNC_get_func_ip), /* needs PTR_TO_CTX */
BPF_EXIT_INSN(),
};
const struct bpf_func_info_min func_infos[] = {
{ 0, 4 }, /* main prog -> FUNC 'a' */
{ 2, 6 }, /* subprog -> FUNC 'b' */
};
LIBBPF_OPTS(bpf_prog_load_opts, opts);
int prog_fd, btf_fd, insn_cnt = ARRAY_SIZE(insns);
if (cached_result >= 0)
return cached_result;
btf_fd = libbpf__load_raw_btf((char *)types, sizeof(types), strs, sizeof(strs), 0);
if (btf_fd < 0)
return 0;
opts.prog_btf_fd = btf_fd;
opts.func_info = &func_infos;
opts.func_info_cnt = ARRAY_SIZE(func_infos);
opts.func_info_rec_size = sizeof(func_infos[0]);
prog_fd = bpf_prog_load(BPF_PROG_TYPE_KPROBE, "det_arg_ctx",
"GPL", insns, insn_cnt, &opts);
close(btf_fd);
cached_result = probe_fd(prog_fd);
return cached_result;
}
/* Check if main program or global subprog's function prototype has `arg:ctx`
* argument tags, and, if necessary, substitute correct type to match what BPF
* verifier would expect, taking into account specific program type. This
@ -6549,7 +6539,7 @@ static int bpf_program_fixup_func_info(struct bpf_object *obj, struct bpf_progra
return 0;
/* don't do any fix ups if kernel natively supports __arg_ctx */
if (probe_kern_arg_ctx_tag() > 0)
if (kernel_supports(obj, FEAT_ARG_CTX_TAG))
return 0;
/* some BPF program types just don't have named context structs, so
@ -9340,7 +9330,9 @@ static struct bpf_map *find_struct_ops_map_by_offset(struct bpf_object *obj,
return NULL;
}
/* Collect the reloc from ELF and populate the st_ops->progs[] */
/* Collect the reloc from ELF, populate the st_ops->progs[], and update
* st_ops->data for shadow type.
*/
static int bpf_object__collect_st_ops_relos(struct bpf_object *obj,
Elf64_Shdr *shdr, Elf_Data *data)
{
@ -9454,6 +9446,14 @@ static int bpf_object__collect_st_ops_relos(struct bpf_object *obj,
}
st_ops->progs[member_idx] = prog;
/* st_ops->data will be exposed to users, being returned by
* bpf_map__initial_value() as a pointer to the shadow
* type. All function pointers in the original struct type
* should be converted to a pointer to struct bpf_program
* in the shadow type.
*/
*((struct bpf_program **)(st_ops->data + moff)) = prog;
}
return 0;
@ -9912,6 +9912,12 @@ int bpf_map__set_initial_value(struct bpf_map *map,
void *bpf_map__initial_value(struct bpf_map *map, size_t *psize)
{
if (bpf_map__is_struct_ops(map)) {
if (psize)
*psize = map->def.value_size;
return map->st_ops->data;
}
if (!map->mmaped)
return NULL;
*psize = map->def.value_size;

5
tools/lib/bpf/libbpf.map
View File

@ -245,7 +245,6 @@ LIBBPF_0.3.0 {
btf__parse_raw_split;
btf__parse_split;
btf__new_empty_split;
btf__new_split;
ring_buffer__epoll_fd;
} LIBBPF_0.2.0;
@ -326,7 +325,6 @@ LIBBPF_0.7.0 {
bpf_xdp_detach;
bpf_xdp_query;
bpf_xdp_query_id;
btf_ext__raw_data;
libbpf_probe_bpf_helper;
libbpf_probe_bpf_map_type;
libbpf_probe_bpf_prog_type;
@ -411,5 +409,8 @@ LIBBPF_1.3.0 {
} LIBBPF_1.2.0;
LIBBPF_1.4.0 {
global:
bpf_token_create;
btf__new_split;
btf_ext__raw_data;
} LIBBPF_1.3.0;

16
tools/lib/bpf/libbpf_internal.h
View File

@ -19,6 +19,20 @@
#include <libelf.h>
#include "relo_core.h"
/* Android's libc doesn't support AT_EACCESS in faccessat() implementation
* ([0]), and just returns -EINVAL even if file exists and is accessible.
* See [1] for issues caused by this.
*
* So just redefine it to 0 on Android.
*
* [0] https://android.googlesource.com/platform/bionic/+/refs/heads/android13-release/libc/bionic/faccessat.cpp#50
* [1] https://github.com/libbpf/libbpf-bootstrap/issues/250#issuecomment-1911324250
*/
#ifdef __ANDROID__
#undef AT_EACCESS
#define AT_EACCESS 0
#endif
/* make sure libbpf doesn't use kernel-only integer typedefs */
#pragma GCC poison u8 u16 u32 u64 s8 s16 s32 s64
@ -358,6 +372,8 @@ enum kern_feature_id {
FEAT_SYSCALL_WRAPPER,
/* BPF multi-uprobe link support */
FEAT_UPROBE_MULTI_LINK,
/* Kernel supports arg:ctx tag (__arg_ctx) for global subprogs natively */
FEAT_ARG_CTX_TAG,
__FEAT_CNT,
};

2
tools/lib/bpf/linker.c
View File

@ -2732,7 +2732,7 @@ static int finalize_btf(struct bpf_linker *linker)
/* Emit .BTF.ext section */
if (linker->btf_ext) {
raw_data = btf_ext__get_raw_data(linker->btf_ext, &raw_sz);
raw_data = btf_ext__raw_data(linker->btf_ext, &raw_sz);
if (!raw_data)
return -ENOMEM;

4
tools/lib/bpf/netlink.c
View File

@ -496,8 +496,8 @@ int bpf_xdp_query(int ifindex, int xdp_flags, struct bpf_xdp_query_opts *opts)
if (err)
return libbpf_err(err);
opts->feature_flags = md.flags;
opts->xdp_zc_max_segs = md.xdp_zc_max_segs;
OPTS_SET(opts, feature_flags, md.flags);
OPTS_SET(opts, xdp_zc_max_segs, md.xdp_zc_max_segs);
skip_feature_flags:
return 0;

1
tools/testing/selftests/bpf/DENYLIST.aarch64
View File

@ -1,6 +1,5 @@
bpf_cookie/multi_kprobe_attach_api # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3
bpf_cookie/multi_kprobe_link_api # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3
exceptions # JIT does not support calling kfunc bpf_throw: -524
fexit_sleep # The test never returns. The remaining tests cannot start.
kprobe_multi_bench_attach # needs CONFIG_FPROBE
kprobe_multi_test # needs CONFIG_FPROBE

49
tools/testing/selftests/bpf/Makefile
View File

@ -41,6 +41,19 @@ CFLAGS += -g $(OPT_FLAGS) -rdynamic \
LDFLAGS += $(SAN_LDFLAGS)
LDLIBS += $(LIBELF_LIBS) -lz -lrt -lpthread
# The following tests perform type punning and they may break strict
# aliasing rules, which are exploited by both GCC and clang by default
# while optimizing. This can lead to broken programs.
progs/bind4_prog.c-CFLAGS := -fno-strict-aliasing
progs/bind6_prog.c-CFLAGS := -fno-strict-aliasing
progs/dynptr_fail.c-CFLAGS := -fno-strict-aliasing
progs/linked_list_fail.c-CFLAGS := -fno-strict-aliasing
progs/map_kptr_fail.c-CFLAGS := -fno-strict-aliasing
progs/syscall.c-CFLAGS := -fno-strict-aliasing
progs/test_pkt_md_access.c-CFLAGS := -fno-strict-aliasing
progs/test_sk_lookup.c-CFLAGS := -fno-strict-aliasing
progs/timer_crash.c-CFLAGS := -fno-strict-aliasing
ifneq ($(LLVM),)
# Silence some warnings when compiled with clang
CFLAGS += -Wno-unused-command-line-argument
@ -64,6 +77,15 @@ TEST_INST_SUBDIRS := no_alu32
ifneq ($(BPF_GCC),)
TEST_GEN_PROGS += test_progs-bpf_gcc
TEST_INST_SUBDIRS += bpf_gcc
# The following tests contain C code that, although technically legal,
# triggers GCC warnings that cannot be disabled: declaration of
# anonymous struct types in function parameter lists.
progs/btf_dump_test_case_bitfields.c-CFLAGS := -Wno-error
progs/btf_dump_test_case_namespacing.c-CFLAGS := -Wno-error
progs/btf_dump_test_case_packing.c-CFLAGS := -Wno-error
progs/btf_dump_test_case_padding.c-CFLAGS := -Wno-error
progs/btf_dump_test_case_syntax.c-CFLAGS := -Wno-error
endif
ifneq ($(CLANG_CPUV4),)
@ -110,7 +132,7 @@ TEST_GEN_PROGS_EXTENDED = test_sock_addr test_skb_cgroup_id_user \
flow_dissector_load test_flow_dissector test_tcp_check_syncookie_user \
test_lirc_mode2_user xdping test_cpp runqslower bench bpf_testmod.ko \
xskxceiver xdp_redirect_multi xdp_synproxy veristat xdp_hw_metadata \
xdp_features
xdp_features bpf_test_no_cfi.ko
TEST_GEN_FILES += liburandom_read.so urandom_read sign-file uprobe_multi
@ -175,8 +197,7 @@ endif
# NOTE: Semicolon at the end is critical to override lib.mk's default static
# rule for binaries.
$(notdir $(TEST_GEN_PROGS) \
$(TEST_GEN_PROGS_EXTENDED) \
$(TEST_CUSTOM_PROGS)): %: $(OUTPUT)/% ;
$(TEST_GEN_PROGS_EXTENDED)): %: $(OUTPUT)/% ;
# sort removes libbpf duplicates when not cross-building
MAKE_DIRS := $(sort $(BUILD_DIR)/libbpf $(HOST_BUILD_DIR)/libbpf \
@ -233,6 +254,12 @@ $(OUTPUT)/bpf_testmod.ko: $(VMLINUX_BTF) $(RESOLVE_BTFIDS) $(wildcard bpf_testmo
$(Q)$(MAKE) $(submake_extras) RESOLVE_BTFIDS=$(RESOLVE_BTFIDS) -C bpf_testmod
$(Q)cp bpf_testmod/bpf_testmod.ko $@
$(OUTPUT)/bpf_test_no_cfi.ko: $(VMLINUX_BTF) $(RESOLVE_BTFIDS) $(wildcard bpf_test_no_cfi/Makefile bpf_test_no_cfi/*.[ch])
$(call msg,MOD,,$@)
$(Q)$(RM) bpf_test_no_cfi/bpf_test_no_cfi.ko # force re-compilation
$(Q)$(MAKE) $(submake_extras) RESOLVE_BTFIDS=$(RESOLVE_BTFIDS) -C bpf_test_no_cfi
$(Q)cp bpf_test_no_cfi/bpf_test_no_cfi.ko $@
DEFAULT_BPFTOOL := $(HOST_SCRATCH_DIR)/sbin/bpftool
ifneq ($(CROSS_COMPILE),)
CROSS_BPFTOOL := $(SCRATCH_DIR)/sbin/bpftool
@ -382,11 +409,11 @@ endif
CLANG_SYS_INCLUDES = $(call get_sys_includes,$(CLANG),$(CLANG_TARGET_ARCH))
BPF_CFLAGS = -g -Wall -Werror -D__TARGET_ARCH_$(SRCARCH) $(MENDIAN) \
-I$(INCLUDE_DIR) -I$(CURDIR) -I$(APIDIR) \
-I$(abspath $(OUTPUT)/../usr/include)
-I$(abspath $(OUTPUT)/../usr/include) \
-Wno-compare-distinct-pointer-types
# TODO: enable me -Wsign-compare
CLANG_CFLAGS = $(CLANG_SYS_INCLUDES) \
-Wno-compare-distinct-pointer-types
CLANG_CFLAGS = $(CLANG_SYS_INCLUDES)
$(OUTPUT)/test_l4lb_noinline.o: BPF_CFLAGS += -fno-inline
$(OUTPUT)/test_xdp_noinline.o: BPF_CFLAGS += -fno-inline
@ -504,7 +531,8 @@ $(TRUNNER_BPF_OBJS): $(TRUNNER_OUTPUT)/%.bpf.o: \
$(wildcard $(BPFDIR)/*.bpf.h) \
| $(TRUNNER_OUTPUT) $$(BPFOBJ)
$$(call $(TRUNNER_BPF_BUILD_RULE),$$<,$$@, \
$(TRUNNER_BPF_CFLAGS))
$(TRUNNER_BPF_CFLAGS) \
$$($$<-CFLAGS))
$(TRUNNER_BPF_SKELS): %.skel.h: %.bpf.o $(BPFTOOL) | $(TRUNNER_OUTPUT)
$$(call msg,GEN-SKEL,$(TRUNNER_BINARY),$$@)
@ -514,6 +542,7 @@ $(TRUNNER_BPF_SKELS): %.skel.h: %.bpf.o $(BPFTOOL) | $(TRUNNER_OUTPUT)
$(Q)diff $$(<:.o=.linked2.o) $$(<:.o=.linked3.o)
$(Q)$$(BPFTOOL) gen skeleton $$(<:.o=.linked3.o) name $$(notdir $$(<:.bpf.o=)) > $$@
$(Q)$$(BPFTOOL) gen subskeleton $$(<:.o=.linked3.o) name $$(notdir $$(<:.bpf.o=)) > $$(@:.skel.h=.subskel.h)
$(Q)rm -f $$(<:.o=.linked1.o) $$(<:.o=.linked2.o) $$(<:.o=.linked3.o)
$(TRUNNER_BPF_LSKELS): %.lskel.h: %.bpf.o $(BPFTOOL) | $(TRUNNER_OUTPUT)
$$(call msg,GEN-SKEL,$(TRUNNER_BINARY),$$@)
@ -522,6 +551,7 @@ $(TRUNNER_BPF_LSKELS): %.lskel.h: %.bpf.o $(BPFTOOL) | $(TRUNNER_OUTPUT)
$(Q)$$(BPFTOOL) gen object $$(<:.o=.llinked3.o) $$(<:.o=.llinked2.o)
$(Q)diff $$(<:.o=.llinked2.o) $$(<:.o=.llinked3.o)
$(Q)$$(BPFTOOL) gen skeleton -L $$(<:.o=.llinked3.o) name $$(notdir $$(<:.bpf.o=_lskel)) > $$@
$(Q)rm -f $$(<:.o=.llinked1.o) $$(<:.o=.llinked2.o) $$(<:.o=.llinked3.o)
$(TRUNNER_BPF_SKELS_LINKED): $(TRUNNER_BPF_OBJS) $(BPFTOOL) | $(TRUNNER_OUTPUT)
$$(call msg,LINK-BPF,$(TRUNNER_BINARY),$$(@:.skel.h=.bpf.o))
@ -532,6 +562,7 @@ $(TRUNNER_BPF_SKELS_LINKED): $(TRUNNER_BPF_OBJS) $(BPFTOOL) | $(TRUNNER_OUTPUT)
$$(call msg,GEN-SKEL,$(TRUNNER_BINARY),$$@)
$(Q)$$(BPFTOOL) gen skeleton $$(@:.skel.h=.linked3.o) name $$(notdir $$(@:.skel.h=)) > $$@
$(Q)$$(BPFTOOL) gen subskeleton $$(@:.skel.h=.linked3.o) name $$(notdir $$(@:.skel.h=)) > $$(@:.skel.h=.subskel.h)
$(Q)rm -f $$(@:.skel.h=.linked1.o) $$(@:.skel.h=.linked2.o) $$(@:.skel.h=.linked3.o)
endif
# ensure we set up tests.h header generation rule just once
@ -606,6 +637,7 @@ TRUNNER_EXTRA_SOURCES := test_progs.c \
flow_dissector_load.h \
ip_check_defrag_frags.h
TRUNNER_EXTRA_FILES := $(OUTPUT)/urandom_read $(OUTPUT)/bpf_testmod.ko \
$(OUTPUT)/bpf_test_no_cfi.ko \
$(OUTPUT)/liburandom_read.so \
$(OUTPUT)/xdp_synproxy \
$(OUTPUT)/sign-file \
@ -729,11 +761,12 @@ $(OUTPUT)/uprobe_multi: uprobe_multi.c
$(call msg,BINARY,,$@)
$(Q)$(CC) $(CFLAGS) $(LDFLAGS) $^ $(LDLIBS) -o $@
EXTRA_CLEAN := $(TEST_CUSTOM_PROGS) $(SCRATCH_DIR) $(HOST_SCRATCH_DIR) \
EXTRA_CLEAN := $(SCRATCH_DIR) $(HOST_SCRATCH_DIR) \
prog_tests/tests.h map_tests/tests.h verifier/tests.h \
feature bpftool \
$(addprefix $(OUTPUT)/,*.o *.skel.h *.lskel.h *.subskel.h \
no_alu32 cpuv4 bpf_gcc bpf_testmod.ko \
bpf_test_no_cfi.ko \
liburandom_read.so)
.PHONY: docs docs-clean

12
tools/testing/selftests/bpf/bench.c
View File

@ -323,14 +323,14 @@ static error_t parse_arg(int key, char *arg, struct argp_state *state)
break;
case 'p':
env.producer_cnt = strtol(arg, NULL, 10);
if (env.producer_cnt <= 0) {
if (env.producer_cnt < 0) {
fprintf(stderr, "Invalid producer count: %s\n", arg);
argp_usage(state);
}
break;
case 'c':
env.consumer_cnt = strtol(arg, NULL, 10);
if (env.consumer_cnt <= 0) {
if (env.consumer_cnt < 0) {
fprintf(stderr, "Invalid consumer count: %s\n", arg);
argp_usage(state);
}
@ -607,6 +607,10 @@ static void setup_benchmark(void)
bench->setup();
for (i = 0; i < env.consumer_cnt; i++) {
if (!bench->consumer_thread) {
fprintf(stderr, "benchmark doesn't support consumers!\n");
exit(1);
}
err = pthread_create(&state.consumers[i], NULL,
bench->consumer_thread, (void *)(long)i);
if (err) {
@ -626,6 +630,10 @@ static void setup_benchmark(void)
env.prod_cpus.next_cpu = env.cons_cpus.next_cpu;
for (i = 0; i < env.producer_cnt; i++) {
if (!bench->producer_thread) {
fprintf(stderr, "benchmark doesn't support producers!\n");
exit(1);
}
err = pthread_create(&state.producers[i], NULL,
bench->producer_thread, (void *)(long)i);
if (err) {

20
tools/testing/selftests/bpf/bpf_kfuncs.h
View File

@ -9,7 +9,7 @@ struct bpf_sock_addr_kern;
* Error code
*/
extern int bpf_dynptr_from_skb(struct __sk_buff *skb, __u64 flags,
struct bpf_dynptr *ptr__uninit) __ksym;
struct bpf_dynptr *ptr__uninit) __ksym __weak;
/* Description
* Initializes an xdp-type dynptr
@ -17,7 +17,7 @@ extern int bpf_dynptr_from_skb(struct __sk_buff *skb, __u64 flags,
* Error code
*/
extern int bpf_dynptr_from_xdp(struct xdp_md *xdp, __u64 flags,
struct bpf_dynptr *ptr__uninit) __ksym;
struct bpf_dynptr *ptr__uninit) __ksym __weak;
/* Description
* Obtain a read-only pointer to the dynptr's data
@ -26,7 +26,7 @@ extern int bpf_dynptr_from_xdp(struct xdp_md *xdp, __u64 flags,
* buffer if unable to obtain a direct pointer
*/
extern void *bpf_dynptr_slice(const struct bpf_dynptr *ptr, __u32 offset,
void *buffer, __u32 buffer__szk) __ksym;
void *buffer, __u32 buffer__szk) __ksym __weak;
/* Description
* Obtain a read-write pointer to the dynptr's data
@ -35,13 +35,13 @@ extern void *bpf_dynptr_slice(const struct bpf_dynptr *ptr, __u32 offset,
* buffer if unable to obtain a direct pointer
*/
extern void *bpf_dynptr_slice_rdwr(const struct bpf_dynptr *ptr, __u32 offset,
void *buffer, __u32 buffer__szk) __ksym;
void *buffer, __u32 buffer__szk) __ksym __weak;
extern int bpf_dynptr_adjust(const struct bpf_dynptr *ptr, __u32 start, __u32 end) __ksym;
extern bool bpf_dynptr_is_null(const struct bpf_dynptr *ptr) __ksym;
extern bool bpf_dynptr_is_rdonly(const struct bpf_dynptr *ptr) __ksym;
extern __u32 bpf_dynptr_size(const struct bpf_dynptr *ptr) __ksym;
extern int bpf_dynptr_clone(const struct bpf_dynptr *ptr, struct bpf_dynptr *clone__init) __ksym;
extern int bpf_dynptr_adjust(const struct bpf_dynptr *ptr, __u32 start, __u32 end) __ksym __weak;
extern bool bpf_dynptr_is_null(const struct bpf_dynptr *ptr) __ksym __weak;
extern bool bpf_dynptr_is_rdonly(const struct bpf_dynptr *ptr) __ksym __weak;
extern __u32 bpf_dynptr_size(const struct bpf_dynptr *ptr) __ksym __weak;
extern int bpf_dynptr_clone(const struct bpf_dynptr *ptr, struct bpf_dynptr *clone__init) __ksym __weak;
/* Description
* Modify the address of a AF_UNIX sockaddr.
@ -63,7 +63,7 @@ extern int bpf_sk_assign_tcp_reqsk(struct __sk_buff *skb, struct sock *sk,
void *bpf_cast_to_kern_ctx(void *) __ksym;
void *bpf_rdonly_cast(void *obj, __u32 btf_id) __ksym;
extern void *bpf_rdonly_cast(const void *obj, __u32 btf_id) __ksym __weak;
extern int bpf_get_file_xattr(struct file *file, const char *name,
struct bpf_dynptr *value_ptr) __ksym;

19
tools/testing/selftests/bpf/bpf_test_no_cfi/Makefile Normal file
View File

@ -0,0 +1,19 @@
BPF_TEST_NO_CFI_DIR := $(realpath $(dir $(abspath $(lastword $(MAKEFILE_LIST)))))
KDIR ?= $(abspath $(BPF_TEST_NO_CFI_DIR)/../../../../..)
ifeq ($(V),1)
Q =
else
Q = @
endif
MODULES = bpf_test_no_cfi.ko
obj-m += bpf_test_no_cfi.o
all:
+$(Q)make -C $(KDIR) M=$(BPF_TEST_NO_CFI_DIR) modules
clean:
+$(Q)make -C $(KDIR) M=$(BPF_TEST_NO_CFI_DIR) clean

84
tools/testing/selftests/bpf/bpf_test_no_cfi/bpf_test_no_cfi.c Normal file
View File

@ -0,0 +1,84 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */
#include <linux/bpf.h>
#include <linux/btf.h>
#include <linux/init.h>
#include <linux/module.h>
struct bpf_test_no_cfi_ops {
void (*fn_1)(void);
void (*fn_2)(void);
};
static int dummy_init(struct btf *btf)
{
return 0;
}
static int dummy_init_member(const struct btf_type *t,
const struct btf_member *member,
void *kdata, const void *udata)
{
return 0;
}
static int dummy_reg(void *kdata)
{
return 0;
}
static void dummy_unreg(void *kdata)
{
}
static const struct bpf_verifier_ops dummy_verifier_ops;
static void bpf_test_no_cfi_ops__fn_1(void)
{
}
static void bpf_test_no_cfi_ops__fn_2(void)
{
}
static struct bpf_test_no_cfi_ops __test_no_cif_ops = {
.fn_1 = bpf_test_no_cfi_ops__fn_1,
.fn_2 = bpf_test_no_cfi_ops__fn_2,
};
static struct bpf_struct_ops test_no_cif_ops = {
.verifier_ops = &dummy_verifier_ops,
.init = dummy_init,
.init_member = dummy_init_member,
.reg = dummy_reg,
.unreg = dummy_unreg,
.name = "bpf_test_no_cfi_ops",
.owner = THIS_MODULE,
};
static int bpf_test_no_cfi_init(void)
{
int ret;
ret = register_bpf_struct_ops(&test_no_cif_ops,
bpf_test_no_cfi_ops);
if (!ret)
return -EINVAL;
test_no_cif_ops.cfi_stubs = &__test_no_cif_ops;
ret = register_bpf_struct_ops(&test_no_cif_ops,
bpf_test_no_cfi_ops);
return ret;
}
static void bpf_test_no_cfi_exit(void)
{
}
module_init(bpf_test_no_cfi_init);
module_exit(bpf_test_no_cfi_exit);
MODULE_AUTHOR("Kuifeng Lee");
MODULE_DESCRIPTION("BPF no cfi_stubs test module");
MODULE_LICENSE("Dual BSD/GPL");

34
tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c
View File

@ -343,12 +343,12 @@ static struct bin_attribute bin_attr_bpf_testmod_file __ro_after_init = {
.write = bpf_testmod_test_write,
};
BTF_SET8_START(bpf_testmod_common_kfunc_ids)
BTF_KFUNCS_START(bpf_testmod_common_kfunc_ids)
BTF_ID_FLAGS(func, bpf_iter_testmod_seq_new, KF_ITER_NEW)
BTF_ID_FLAGS(func, bpf_iter_testmod_seq_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_testmod_seq_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_kfunc_common_test)
BTF_SET8_END(bpf_testmod_common_kfunc_ids)
BTF_KFUNCS_END(bpf_testmod_common_kfunc_ids)
static const struct btf_kfunc_id_set bpf_testmod_common_kfunc_set = {
.owner = THIS_MODULE,
@ -494,7 +494,7 @@ __bpf_kfunc static u32 bpf_kfunc_call_test_static_unused_arg(u32 arg, u32 unused
return arg;
}
BTF_SET8_START(bpf_testmod_check_kfunc_ids)
BTF_KFUNCS_START(bpf_testmod_check_kfunc_ids)
BTF_ID_FLAGS(func, bpf_testmod_test_mod_kfunc)
BTF_ID_FLAGS(func, bpf_kfunc_call_test1)
BTF_ID_FLAGS(func, bpf_kfunc_call_test2)
@ -520,7 +520,7 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test_ref, KF_TRUSTED_ARGS | KF_RCU)
BTF_ID_FLAGS(func, bpf_kfunc_call_test_destructive, KF_DESTRUCTIVE)
BTF_ID_FLAGS(func, bpf_kfunc_call_test_static_unused_arg)
BTF_ID_FLAGS(func, bpf_kfunc_call_test_offset)
BTF_SET8_END(bpf_testmod_check_kfunc_ids)
BTF_KFUNCS_END(bpf_testmod_check_kfunc_ids)
static int bpf_testmod_ops_init(struct btf *btf)
{
@ -539,6 +539,15 @@ static int bpf_testmod_ops_init_member(const struct btf_type *t,
const struct btf_member *member,
void *kdata, const void *udata)
{
if (member->offset == offsetof(struct bpf_testmod_ops, data) * 8) {
/* For data fields, this function has to copy it and return
* 1 to indicate that the data has been handled by the
* struct_ops type, or the verifier will reject the map if
* the value of the data field is not zero.
*/
((struct bpf_testmod_ops *)kdata)->data = ((struct bpf_testmod_ops *)udata)->data;
return 1;
}
return 0;
}
@ -554,9 +563,12 @@ static const struct bpf_verifier_ops bpf_testmod_verifier_ops = {
static int bpf_dummy_reg(void *kdata)
{
struct bpf_testmod_ops *ops = kdata;
int r;
r = ops->test_2(4, 3);
/* Some test cases (ex. struct_ops_maybe_null) may not have test_2
* initialized, so we need to check for NULL.
*/
if (ops->test_2)
ops->test_2(4, ops->data);
return 0;
}
@ -570,7 +582,12 @@ static int bpf_testmod_test_1(void)
return 0;
}
static int bpf_testmod_test_2(int a, int b)
static void bpf_testmod_test_2(int a, int b)
{
}
static int bpf_testmod_ops__test_maybe_null(int dummy,
struct task_struct *task__nullable)
{
return 0;
}
@ -578,6 +595,7 @@ static int bpf_testmod_test_2(int a, int b)
static struct bpf_testmod_ops __bpf_testmod_ops = {
.test_1 = bpf_testmod_test_1,
.test_2 = bpf_testmod_test_2,
.test_maybe_null = bpf_testmod_ops__test_maybe_null,
};
struct bpf_struct_ops bpf_bpf_testmod_ops = {
@ -619,7 +637,7 @@ static void bpf_testmod_exit(void)
while (refcount_read(&prog_test_struct.cnt) > 1)
msleep(20);
return sysfs_remove_bin_file(kernel_kobj, &bin_attr_bpf_testmod_file);
sysfs_remove_bin_file(kernel_kobj, &bin_attr_bpf_testmod_file);
}
module_init(bpf_testmod_init);

14
tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.h
View File

@ -5,6 +5,8 @@
#include <linux/types.h>
struct task_struct;
struct bpf_testmod_test_read_ctx {
char *buf;
loff_t off;
@ -30,7 +32,17 @@ struct bpf_iter_testmod_seq {
struct bpf_testmod_ops {
int (*test_1)(void);
int (*test_2)(int a, int b);
void (*test_2)(int a, int b);
/* Used to test nullable arguments. */
int (*test_maybe_null)(int dummy, struct task_struct *task);
/* The following fields are used to test shadow copies. */
char onebyte;
struct {
int a;
int b;
} unsupported;
int data;
};
#endif /* _BPF_TESTMOD_H */

6
tools/testing/selftests/bpf/prog_tests/cpumask.c
View File

@ -27,7 +27,7 @@ static void verify_success(const char *prog_name)
struct bpf_program *prog;
struct bpf_link *link = NULL;
pid_t child_pid;
int status;
int status, err;
skel = cpumask_success__open();
if (!ASSERT_OK_PTR(skel, "cpumask_success__open"))
@ -36,8 +36,8 @@ static void verify_success(const char *prog_name)
skel->bss->pid = getpid();
skel->bss->nr_cpus = libbpf_num_possible_cpus();
cpumask_success__load(skel);
if (!ASSERT_OK_PTR(skel, "cpumask_success__load"))
err = cpumask_success__load(skel);
if (!ASSERT_OK(err, "cpumask_success__load"))
goto cleanup;
prog = bpf_object__find_program_by_name(skel->obj, prog_name);

2
tools/testing/selftests/bpf/prog_tests/decap_sanity.c
View File

@ -72,6 +72,6 @@ fail:
bpf_tc_hook_destroy(&qdisc_hook);
close_netns(nstoken);
}
SYS_NOFAIL("ip netns del " NS_TEST " &> /dev/null");
SYS_NOFAIL("ip netns del " NS_TEST);
decap_sanity__destroy(skel);
}

2
tools/testing/selftests/bpf/prog_tests/fib_lookup.c
View File

@ -298,6 +298,6 @@ void test_fib_lookup(void)
fail:
if (nstoken)
close_netns(nstoken);
SYS_NOFAIL("ip netns del " NS_TEST " &> /dev/null");
SYS_NOFAIL("ip netns del " NS_TEST);
fib_lookup__destroy(skel);
}

4
tools/testing/selftests/bpf/prog_tests/ip_check_defrag.c
View File

@ -59,9 +59,9 @@ static int setup_topology(bool ipv6)
/* Wait for up to 5s for links to come up */
for (i = 0; i < 5; ++i) {
if (ipv6)
up = !system("ip netns exec " NS0 " ping -6 -c 1 -W 1 " VETH1_ADDR6 " &>/dev/null");
up = !SYS_NOFAIL("ip netns exec " NS0 " ping -6 -c 1 -W 1 " VETH1_ADDR6);
else
up = !system("ip netns exec " NS0 " ping -c 1 -W 1 " VETH1_ADDR " &>/dev/null");
up = !SYS_NOFAIL("ip netns exec " NS0 " ping -c 1 -W 1 " VETH1_ADDR);
if (up)
break;

3
tools/testing/selftests/bpf/prog_tests/kptr_xchg_inline.c
View File

@ -13,7 +13,8 @@ void test_kptr_xchg_inline(void)
unsigned int cnt;
int err;
#if !(defined(__x86_64__) || defined(__aarch64__))
#if !(defined(__x86_64__) || defined(__aarch64__) || \
(defined(__riscv) && __riscv_xlen == 64))
test__skip();
return;
#endif

4
tools/testing/selftests/bpf/prog_tests/log_fixup.c
View File

@ -169,9 +169,9 @@ void test_log_fixup(void)
if (test__start_subtest("bad_core_relo_trunc_none"))
bad_core_relo(0, TRUNC_NONE /* full buf */);
if (test__start_subtest("bad_core_relo_trunc_partial"))
bad_core_relo(280, TRUNC_PARTIAL /* truncate original log a bit */);
bad_core_relo(300, TRUNC_PARTIAL /* truncate original log a bit */);
if (test__start_subtest("bad_core_relo_trunc_full"))
bad_core_relo(220, TRUNC_FULL /* truncate also libbpf's message patch */);
bad_core_relo(240, TRUNC_FULL /* truncate also libbpf's message patch */);
if (test__start_subtest("bad_core_relo_subprog"))
bad_core_relo_subprog();
if (test__start_subtest("missing_map"))

2
tools/testing/selftests/bpf/prog_tests/lwt_helpers.h
View File

@ -27,8 +27,6 @@
} \
})
#define NETNS "ns_lwt"
static inline int netns_create(void)
{
return system("ip netns add " NETNS);

4
tools/testing/selftests/bpf/prog_tests/lwt_redirect.c
View File

@ -54,6 +54,7 @@
#include <stdbool.h>
#include <stdlib.h>
#define NETNS "ns_lwt_redirect"
#include "lwt_helpers.h"
#include "test_progs.h"
#include "network_helpers.h"
@ -85,7 +86,7 @@ static void ping_dev(const char *dev, bool is_ingress)
snprintf(ip, sizeof(ip), "20.0.0.%d", link_index);
/* We won't get a reply. Don't fail here */
SYS_NOFAIL("ping %s -c1 -W1 -s %d >/dev/null 2>&1",
SYS_NOFAIL("ping %s -c1 -W1 -s %d",
ip, ICMP_PAYLOAD_SIZE);
}
@ -203,6 +204,7 @@ static int setup_redirect_target(const char *target_dev, bool need_mac)
if (!ASSERT_GE(target_index, 0, "if_nametoindex"))
goto fail;
SYS(fail, "sysctl -w net.ipv6.conf.all.disable_ipv6=1");
SYS(fail, "ip link add link_err type dummy");
SYS(fail, "ip link set lo up");
SYS(fail, "ip addr add dev lo " LOCAL_SRC "/32");

3
tools/testing/selftests/bpf/prog_tests/lwt_reroute.c
View File

@ -48,6 +48,7 @@
* For case 2, force UDP packets to overflow fq limit. As long as kernel
* is not crashed, it is considered successful.
*/
#define NETNS "ns_lwt_reroute"
#include "lwt_helpers.h"
#include "network_helpers.h"
#include <linux/net_tstamp.h>
@ -63,7 +64,7 @@
static void ping_once(const char *ip)
{
/* We won't get a reply. Don't fail here */
SYS_NOFAIL("ping %s -c1 -W1 -s %d >/dev/null 2>&1",
SYS_NOFAIL("ping %s -c1 -W1 -s %d",
ip, ICMP_PAYLOAD_SIZE);
}

2
tools/testing/selftests/bpf/prog_tests/mptcp.c
View File

@ -79,7 +79,7 @@ static void cleanup_netns(struct nstoken *nstoken)
if (nstoken)
close_netns(nstoken);
SYS_NOFAIL("ip netns del %s &> /dev/null", NS_TEST);
SYS_NOFAIL("ip netns del %s", NS_TEST);
}
static int verify_tsk(int map_fd, int client_fd)

6
tools/testing/selftests/bpf/prog_tests/rcu_read_lock.c
View File

@ -29,6 +29,10 @@ static void test_success(void)
bpf_program__set_autoload(skel->progs.non_sleepable_1, true);
bpf_program__set_autoload(skel->progs.non_sleepable_2, true);
bpf_program__set_autoload(skel->progs.task_trusted_non_rcuptr, true);
bpf_program__set_autoload(skel->progs.rcu_read_lock_subprog, true);
bpf_program__set_autoload(skel->progs.rcu_read_lock_global_subprog, true);
bpf_program__set_autoload(skel->progs.rcu_read_lock_subprog_lock, true);
bpf_program__set_autoload(skel->progs.rcu_read_lock_subprog_unlock, true);
err = rcu_read_lock__load(skel);
if (!ASSERT_OK(err, "skel_load"))
goto out;
@ -75,6 +79,8 @@ static const char * const inproper_region_tests[] = {
"inproper_sleepable_helper",
"inproper_sleepable_kfunc",
"nested_rcu_region",
"rcu_read_lock_global_subprog_lock",
"rcu_read_lock_global_subprog_unlock",
};
static void test_inproper_region(void)

2
tools/testing/selftests/bpf/prog_tests/sock_destroy.c
View File

@ -214,7 +214,7 @@ void test_sock_destroy(void)
cleanup:
if (nstoken)
close_netns(nstoken);
SYS_NOFAIL("ip netns del " TEST_NS " &> /dev/null");
SYS_NOFAIL("ip netns del " TEST_NS);
if (cgroup_fd >= 0)
close(cgroup_fd);
sock_destroy_prog__destroy(skel);

4
tools/testing/selftests/bpf/prog_tests/sock_iter_batch.c
View File

@ -112,7 +112,7 @@ void test_sock_iter_batch(void)
{
struct nstoken *nstoken = NULL;
SYS_NOFAIL("ip netns del " TEST_NS " &> /dev/null");
SYS_NOFAIL("ip netns del " TEST_NS);
SYS(done, "ip netns add %s", TEST_NS);
SYS(done, "ip -net %s link set dev lo up", TEST_NS);
@ -131,5 +131,5 @@ void test_sock_iter_batch(void)
close_netns(nstoken);
done:
SYS_NOFAIL("ip netns del " TEST_NS " &> /dev/null");
SYS_NOFAIL("ip netns del " TEST_NS);
}

2
tools/testing/selftests/bpf/prog_tests/spin_lock.c
View File

@ -48,6 +48,8 @@ static struct {
{ "lock_id_mismatch_innermapval_kptr", "bpf_spin_unlock of different lock" },
{ "lock_id_mismatch_innermapval_global", "bpf_spin_unlock of different lock" },
{ "lock_id_mismatch_innermapval_mapval", "bpf_spin_unlock of different lock" },
{ "lock_global_subprog_call1", "global function calls are not allowed while holding a lock" },
{ "lock_global_subprog_call2", "global function calls are not allowed while holding a lock" },
};
static int match_regex(const char *pattern, const char *string)

6
tools/testing/selftests/bpf/prog_tests/task_local_storage.c
View File

@ -117,12 +117,6 @@ static void test_recursion(void)
ASSERT_OK(err, "lookup map_b");
ASSERT_EQ(value, 100, "map_b value");
prog_fd = bpf_program__fd(skel->progs.on_lookup);
memset(&info, 0, sizeof(info));
err = bpf_prog_get_info_by_fd(prog_fd, &info, &info_len);
ASSERT_OK(err, "get prog info");
ASSERT_GT(info.recursion_misses, 0, "on_lookup prog recursion");
prog_fd = bpf_program__fd(skel->progs.on_update);
memset(&info, 0, sizeof(info));
err = bpf_prog_get_info_by_fd(prog_fd, &info, &info_len);

46
tools/testing/selftests/bpf/prog_tests/test_struct_ops_maybe_null.c Normal file
View File

@ -0,0 +1,46 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */
#include <test_progs.h>
#include "struct_ops_maybe_null.skel.h"
#include "struct_ops_maybe_null_fail.skel.h"
/* Test that the verifier accepts a program that access a nullable pointer
* with a proper check.
*/
static void maybe_null(void)
{
struct struct_ops_maybe_null *skel;
skel = struct_ops_maybe_null__open_and_load();
if (!ASSERT_OK_PTR(skel, "struct_ops_module_open_and_load"))
return;
struct_ops_maybe_null__destroy(skel);
}
/* Test that the verifier rejects a program that access a nullable pointer
* without a check beforehand.
*/
static void maybe_null_fail(void)
{
struct struct_ops_maybe_null_fail *skel;
skel = struct_ops_maybe_null_fail__open_and_load();
if (ASSERT_ERR_PTR(skel, "struct_ops_module_fail__open_and_load"))
return;
struct_ops_maybe_null_fail__destroy(skel);
}
void test_struct_ops_maybe_null(void)
{
/* The verifier verifies the programs at load time, so testing both
* programs in the same compile-unit is complicated. We run them in
* separate objects to simplify the testing.
*/
if (test__start_subtest("maybe_null"))
maybe_null();
if (test__start_subtest("maybe_null_fail"))
maybe_null_fail();
}

19
tools/testing/selftests/bpf/prog_tests/test_struct_ops_module.c
View File

@ -32,17 +32,23 @@ cleanup:
static void test_struct_ops_load(void)
{
DECLARE_LIBBPF_OPTS(bpf_object_open_opts, opts);
struct struct_ops_module *skel;
struct bpf_map_info info = {};
struct bpf_link *link;
int err;
u32 len;
skel = struct_ops_module__open_opts(&opts);
skel = struct_ops_module__open();
if (!ASSERT_OK_PTR(skel, "struct_ops_module_open"))
return;
skel->struct_ops.testmod_1->data = 13;
skel->struct_ops.testmod_1->test_2 = skel->progs.test_3;
/* Since test_2() is not being used, it should be disabled from
* auto-loading, or it will fail to load.
*/
bpf_program__set_autoload(skel->progs.test_2, false);
err = struct_ops_module__load(skel);
if (!ASSERT_OK(err, "struct_ops_module_load"))
goto cleanup;
@ -56,8 +62,13 @@ static void test_struct_ops_load(void)
link = bpf_map__attach_struct_ops(skel->maps.testmod_1);
ASSERT_OK_PTR(link, "attach_test_mod_1");
/* test_2() will be called from bpf_dummy_reg() in bpf_testmod.c */
ASSERT_EQ(skel->bss->test_2_result, 7, "test_2_result");
/* test_3() will be called from bpf_dummy_reg() in bpf_testmod.c
*
* In bpf_testmod.c it will pass 4 and 13 (the value of data) to
* .test_2. So, the value of test_2_result should be 20 (4 + 13 +
* 3).
*/
ASSERT_EQ(skel->bss->test_2_result, 20, "check_shadow_variables");
bpf_link__destroy(link);

35
tools/testing/selftests/bpf/prog_tests/test_struct_ops_no_cfi.c Normal file
View File

@ -0,0 +1,35 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */
#include <test_progs.h>
#include <testing_helpers.h>
static void load_bpf_test_no_cfi(void)
{
int fd;
int err;
fd = open("bpf_test_no_cfi.ko", O_RDONLY);
if (!ASSERT_GE(fd, 0, "open"))
return;
/* The module will try to register a struct_ops type without
* cfi_stubs and with cfi_stubs.
*
* The one without cfi_stub should fail. The module will be loaded
* successfully only if the result of the registration is as
* expected, or it fails.
*/
err = finit_module(fd, "", 0);
close(fd);
if (!ASSERT_OK(err, "finit_module"))
return;
err = delete_module("bpf_test_no_cfi", 0);
ASSERT_OK(err, "delete_module");
}
void test_struct_ops_no_cfi(void)
{
if (test__start_subtest("load_bpf_test_no_cfi"))
load_bpf_test_no_cfi();
}

18
tools/testing/selftests/bpf/prog_tests/test_tunnel.c
View File

@ -118,9 +118,9 @@ fail:
static void cleanup(void)
{
SYS_NOFAIL("test -f /var/run/netns/at_ns0 && ip netns delete at_ns0");
SYS_NOFAIL("ip link del veth1 2> /dev/null");
SYS_NOFAIL("ip link del %s 2> /dev/null", VXLAN_TUNL_DEV1);
SYS_NOFAIL("ip link del %s 2> /dev/null", IP6VXLAN_TUNL_DEV1);
SYS_NOFAIL("ip link del veth1");
SYS_NOFAIL("ip link del %s", VXLAN_TUNL_DEV1);
SYS_NOFAIL("ip link del %s", IP6VXLAN_TUNL_DEV1);
}
static int add_vxlan_tunnel(void)
@ -265,9 +265,9 @@ fail:
static void delete_ipip_tunnel(void)
{
SYS_NOFAIL("ip -n at_ns0 link delete dev %s", IPIP_TUNL_DEV0);
SYS_NOFAIL("ip -n at_ns0 fou del port 5555 2> /dev/null");
SYS_NOFAIL("ip -n at_ns0 fou del port 5555");
SYS_NOFAIL("ip link delete dev %s", IPIP_TUNL_DEV1);
SYS_NOFAIL("ip fou del port 5555 2> /dev/null");
SYS_NOFAIL("ip fou del port 5555");
}
static int add_xfrm_tunnel(void)
@ -346,13 +346,13 @@ fail:
static void delete_xfrm_tunnel(void)
{
SYS_NOFAIL("ip xfrm policy delete dir out src %s/32 dst %s/32 2> /dev/null",
SYS_NOFAIL("ip xfrm policy delete dir out src %s/32 dst %s/32",
IP4_ADDR_TUNL_DEV1, IP4_ADDR_TUNL_DEV0);
SYS_NOFAIL("ip xfrm policy delete dir in src %s/32 dst %s/32 2> /dev/null",
SYS_NOFAIL("ip xfrm policy delete dir in src %s/32 dst %s/32",
IP4_ADDR_TUNL_DEV0, IP4_ADDR_TUNL_DEV1);
SYS_NOFAIL("ip xfrm state delete src %s dst %s proto esp spi %d 2> /dev/null",
SYS_NOFAIL("ip xfrm state delete src %s dst %s proto esp spi %d",
IP4_ADDR_VETH0, IP4_ADDR1_VETH1, XFRM_SPI_IN_TO_OUT);
SYS_NOFAIL("ip xfrm state delete src %s dst %s proto esp spi %d 2> /dev/null",
SYS_NOFAIL("ip xfrm state delete src %s dst %s proto esp spi %d",
IP4_ADDR1_VETH1, IP4_ADDR_VETH0, XFRM_SPI_OUT_TO_IN);
}

37
tools/testing/selftests/bpf/prog_tests/tracing_failure.c Normal file
View File

@ -0,0 +1,37 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */
#include <test_progs.h>
#include "tracing_failure.skel.h"
static void test_bpf_spin_lock(bool is_spin_lock)
{
struct tracing_failure *skel;
int err;
skel = tracing_failure__open();
if (!ASSERT_OK_PTR(skel, "tracing_failure__open"))
return;
if (is_spin_lock)
bpf_program__set_autoload(skel->progs.test_spin_lock, true);
else
bpf_program__set_autoload(skel->progs.test_spin_unlock, true);
err = tracing_failure__load(skel);
if (!ASSERT_OK(err, "tracing_failure__load"))
goto out;
err = tracing_failure__attach(skel);
ASSERT_ERR(err, "tracing_failure__attach");
out:
tracing_failure__destroy(skel);
}
void test_tracing_failure(void)
{
if (test__start_subtest("bpf_spin_lock"))
test_bpf_spin_lock(true);
if (test__start_subtest("bpf_spin_unlock"))
test_bpf_spin_lock(false);
}

2
tools/testing/selftests/bpf/prog_tests/verifier.c
View File

@ -28,6 +28,7 @@
#include "verifier_div0.skel.h"
#include "verifier_div_overflow.skel.h"
#include "verifier_global_subprogs.skel.h"
#include "verifier_global_ptr_args.skel.h"
#include "verifier_gotol.skel.h"
#include "verifier_helper_access_var_len.skel.h"
#include "verifier_helper_packet_access.skel.h"
@ -140,6 +141,7 @@ void test_verifier_direct_stack_access_wraparound(void) { RUN(verifier_direct_st
void test_verifier_div0(void) { RUN(verifier_div0); }
void test_verifier_div_overflow(void) { RUN(verifier_div_overflow); }
void test_verifier_global_subprogs(void) { RUN(verifier_global_subprogs); }
void test_verifier_global_ptr_args(void) { RUN(verifier_global_ptr_args); }
void test_verifier_gotol(void) { RUN(verifier_gotol); }
void test_verifier_helper_access_var_len(void) { RUN(verifier_helper_access_var_len); }
void test_verifier_helper_packet_access(void) { RUN(verifier_helper_packet_access); }

4
tools/testing/selftests/bpf/progs/async_stack_depth.c
View File

@ -30,7 +30,7 @@ static int bad_timer_cb(void *map, int *key, struct bpf_timer *timer)
}
SEC("tc")
__failure __msg("combined stack size of 2 calls is 576. Too large")
__failure __msg("combined stack size of 2 calls is")
int pseudo_call_check(struct __sk_buff *ctx)
{
struct hmap_elem *elem;
@ -45,7 +45,7 @@ int pseudo_call_check(struct __sk_buff *ctx)
}
SEC("tc")
__failure __msg("combined stack size of 2 calls is 608. Too large")
__failure __msg("combined stack size of 2 calls is")
int async_call_root_check(struct __sk_buff *ctx)
{
struct hmap_elem *elem;

33
tools/testing/selftests/bpf/progs/bpf_compiler.h Normal file
View File

@ -0,0 +1,33 @@
/* SPDX-License-Identifier: GPL-2.0 */
#ifndef __BPF_COMPILER_H__
#define __BPF_COMPILER_H__
#define DO_PRAGMA_(X) _Pragma(#X)
#if __clang__
#define __pragma_loop_unroll DO_PRAGMA_(clang loop unroll(enable))
#else
/* In GCC -funroll-loops, which is enabled with -O2, should have the
same impact than the loop-unroll-enable pragma above. */
#define __pragma_loop_unroll
#endif
#if __clang__
#define __pragma_loop_unroll_count(N) DO_PRAGMA_(clang loop unroll_count(N))
#else
#define __pragma_loop_unroll_count(N) DO_PRAGMA_(GCC unroll N)
#endif
#if __clang__
#define __pragma_loop_unroll_full DO_PRAGMA_(clang loop unroll(full))
#else
#define __pragma_loop_unroll_full DO_PRAGMA_(GCC unroll 65534)
#endif
#if __clang__
#define __pragma_loop_no_unroll DO_PRAGMA_(clang loop unroll(disable))
#else
#define __pragma_loop_no_unroll DO_PRAGMA_(GCC unroll 1)
#endif
#endif

26
tools/testing/selftests/bpf/progs/cgrp_ls_recursion.c
View File

@ -27,32 +27,6 @@ bool is_cgroup1 = 0;
struct cgroup *bpf_task_get_cgroup1(struct task_struct *task, int hierarchy_id) __ksym;
void bpf_cgroup_release(struct cgroup *cgrp) __ksym;
static void __on_lookup(struct cgroup *cgrp)
{
bpf_cgrp_storage_delete(&map_a, cgrp);
bpf_cgrp_storage_delete(&map_b, cgrp);
}
SEC("fentry/bpf_local_storage_lookup")
int BPF_PROG(on_lookup)
{
struct task_struct *task = bpf_get_current_task_btf();
struct cgroup *cgrp;
if (is_cgroup1) {
cgrp = bpf_task_get_cgroup1(task, target_hid);
if (!cgrp)
return 0;
__on_lookup(cgrp);
bpf_cgroup_release(cgrp);
return 0;
}
__on_lookup(task->cgroups->dfl_cgrp);
return 0;
}
static void __on_update(struct cgroup *cgrp)
{
long *ptr;

Some files were not shown because too many files have changed in this diff Show More