linux-stable/lib
wuqiang.matt d67f39d2b8 lib: objpool: fix head overrun on RK3588 SBC
objpool overrun stress with test_objpool on OrangePi5+ SBC triggered the
following kernel warnings:

    WARNING: CPU: 6 PID: 3115 at lib/objpool.c:168 objpool_push+0xc0/0x100

This message is from objpool.c:168:

    WARN_ON_ONCE(tail - head > pool->nr_objs);

The overrun test case is to validate the case that pre-allocated objects
are insufficient: 8 objects are pre-allocated for each node and consumer
thread per node tries to grab 16 objects in a row. The testing system is
OrangePI 5+, with RK3588, a big.LITTLE SOC with 4x A76 and 4x A55. When
disabling either all 4 big or 4 little cores, the overrun tests run well,
and once with big and little cores mixed together, the overrun test would
always cause an overrun loop. It's likely the memory timing differences
of big and little cores cause this trouble. Here are the debugging data
of objpool_try_get_slot after try_cmpxchg_release:

    objpool_pop: cpu: 4/0 0:0 head: 278/279 tail:278 last:276/278

The local copies of 'head' and 'last' were 278 and 276, and reloading of
'slot->head' and 'slot->last' got 279 and 278. After try_cmpxchg_release
'slot->head' became 'head + 1', which is correct. But what's wrong here
is the stale value of 'last', and that stale value of 'last' finally led
the overrun of 'head'.

Memory updating of 'last' and 'head' are performed in push() and pop()
independently, which could be the culprit leading this out of order
visibility of 'last' and 'head'. So for objpool_try_get_slot(), it's
not enough only checking the condition of 'head != slot', the implicit
condition 'last - head <= nr_objs' must also be explicitly asserted to
guarantee 'last' is always behind 'head' before the object retrieving.

This patch will check and try reloading of 'head' and 'last' to ensure
'last' is behind 'head' at the time of object retrieving. Performance
testings show the average impact is about 0.1% for X86_64 and 1.12% for
ARM64. Here are the results:

    OS: Debian 10 X86_64, Linux 6.6rc
    HW: XEON 8336C x 2, 64 cores/128 threads, DDR4 3200MT/s
                      1T         2T         4T         8T        16T
    native:     49543304   99277826  199017659  399070324  795185848
    objpool:    29909085   59865637  119692073  239750369  478005250
    objpool+:   29879313   59230743  119609856  239067773  478509029
                     32T        48T        64T        96T       128T
    native:   1596927073 2390099988 2929397330 3183875848 3257546602
    objpool:   957553042 1435814086 1680872925 2043126796 2165424198
    objpool+:  956476281 1434491297 1666055740 2041556569 2157415622

    OS: Debian 11 AARCH64, Linux 6.6rc
    HW: Kunpeng-920 96 cores/2 sockets/4 NUMA nodes, DDR4 2933 MT/s
                      1T         2T         4T         8T        16T
    native:     30890508   60399915  123111980  242257008  494002946
    objpool:    14742531   28883047   57739948  115886644  232455421
    objpool+:   14107220   29032998   57286084  113730493  232232850
                     24T        32T        48T        64T        96T
    native:    746406039 1000174750 1493236240 1998318364 2942911180
    objpool:   349164852  467284332  702296756  934459713 1387898285
    objpool+:  348388180  462750976  696606096  927865887 1368402195

Link: https://lore.kernel.org/all/20231114115148.298821-1-wuqiang.matt@bytedance.com/

Fixes: b4edb8d2d4 ("lib: objpool added: ring-array based lockless MPMC")
Signed-off-by: wuqiang.matt <wuqiang.matt@bytedance.com>
Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
2023-12-01 14:53:55 +09:00
..
842
crypto crypto: lib/mpi - avoid null pointer deref in mpi_cmp_ui() 2023-08-11 19:19:52 +08:00
dim linux/dim: Do nothing if no time delta between samples 2023-05-09 11:06:45 +02:00
fonts powerpc: Use shared font data 2023-10-01 23:09:02 +11:00
kunit kunit: test: Fix the possible memory leak in executor_test 2023-09-28 08:51:07 -06:00
livepatch
lz4 lib: make LZ4_decompress_safe_forceExtDict() static 2022-07-17 17:31:39 -07:00
lzo lib/lzo/lzo1x_compress.c: replace ternary operator with min() and min_t() 2022-07-29 18:12:34 -07:00
math lib/math/int_log: Replace LGPL-2.1-or-later boilerplate with SPDX identifier 2023-07-09 22:47:50 +01:00
pldmfw lib: remove MODULE_LICENSE in non-modules 2023-04-13 13:13:53 -07:00
raid6 lib/raid6: Drop IA64 support 2023-09-11 08:13:18 +00:00
reed_solomon treewide: use get_random_u32_below() instead of deprecated function 2022-11-18 02:15:15 +01:00
test_fortify
vdso vdso: Improve cmd_vdso_check to check all dynamic relocations 2023-03-21 21:15:34 +01:00
xz arch: Remove Itanium (IA-64) architecture 2023-09-11 08:13:17 +00:00
zlib_deflate lib/zlib: DFLTCC deflate does not write all available bits for Z_NO_FLUSH 2023-02-27 17:00:14 -08:00
zlib_dfltcc lib/zlib: remove redundation assignement of avail_in dfltcc_gdht() 2023-02-02 22:50:10 -08:00
zlib_inflate lib/zlib: Split deflate and inflate states for DFLTCC 2023-02-02 22:50:09 -08:00
zstd zstd: Fix array-index-out-of-bounds UBSAN warning 2023-11-14 17:12:52 -08:00
.gitignore
Kconfig cxl for v6.7 2023-11-04 16:20:36 -10:00
Kconfig.debug Probes updates for v6.7: 2023-11-01 16:15:42 -10:00
Kconfig.kasan kasan: treat meminstrinsic as builtins in uninstrumented files 2023-03-02 21:54:22 -08:00
Kconfig.kcsan Kernel concurrency sanitizer (KCSAN) updates for v6.3 2023-02-25 13:02:20 -08:00
Kconfig.kfence
Kconfig.kgdb parisc: Convert PDC console to an early console 2022-10-11 12:01:24 +02:00
Kconfig.kmsan kmsan: make sure PREEMPT_RT is off 2022-11-08 15:57:24 -08:00
Kconfig.ubsan ubsan: Clarify Kconfig text for CONFIG_UBSAN_TRAP 2023-07-17 16:05:19 -07:00
Makefile cxl for v6.7 2023-11-04 16:20:36 -10:00
argv_split.c argv_split: fix kernel-doc warnings 2023-09-19 13:21:33 -07:00
ashldi3.c
ashrdi3.c
asn1_decoder.c
asn1_encoder.c
assoc_array.c
atomic64.c
atomic64_test.c
audit.c
base64.c lib/base64: RFC4648-compliant base64 encoding 2022-08-02 17:14:47 -06:00
bcd.c
bch.c lib/bch.c: use bitrev instead of internal logic 2023-08-18 10:18:58 -07:00
bitfield_kunit.c
bitmap-str.c lib/bitmap: split-out string-related operations to a separate files 2023-10-14 20:25:22 -07:00
bitmap.c bitmap: move bitmap_*_region() functions to bitmap.h 2023-10-16 16:14:45 -07:00
bitrev.c
bootconfig-data.S
bootconfig.c
bsearch.c
btree.c btree: remove MODULE_LICENSE in non-modules 2023-04-13 13:13:54 -07:00
bucket_locks.c
bug.c cpuidle: lib/bug: Disable rcu_is_watching() during WARN/BUG 2023-01-31 15:01:45 +01:00
build_OID_registry
buildid.c ELF: fix all "Elf" typos 2023-04-08 13:45:37 -07:00
bust_spinlocks.c kernel/panic: Drop unblank_screen call 2022-09-01 16:55:35 +02:00
check_signature.c
checksum.c
checksum_kunit.c kunit: Fix checksum tests on big endian CPUs 2023-08-25 10:14:34 +01:00
closure.c closures: Fix race in closure_sync() 2023-10-30 21:48:22 -04:00
clz_ctz.c lib/clz_ctz.c: Fix __clzdi2() and __ctzdi2() for 32-bit kernels 2023-08-25 13:22:10 -07:00
clz_tab.c
cmdline.c lib/cmdline: avoid page fault in next_arg 2022-09-11 21:55:06 -07:00
cmdline_kunit.c treewide: use get_random_{u8,u16}() when possible, part 1 2022-10-11 17:42:58 -06:00
cmpdi2.c
compat_audit.c
cpu_rmap.c lib: cpu_rmap: Fix potential use-after-free in irq_cpu_rmap_release() 2023-06-07 21:25:00 -07:00
cpumask.c bitmap patches for v6.7 2023-11-03 07:08:36 -10:00
cpumask_kunit.c cpumask: re-introduce constant-sized cpumask optimizations 2023-03-05 14:30:34 -08:00
crc-ccitt.c
crc-itu-t.c
crc-t10dif.c
crc4.c
crc7.c
crc8.c
crc16.c
crc32.c
crc32defs.h
crc32test.c
crc64-rocksoft.c
crc64.c
ctype.c
debug_info.c
debug_locks.c
debugobjects.c treewide: mark stuff as __ro_after_init 2023-10-18 14:43:23 -07:00
dec_and_lock.c perf: Fix perf_event_pmu_context serialization 2023-01-31 20:37:18 +01:00
decompress.c
decompress_bunzip2.c
decompress_inflate.c decompressor: provide missing prototypes 2023-06-09 17:44:17 -07:00
decompress_unlz4.c
decompress_unlzma.c
decompress_unlzo.c
decompress_unxz.c arch: Remove Itanium (IA-64) architecture 2023-09-11 08:13:17 +00:00
decompress_unzstd.c decompressor: provide missing prototypes 2023-06-09 17:44:17 -07:00
devmem_is_allowed.c lib: devmem_is_allowed: include linux/io.h 2023-06-09 17:44:15 -07:00
devres.c devres: show which resource was invalid in __devm_ioremap_resource() 2023-06-23 17:04:04 -07:00
dhry.h lib: add Dhrystone benchmark test 2023-02-02 22:50:01 -08:00
dhry_1.c lib: dhry: fix sleeping allocations inside non-preemptable section 2023-07-08 09:29:32 -07:00
dhry_2.c lib: add Dhrystone benchmark test 2023-02-02 22:50:01 -08:00
dhry_run.c lib: dhry: fix unstable smp_processor_id(_) usage 2023-03-23 17:18:35 -07:00
digsig.c
dump_stack.c
dynamic_debug.c dyndbg: add source filename to prefix 2023-08-04 15:28:41 +02:00
dynamic_queue_limits.c
earlycpio.c lib: move from strlcpy with unused retval to strscpy 2022-09-11 21:55:10 -07:00
errname.c parisc: Drop the HP-UX ENOSYM and EREMOTERELEASE error codes 2023-11-25 09:43:18 +01:00
error-inject.c lib: error-inject: remove error checking for debugfs_create_dir() 2023-08-18 10:18:55 -07:00
errseq.c
extable.c
fault-inject-usercopy.c
fault-inject.c fault-inject: allow configuration via configfs 2023-04-13 07:38:54 -06:00
fdt.c
fdt_addresses.c
fdt_empty_tree.c
fdt_ro.c
fdt_rw.c
fdt_strerror.c
fdt_sw.c
fdt_wip.c
find_bit.c cpumask: introduce for_each_cpu_or 2023-03-19 10:02:04 -07:00
find_bit_benchmark.c treewide: use get_random_u32_below() instead of deprecated function 2022-11-18 02:15:15 +01:00
flex_proportions.c flex_proportions: Disable preemption entering the write section. 2022-09-19 14:35:08 +02:00
fortify_kunit.c Revert "fortify: Allow KUnit test to build without FORTIFY" 2023-07-03 12:50:26 -07:00
fw_table.c lib/fw_table: Remove acpi_parse_entries_array() export 2023-11-02 15:17:21 -07:00
gen_crc32table.c
gen_crc64table.c
genalloc.c Devicetree include cleanups for v6.6: 2023-08-30 17:04:28 -07:00
generic-radix-tree.c lib/generic-radix-tree.c: Add peek_prev() 2023-10-19 14:47:33 -04:00
glob.c
globtest.c
group_cpus.c lib/group_cpus: Export group_cpus_evenly() 2023-04-21 03:02:31 -04:00
hashtable_test.c lib/hashtable_test.c: add test for the hashtable structure 2023-02-08 14:28:17 -07:00
hexdump.c
hweight.c
idr.c idr: fix param name in idr_alloc_cyclic() doc 2023-09-05 19:01:38 -04:00
inflate.c
interval_tree.c interval-tree: Add a utility to iterate over spans in an interval tree 2022-11-29 16:34:15 -04:00
interval_tree_test.c
iomap.c kmsan: add iomap support 2022-10-03 14:03:21 -07:00
iomap_copy.c
iommu-helper.c
iov_iter.c iov_iter: fix copy_page_to_iter_nofault() 2023-11-18 16:42:07 +01:00
irq_poll.c
irq_regs.c
is_signed_type_kunit.c lib: assume char is unsigned 2022-11-19 00:56:15 +01:00
is_single_threaded.c
kasprintf.c
kfifo.c
klist.c
kobject.c kobject: Remove redundant checks for whether ktype is NULL 2023-08-19 19:37:53 +02:00
kobject_uevent.c kobject: Replace strlcpy with strscpy 2023-09-22 09:50:56 -07:00
kstrtox.c kstrtox: consistently use _tolower() 2023-08-21 13:46:25 -07:00
kstrtox.h
kunit_iov_iter.c iov_iter: Kunit tests for page extraction 2023-09-09 15:11:49 -07:00
libcrc32c.c libcrc32c: remove crc32c_impl 2023-04-17 18:01:23 +02:00
linear_ranges.c
list-test.c list: test: Test the klist structure 2023-03-31 09:21:35 -06:00
list_debug.c list: Introduce CONFIG_LIST_HARDENED 2023-08-15 14:57:25 -07:00
list_sort.c
llist.c llist: add llist_del_first_this() 2023-10-16 12:44:06 -04:00
locking-selftest-hardirq.h
locking-selftest-mutex.h
locking-selftest-rlock-hardirq.h
locking-selftest-rlock-softirq.h
locking-selftest-rlock.h
locking-selftest-rsem.h
locking-selftest-rtmutex.h
locking-selftest-softirq.h
locking-selftest-spin-hardirq.h
locking-selftest-spin-softirq.h
locking-selftest-spin.h
locking-selftest-wlock-hardirq.h
locking-selftest-wlock-softirq.h
locking-selftest-wlock.h
locking-selftest-wsem.h
locking-selftest.c lockdep/selftests: Use SBRM APIs for wait context tests 2023-07-26 12:29:13 +02:00
lockref.c lockref: stop doing cpu_relax in the cmpxchg loop 2023-01-13 14:35:38 -06:00
logic_iomem.c
logic_pio.c minmax: add in_range() macro 2023-08-24 16:20:18 -07:00
lru_cache.c lru_cache: remove unused lc_private, lc_set, lc_index_of 2022-11-22 19:38:39 -07:00
lshrdi3.c
lwq.c lib: add light-weight queuing mechanism. 2023-10-16 12:44:06 -04:00
maple_tree.c maple_tree: add GFP_KERNEL to allocations in mas_expected_entries() 2023-10-18 12:12:41 -07:00
memcat_p.c
memcpy_kunit.c kunit: memcpy: Mark tests as slow using test attributes 2023-07-26 13:29:28 -06:00
memory-notifier-error-inject.c
memregion.c
memweight.c
muldi3.c
net_utils.c mac_pton: Clean up the header inclusions 2023-06-06 13:18:32 +02:00
netdev-notifier-error-inject.c
nlattr.c netlink: add variable-length / auto integers 2023-10-20 11:43:35 +01:00
nmi_backtrace.c nmi_backtrace: allow excluding an arbitrary CPU 2023-08-18 10:19:00 -07:00
notifier-error-inject.c lib: remove error checking for debugfs_create_dir() 2023-08-18 10:18:55 -07:00
notifier-error-inject.h
objagg.c
objpool.c lib: objpool: fix head overrun on RK3588 SBC 2023-12-01 14:53:55 +09:00
of-reconfig-notifier-error-inject.c
oid_registry.c lib/oid_registry.c: remove redundant assignment to variable num 2022-11-18 13:55:06 -08:00
once.c once: rename _SLOW to _SLEEPABLE 2022-10-03 17:34:32 -07:00
overflow_kunit.c overflow: Add struct_size_t() helper 2023-05-26 13:52:19 -07:00
packing.c lib: packing: remove MODULE_LICENSE in non-modules 2023-03-09 23:08:04 -08:00
parman.c
parser.c lib: parser: update documentation for match_NUMBER functions 2023-03-02 21:54:22 -08:00
pci_iomap.c
percpu-refcount.c percpu-refcount: Use call_rcu_hurry() for atomic switch 2022-11-30 13:16:40 -08:00
percpu_counter.c percpu_counter: extend _limited_add() to negative amounts 2023-10-18 14:34:14 -07:00
percpu_test.c
plist.c
pm-notifier-error-inject.c
polynomial.c
radix-tree.c radix tree: remove unused variable 2023-08-21 13:07:22 -07:00
radix-tree.h radix-tree: move declarations to header 2023-06-12 11:31:50 -07:00
random32.c treewide: use get_random_bytes() when possible 2022-10-11 17:42:58 -06:00
ratelimit.c ratelimit: Fix data-races in ___ratelimit(). 2022-08-24 13:46:57 +01:00
rbtree.c lib/rbtree: use '+' instead of '|' for setting color. 2023-04-18 16:39:33 -07:00
rbtree_test.c
rcuref.c locking/atomics: Use atomic_try_cmpxchg_release() to micro-optimize rcuref_put_slowpath() 2023-10-10 10:14:27 +02:00
ref_tracker.c lib/ref_tracker: remove warnings in case of allocation failure 2023-06-05 15:28:42 -07:00
refcount.c
rhashtable.c rhashtable: Allow rhashtable to be used from irq-safe contexts 2022-12-09 10:42:56 +00:00
sbitmap.c sbitmap: fix batching wakeup 2023-07-21 11:40:20 -06:00
scatterlist.c scatterlist: add missing function params to kernel-doc 2023-09-19 13:21:33 -07:00
seq_buf.c seq_buf: Export seq_buf_puts() 2023-11-02 00:19:44 -04:00
sg_pool.c lib/sg_pool: change module_init(sg_pool_init) to subsys_initcall 2022-09-23 16:46:19 +02:00
sg_split.c
siphash.c
siphash_kunit.c siphash: Convert selftest to KUnit 2022-11-01 10:04:52 -07:00
slub_kunit.c linux-kselftest-kunit-next-6.2-rc1 2022-12-12 16:42:57 -08:00
smp_processor_id.c lib/smp_processor_id: fix imbalanced instrumentation_end() call 2022-07-17 17:31:41 -07:00
sort.c
stackdepot.c lib/stackdepot: kmsan: mark API outputs as initialized 2023-03-28 16:20:13 -07:00
stackinit_kunit.c kernel/range: Uplevel the cxl subsystem's range_contains() helper 2023-02-10 17:32:37 -08:00
stmp_device.c
strcat_kunit.c string: Add Kunit tests for strcat() family 2023-05-16 14:08:02 -07:00
string.c string: use __builtin_memcpy() in strlcpy/strlcat 2023-06-01 11:24:50 -07:00
string_helpers.c lib/string_helpers: string_get_size() now returns characters wrote 2023-09-11 23:59:47 -04:00
strncpy_from_user.c
strnlen_user.c
strscpy_kunit.c fortify: Short-circuit known-safe calls to strscpy() 2022-11-01 10:04:52 -07:00
syscall.c
test-kstrtox.c
test-string_helpers.c lib/test-string_helpers: replace UNESCAPE_ANY by UNESCAPE_ALL_MASK 2023-04-08 13:45:39 -07:00
test_bitmap.c bitmap: add test for bitmap_*_region() functions 2023-10-14 20:25:22 -07:00
test_bitops.c
test_bits.c
test_blackhole_dev.c
test_bpf.c bpf/tests: add tests for cpuv4 instructions 2023-09-15 17:16:57 -07:00
test_debug_virtual.c
test_dynamic_debug.c dyndbg: test DECLARE_DYNDBG_CLASSMAP, sysfs nodes 2022-09-07 17:04:49 +02:00
test_firmware.c test_firmware: return ENOMEM instead of ENOSPC on failed memory allocation 2023-06-15 13:42:18 +02:00
test_fprobe.c fprobe: Pass return address to the handlers 2023-06-06 21:39:55 +09:00
test_fpu.c
test_free_pages.c lib/test_free_pages.c: pass a pointer to virt_to_page() 2022-07-17 17:14:36 -07:00
test_hash.c
test_hexdump.c treewide: use get_random_u32_inclusive() when possible 2022-11-18 02:18:02 +01:00
test_hmm.c lib: replace kmap() with kmap_local_page() 2023-08-18 10:18:50 -07:00
test_hmm_uapi.h hmm-tests: add test for migrate_device_range() 2022-10-12 18:51:50 -07:00
test_ida.c
test_kmod.c test_kmod: stop kernel-doc warnings 2023-01-25 14:07:21 -08:00
test_kprobes.c test_kprobes: Add recursed kprobe test case 2023-02-21 08:52:42 +09:00
test_linear_ranges.c lib/test_linear_ranges: Use LINEAR_RANGE() 2022-11-16 13:32:32 +00:00
test_list_sort.c treewide: use get_random_u32_below() instead of deprecated function 2022-11-18 02:15:15 +01:00
test_lockup.c
test_maple_tree.c maple_tree: add GFP_KERNEL to allocations in mas_expected_entries() 2023-10-18 12:12:41 -07:00
test_memcat_p.c
test_meminit.c lib/test_meminit: allocate pages up to order MAX_ORDER 2023-08-18 10:12:32 -07:00
test_min_heap.c treewide: use get_random_u32() when possible 2022-10-11 17:42:58 -06:00
test_module.c
test_objagg.c treewide: use get_random_bytes() when possible 2022-10-11 17:42:58 -06:00
test_objpool.c lib: test_objpool: make global variables static 2023-11-10 19:59:04 +09:00
test_parman.c
test_printf.c lib/vsprintf: declare no_hash_pointers in sprintf.h 2023-08-21 13:46:24 -07:00
test_ref_tracker.c lib/ref_tracker: improve printing stats 2023-06-05 15:28:42 -07:00
test_rhashtable.c Networking changes for 6.2. 2022-12-13 15:47:48 -08:00
test_scanf.c lib: test_scanf: Add explicit type cast to result initialization in test_number_prefix() 2023-08-16 11:47:29 +02:00
test_sort.c
test_static_key_base.c
test_static_keys.c
test_string.c
test_sysctl.c test_sysclt: Test for registering a mount point 2023-06-18 02:32:54 -07:00
test_ubsan.c
test_user_copy.c
test_uuid.c
test_vmalloc.c lib/test_vmalloc.c: avoid garbage in page array 2023-06-12 11:31:51 -07:00
test_xarray.c
textsearch.c
timerqueue.c
trace_readwrite.c asm-generic/io: Add _RET_IP_ to MMIO trace for more accurate debug info 2022-11-21 22:02:10 +01:00
ts_bm.c lib/ts_bm: add helper to reduce indentation and improve readability 2023-07-27 13:45:51 +02:00
ts_fsm.c
ts_kmp.c
ubsan.c ubsan: add prototypes for internal functions 2023-05-30 16:42:01 -07:00
ubsan.h ubsan: add prototypes for internal functions 2023-05-30 16:42:01 -07:00
ucmpdi2.c
ucs2_string.c lib/ucs2_string: Add UCS-2 strscpy function 2023-09-13 10:18:42 -07:00
usercopy.c uaccess: Add speculation barrier to copy_from_user() 2023-02-21 14:45:22 -08:00
uuid.c treewide: use get_random_bytes() when possible 2022-10-11 17:42:58 -06:00
vsprintf.c vsprintf: uninline simple_strntoull(), reorder arguments 2023-11-01 15:55:39 +01:00
win_minmax.c lib/win_minmax: use /* notation for regular comments 2023-01-11 16:14:21 -08:00
xarray.c xarray: Document necessary flag in alloc functions 2023-09-05 19:01:38 -04:00
xxhash.c