linux-stable

mirror of https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git synced 2024-09-29 05:44:11 +00:00

No description

Find a file

Mingzhe Zou 12f4971589 bcache: fixup multi-threaded bch_sectors_dirty_init() wake-up race commit `2faac25d79` upstream. We get a kernel crash about "unable to handle kernel paging request": ```dmesg [368033.032005] BUG: unable to handle kernel paging request at ffffffffad9ae4b5 [368033.032007] PGD fc3a0d067 P4D fc3a0d067 PUD fc3a0e063 PMD 8000000fc38000e1 [368033.032012] Oops: 0003 [#1] SMP PTI [368033.032015] CPU: 23 PID: 55090 Comm: bch_dirtcnt[0] Kdump: loaded Tainted: G OE --------- - - 4.18.0-147.5.1.es8_24.x86_64 #1 [368033.032017] Hardware name: Tsinghua Tongfang THTF Chaoqiang Server/072T6D, BIOS 2.4.3 01/17/2017 [368033.032027] RIP: 0010:native_queued_spin_lock_slowpath+0x183/0x1d0 [368033.032029] Code: 8b 02 48 85 c0 74 f6 48 89 c1 eb d0 c1 e9 12 83 e0 03 83 e9 01 48 c1 e0 05 48 63 c9 48 05 c0 3d 02 00 48 03 04 cd 60 68 93 ad <48> 89 10 8b 42 08 85 c0 75 09 f3 90 8b 42 08 85 c0 74 f7 48 8b 02 [368033.032031] RSP: 0018:ffffbb48852abe00 EFLAGS: 00010082 [368033.032032] RAX: ffffffffad9ae4b5 RBX: 0000000000000246 RCX: 0000000000003bf3 [368033.032033] RDX: ffff97b0ff8e3dc0 RSI: 0000000000600000 RDI: ffffbb4884743c68 [368033.032034] RBP: 0000000000000001 R08: 0000000000000000 R09: 000007ffffffffff [368033.032035] R10: ffffbb486bb01000 R11: 0000000000000001 R12: ffffffffc068da70 [368033.032036] R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000000 [368033.032038] FS: 0000000000000000(0000) GS:ffff97b0ff8c0000(0000) knlGS:0000000000000000 [368033.032039] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [368033.032040] CR2: ffffffffad9ae4b5 CR3: 0000000fc3a0a002 CR4: 00000000003626e0 [368033.032042] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [368033.032043] bcache: bch_cached_dev_attach() Caching rbd479 as bcache462 on set 8cff3c36-4a76-4242-afaa-7630206bc70b [368033.032045] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [368033.032046] Call Trace: [368033.032054] _raw_spin_lock_irqsave+0x32/0x40 [368033.032061] __wake_up_common_lock+0x63/0xc0 [368033.032073] ? bch_ptr_invalid+0x10/0x10 [bcache] [368033.033502] bch_dirty_init_thread+0x14c/0x160 [bcache] [368033.033511] ? read_dirty_submit+0x60/0x60 [bcache] [368033.033516] kthread+0x112/0x130 [368033.033520] ? kthread_flush_work_fn+0x10/0x10 [368033.034505] ret_from_fork+0x35/0x40 ``` The crash occurred when call wake_up(&state->wait), and then we want to look at the value in the state. However, bch_sectors_dirty_init() is not found in the stack of any task. Since state is allocated on the stack, we guess that bch_sectors_dirty_init() has exited, causing bch_dirty_init_thread() to be unable to handle kernel paging request. In order to verify this idea, we added some printing information during wake_up(&state->wait). We find that "wake up" is printed twice, however we only expect the last thread to wake up once. ```dmesg [ 994.641004] alcache: bch_dirty_init_thread() wake up [ 994.641018] alcache: bch_dirty_init_thread() wake up [ 994.641523] alcache: bch_sectors_dirty_init() init exit ``` There is a race. If bch_sectors_dirty_init() exits after the first wake up, the second wake up will trigger this bug("unable to handle kernel paging request"). Proceed as follows: bch_sectors_dirty_init kthread_run ==============> bch_dirty_init_thread(bch_dirtcnt[0]) ... ... atomic_inc(&state.started) ... ... ... atomic_read(&state.enough) ... ... atomic_set(&state->enough, 1) kthread_run ======================================================> bch_dirty_init_thread(bch_dirtcnt[1]) ... atomic_dec_and_test(&state->started) ... atomic_inc(&state.started) ... ... ... wake_up(&state->wait) ... atomic_read(&state.enough) atomic_dec_and_test(&state->started) ... ... wait_event(state.wait, atomic_read(&state.started) == 0) ... return ... wake_up(&state->wait) We believe it is very common to wake up twice if there is no dirty, but crash is an extremely low probability event. It's hard for us to reproduce this issue. We attached and detached continuously for a week, with a total of more than one million attaches and only one crash. Putting atomic_inc(&state.started) before kthread_run() can avoid waking up twice. Fixes: `b144e45fc5` ("bcache: make bch_sectors_dirty_init() to be multithreaded") Signed-off-by: Mingzhe Zou <mingzhe.zou@easystack.cn> Cc: <stable@vger.kernel.org> Signed-off-by: Coly Li <colyli@suse.de> Link: https://lore.kernel.org/r/20231120052503.6122-8-colyli@suse.de Signed-off-by: Jens Axboe <axboe@kernel.dk> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>		2023-12-03 07:32:11 +01:00
arch	arm64: dts: imx8mn-var-som: add 20ms delay to ethernet regulator enable	2023-12-03 07:32:11 +01:00
block	blk-core: use pr_warn_ratelimited() in bio_check_ro()	2023-11-20 11:52:17 +01:00
certs	certs: Fix build error when PKCS#11 URI contains semicolon	2023-02-09 11:28:11 +01:00
crypto	crypto: pcrypt - Fix hungtask for PADATA_RESET	2023-11-28 17:06:58 +00:00
Documentation	smp,csd: Throw an error if a CSD lock is stuck for too long	2023-11-28 17:06:55 +00:00
drivers	bcache: fixup multi-threaded bch_sectors_dirty_init() wake-up race	2023-12-03 07:32:11 +01:00
fs	NFSD: Fix checksum mismatches in the duplicate reply cache	2023-12-03 07:32:10 +01:00
include	HID: fix HID device resource race between HID core and debugging support	2023-12-03 07:32:07 +01:00
init	proc: sysctl: prevent aliased sysctls from getting passed to init	2023-11-28 17:07:08 +00:00
io_uring	io_uring/net: ensure socket is marked connected on connect retry	2023-11-20 11:52:18 +01:00
ipc
kernel	lockdep: Fix block chain corruption	2023-12-03 07:32:09 +01:00
lib	lib/generic-radix-tree.c: Don't overflow in peek()	2023-11-28 17:06:54 +00:00
LICENSES
mm	mm,kfence: decouple kfence from page granularity mapping judgement	2023-12-03 07:32:08 +01:00
net	net/smc: avoid data corruption caused by decline	2023-12-03 07:32:08 +01:00
rust	rust: allocator: Prevent mis-aligned allocation	2023-08-11 12:08:18 +02:00
samples	fprobe: Pass entry_data to handlers	2023-10-25 12:03:12 +02:00
scripts	randstruct: Fix gcc-plugin performance mode to stay in group	2023-11-28 17:07:07 +00:00
security	ima: detect changes to the backing overlay file	2023-11-28 17:07:12 +00:00
sound	ALSA: hda/realtek: Add quirks for HP Laptops	2023-11-28 17:07:16 +00:00
tools	cxl/region: Fix x1 root-decoder granularity calculations	2023-11-28 17:07:18 +00:00
usr
virt	kvm/vfio: ensure kvg instance stays around in kvm_vfio_group_add()	2023-09-13 09:42:46 +02:00
.clang-format
.cocciconfig
.get_maintainer.ignore
.gitattributes
.gitignore
.mailmap
.rustfmt.toml
COPYING
CREDITS
Kbuild
Kconfig
MAINTAINERS	devlink: move code to a dedicated directory	2023-08-30 16:11:00 +02:00
Makefile	Linux 6.1.64	2023-11-28 17:07:23 +00:00
README

README

Linux kernel
============

There are several guides for kernel developers and users. These guides can
be rendered in a number of formats, like HTML and PDF. Please read
Documentation/admin-guide/README.rst first.

In order to build the documentation, use ``make htmldocs`` or
``make pdfdocs``.  The formatted documentation can also be read online at:

    https://www.kernel.org/doc/html/latest/

There are various text files in the Documentation/ subdirectory,
several of them using the Restructured Text markup notation.

Please read the Documentation/process/changes.rst file, as it contains the
requirements for building and running the kernel, and information about
the problems which may result by upgrading your kernel.