mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
synced 2024-10-30 08:02:30 +00:00
rcu: Fix rcu_barrier_callback() race condition
The rcu_barrier_callback() function does an atomic_dec_and_test(), and if it is the last CPU to check in, does the required wakeup. Either way, it does an event trace. Unfortunately, this is susceptible to the following sequence of events: o CPU 0 invokes rcu_barrier_callback(), but atomic_dec_and_test() says that it is not last. But at this point, CPU 0 is delayed, perhaps due to an NMI, SMI, or vCPU preemption. o CPU 1 invokes rcu_barrier_callback(), and atomic_dec_and_test() says that it is last. So CPU 1 traces completion and does the needed wakeup. o The awakened rcu_barrier() function does cleanup and releases rcu_state.barrier_mutex. o Another CPU now acquires rcu_state.barrier_mutex and starts another round of rcu_barrier() processing, including updating rcu_state.barrier_sequence. o CPU 0 gets its act back together and does its tracing. Except that rcu_state.barrier_sequence has already been updated, so its tracing is incorrect and probably quite confusing. (Wait! Why did this CPU check in twice for one rcu_barrier() invocation???) This commit therefore causes rcu_barrier_callback() to take a snapshot of the value of rcu_state.barrier_sequence before invoking atomic_dec_and_test(), thus guaranteeing that the event-trace output is sensible, even if the timing of the event-trace output might still be confusing. (Wait! Why did the old rcu_barrier() complete before all of its CPUs checked in???) But being that this is RCU, only so much confusion can reasonably be eliminated. This data race was reported by KCSAN. Not appropriate for backporting due to failure being unlikely and due to the mild consequences of the failure, namely a confusing event trace. Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit is contained in:
parent
59881bcd85
commit
aa24f93753
1 changed files with 10 additions and 3 deletions
|
@ -3077,15 +3077,22 @@ static void rcu_barrier_trace(const char *s, int cpu, unsigned long done)
|
|||
/*
|
||||
* RCU callback function for rcu_barrier(). If we are last, wake
|
||||
* up the task executing rcu_barrier().
|
||||
*
|
||||
* Note that the value of rcu_state.barrier_sequence must be captured
|
||||
* before the atomic_dec_and_test(). Otherwise, if this CPU is not last,
|
||||
* other CPUs might count the value down to zero before this CPU gets
|
||||
* around to invoking rcu_barrier_trace(), which might result in bogus
|
||||
* data from the next instance of rcu_barrier().
|
||||
*/
|
||||
static void rcu_barrier_callback(struct rcu_head *rhp)
|
||||
{
|
||||
unsigned long __maybe_unused s = rcu_state.barrier_sequence;
|
||||
|
||||
if (atomic_dec_and_test(&rcu_state.barrier_cpu_count)) {
|
||||
rcu_barrier_trace(TPS("LastCB"), -1,
|
||||
rcu_state.barrier_sequence);
|
||||
rcu_barrier_trace(TPS("LastCB"), -1, s);
|
||||
complete(&rcu_state.barrier_completion);
|
||||
} else {
|
||||
rcu_barrier_trace(TPS("CB"), -1, rcu_state.barrier_sequence);
|
||||
rcu_barrier_trace(TPS("CB"), -1, s);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
Loading…
Reference in a new issue