mirrors
/
linux-stable
mirror of https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
1
0
Fork 0
Code Issues Releases Wiki Activity

RCU changes for this cycle were: 

- Bitmap support for "N" as alias for last bit
  - kvfree_rcu updates
  - mm_dump_obj() updates.  (One of these is to mm, but was suggested by Andrew Morton.)
  - RCU callback offloading update
  - Polling RCU grace-period interfaces
  - Realtime-related RCU updates
  - Tasks-RCU updates
  - Torture-test updates
  - Torture-test scripting updates
  - Miscellaneous fixes
 
 Signed-off-by: Ingo Molnar <mingo@kernel.org>
 -----BEGIN PGP SIGNATURE-----
 
 iQJFBAABCgAvFiEEBpT5eoXrXCwVQwEKEnMQ0APhK1gFAmCJCZERHG1pbmdvQGtl
 cm5lbC5vcmcACgkQEnMQ0APhK1hRjw/+Jkb9KvR9odPt/zqN/KPtIlburCUWgsFb
 2zAlWN4uMocPAiXT2Xq58/8gqMkpyn7ZVZtL1tD8fZSvlwEr0U8Z74+/NdoQvYE+
 kMXIYIuhIAGRyAupmzkriqN33iY+BSZPacX3u6ziPj57/0OZzbWVN/DAhbuvyLqG
 J/oL4PHCa7XAqXbf95rd5Zjs680QJ3CbTRh4nA8uHArzJmKZOaaHJ05Pxd1LpULe
 SJ+5p1GQnnwxd1HqmlHMDu/dW+2hE35BGykF8zi78je9OJXualDoM/6JpIYGhMNY
 5qlhU55QYP1jzjuNGVZZUS4L77eS2/W7SpPAaTmMEy/SsVB59G8Kf22oNDpVaEqQ
 m+2ErqwaHvlkMjqnsx+JQbsOP0yCi2NZBoEPFdfk1H23E2deVlSDbxPso4Zb1oUD
 E12769kN+SWDytuLSOAe1PY/KXqmNUKjPZl1GDCGXL7HlCnWyggUDschTsKJa19O
 XXl+yCTGMUH4XAPSqavAKQbBjurqpT6i4zfooSH4TBtOHm1ExgZOUS8gglZ1JuJd
 q+uJdZIgS8BcGkGw/k1bYDWY5TA4Rjv3sAOKQL1PgYBl1t/yLK441mE7LI9gWOwz
 Crz7vlSxD6Jc2cYQeUVW0KPGt5aVd63Gd9HjpXxGkqYQSDRqYMCebHEAGagz+jj7
 Nv/nOnf34Uc=
 =mpNt
 -----END PGP SIGNATURE-----

Merge tag 'core-rcu-2021-04-28' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

Pull RCU updates from Ingo Molnar:

 - Support for "N" as alias for last bit in bitmap parsing library (eg
   using syntax like "nohz_full=2-N")

 - kvfree_rcu updates

 - mm_dump_obj() updates. (One of these is to mm, but was suggested by
   Andrew Morton.)

 - RCU callback offloading update

 - Polling RCU grace-period interfaces

 - Realtime-related RCU updates

 - Tasks-RCU updates

 - Torture-test updates

 - Torture-test scripting updates

 - Miscellaneous fixes

* tag 'core-rcu-2021-04-28' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (77 commits)
  rcutorture: Test start_poll_synchronize_rcu() and poll_state_synchronize_rcu()
  rcu: Provide polling interfaces for Tiny RCU grace periods
  torture: Fix kvm.sh --datestamp regex check
  torture: Consolidate qemu-cmd duration editing into kvm-transform.sh
  torture: Print proper vmlinux path for kvm-again.sh runs
  torture: Make TORTURE_TRUST_MAKE available in kvm-again.sh environment
  torture: Make kvm-transform.sh update jitter commands
  torture: Add --duration argument to kvm-again.sh
  torture: Add kvm-again.sh to rerun a previous torture-test
  torture: Create a "batches" file for build reuse
  torture: De-capitalize TORTURE_SUITE
  torture: Make upper-case-only no-dot no-slash scenario names official
  torture: Rename SRCU-t and SRCU-u to avoid lowercase characters
  torture: Remove no-mpstat error message
  torture: Record kvm-test-1-run.sh and kvm-test-1-run-qemu.sh PIDs
  torture: Record jitter start/stop commands
  torture: Extract kvm-test-1-run-qemu.sh from kvm-test-1-run.sh
  torture: Record TORTURE_KCONFIG_GDB_ARG in qemu-cmd
  torture: Abstract jitter.sh start/stop into scripts
  rcu: Provide polling interfaces for Tree RCU grace periods
  ...
This commit is contained in:
Linus Torvalds 2021-04-28 12:00:13 -07:00
parent 68a32ba141 120b566d1d
commit 9a45da9270
54 changed files with 1327 additions and 445 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
Documentation/RCU/RTFP.txt
View File

@ -847,7 +847,7 @@ Symposium on Distributed Computing}
'It's entirely possible that the current user could be replaced
by RCU and/or seqlocks, and we could get rid of brlocks entirely.'
.
Steve Hemminger responds by replacing them with RCU.
Stephen Hemminger responds by replacing them with RCU.
}
}

7
Documentation/admin-guide/kernel-parameters.rst
View File

@ -68,6 +68,13 @@ For example one can add to the command line following parameter:
where the final item represents CPUs 100,101,125,126,150,151,...
The value "N" can be used to represent the numerically last CPU on the system,
i.e "foo_cpus=16-N" would be equivalent to "16-31" on a 32 core system.
Keep in mind that "N" is dynamic, so if system changes cause the bitmap width
to change, such as less cores in the CPU list, then N and any ranges using N
will also change. Use the same on a small 4 core system, and "16-N" becomes
"16-3" and now the same boot input will be flagged as invalid (start > end).
This document may not be entirely up to date and comprehensive. The command

16
Documentation/admin-guide/kernel-parameters.txt
View File

@ -4077,9 +4077,7 @@
see CONFIG_RAS_CEC help text.
rcu_nocbs= [KNL]
The argument is a cpu list, as described above,
except that the string "all" can be used to
specify every CPU on the system.
The argument is a cpu list, as described above.
In kernels built with CONFIG_RCU_NOCB_CPU=y, set
the specified list of CPUs to be no-callback CPUs.
@ -4268,6 +4266,18 @@
rcuscale.kfree_rcu_test= [KNL]
Set to measure performance of kfree_rcu() flooding.
rcuscale.kfree_rcu_test_double= [KNL]
Test the double-argument variant of kfree_rcu().
If this parameter has the same value as
rcuscale.kfree_rcu_test_single, both the single-
and double-argument variants are tested.
rcuscale.kfree_rcu_test_single= [KNL]
Test the single-argument variant of kfree_rcu().
If this parameter has the same value as
rcuscale.kfree_rcu_test_double, both the single-
and double-argument variants are tested.
rcuscale.kfree_nthreads= [KNL]
The number of threads running loops of kfree_rcu().

4
include/linux/mm.h
View File

@ -3180,7 +3180,11 @@ unsigned long wp_shared_mapping_range(struct address_space *mapping,
extern int sysctl_nr_trim_pages;
#ifdef CONFIG_PRINTK
void mem_dump_obj(void *object);
#else
static inline void mem_dump_obj(void *object) {}
#endif
#endif /* __KERNEL__ */
#endif /* _LINUX_MM_H */

7
include/linux/rcu_segcblist.h
View File

@ -109,7 +109,7 @@ struct rcu_cblist {
* | SEGCBLIST_KTHREAD_GP |
* | |
* | Kthreads handle callbacks holding nocb_lock, local rcu_core() stops |
* | handling callbacks. |
* | handling callbacks. Enable bypass queueing. |
* ----------------------------------------------------------------------------
*/
@ -125,7 +125,7 @@ struct rcu_cblist {
* | SEGCBLIST_KTHREAD_GP |
* | |
* | CB/GP kthreads handle callbacks holding nocb_lock, local rcu_core() |
* | ignores callbacks. |
* | ignores callbacks. Bypass enqueue is enabled. |
* ----------------------------------------------------------------------------
* |
* v
@ -134,7 +134,8 @@ struct rcu_cblist {
* | SEGCBLIST_KTHREAD_GP |
* | |
* | CB/GP kthreads and local rcu_core() handle callbacks concurrently |
* | holding nocb_lock. Wake up CB and GP kthreads if necessary. |
* | holding nocb_lock. Wake up CB and GP kthreads if necessary. Disable |
* | bypass enqueue. |
* ----------------------------------------------------------------------------
* |
* v

2
include/linux/rculist_nulls.h
View File

@ -161,7 +161,7 @@ static inline void hlist_nulls_add_fake(struct hlist_nulls_node *n)
*
* The barrier() is needed to make sure compiler doesn't cache first element [1],
* as this loop can be restarted [2]
* [1] Documentation/core-api/atomic_ops.rst around line 114
* [1] Documentation/memory-barriers.txt around line 1533
* [2] Documentation/RCU/rculist_nulls.rst around line 146
*/
#define hlist_nulls_for_each_entry_rcu(tpos, pos, head, member) \

2
include/linux/rcupdate.h
View File

@ -882,7 +882,7 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)
* The BUILD_BUG_ON check must not involve any function calls, hence the
* checks are done in macros here.
*/
#define kfree_rcu kvfree_rcu
#define kfree_rcu(ptr, rhf...) kvfree_rcu(ptr, ## rhf)
/**
* kvfree_rcu() - kvfree an object after a grace period.

7
include/linux/rcutiny.h
View File

@ -17,10 +17,9 @@
/* Never flag non-existent other CPUs! */
static inline bool rcu_eqs_special_set(int cpu) { return false; }
static inline unsigned long get_state_synchronize_rcu(void)
{
return 0;
}
unsigned long get_state_synchronize_rcu(void);
unsigned long start_poll_synchronize_rcu(void);
bool poll_state_synchronize_rcu(unsigned long oldstate);
static inline void cond_synchronize_rcu(unsigned long oldstate)
{

2
include/linux/rcutree.h
View File

@ -41,6 +41,8 @@ void rcu_momentary_dyntick_idle(void);
void kfree_rcu_scheduler_running(void);
bool rcu_gp_might_be_stalled(void);
unsigned long get_state_synchronize_rcu(void);
unsigned long start_poll_synchronize_rcu(void);
bool poll_state_synchronize_rcu(unsigned long oldstate);
void cond_synchronize_rcu(unsigned long oldstate);
void rcu_idle_enter(void);

2
include/linux/slab.h
View File

@ -186,8 +186,10 @@ void kfree(const void *);
void kfree_sensitive(const void *);
size_t __ksize(const void *);
size_t ksize(const void *);
#ifdef CONFIG_PRINTK
bool kmem_valid_obj(void *object);
void kmem_dump_obj(void *object);
#endif
#ifdef CONFIG_HAVE_HARDENED_USERCOPY_ALLOCATOR
void __check_heap_object(const void *ptr, unsigned long n, struct page *page,

2
include/linux/vmalloc.h
View File

@ -241,7 +241,7 @@ pcpu_free_vm_areas(struct vm_struct **vms, int nr_vms)
int register_vmap_purge_notifier(struct notifier_block *nb);
int unregister_vmap_purge_notifier(struct notifier_block *nb);
#ifdef CONFIG_MMU
#if defined(CONFIG_MMU) && defined(CONFIG_PRINTK)
bool vmalloc_dump_obj(void *object);
#else
static inline bool vmalloc_dump_obj(void *object) { return false; }

28
include/trace/events/rcu.h
View File

@ -432,6 +432,34 @@ TRACE_EVENT_RCU(rcu_fqs,
__entry->cpu, __entry->qsevent)
);
/*
* Tracepoint for RCU stall events. Takes a string identifying the RCU flavor
* and a string identifying which function detected the RCU stall as follows:
*
* "StallDetected": Scheduler-tick detects other CPU's stalls.
* "SelfDetected": Scheduler-tick detects a current CPU's stall.
* "ExpeditedStall": Expedited grace period detects stalls.
*/
TRACE_EVENT(rcu_stall_warning,
TP_PROTO(const char *rcuname, const char *msg),
TP_ARGS(rcuname, msg),
TP_STRUCT__entry(
__field(const char *, rcuname)
__field(const char *, msg)
),
TP_fast_assign(
__entry->rcuname = rcuname;
__entry->msg = msg;
),
TP_printk("%s %s",
__entry->rcuname, __entry->msg)
);
#endif /* #if defined(CONFIG_TREE_RCU) */
/*

3
kernel/rcu/rcu_segcblist.c
View File

@ -261,8 +261,7 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp)
}
/*
* Mark the specified rcu_segcblist structure as offloaded. This
* structure must be empty.
* Mark the specified rcu_segcblist structure as offloaded.
*/
void rcu_segcblist_offload(struct rcu_segcblist *rsclp, bool offload)
{

15
kernel/rcu/rcuscale.c
View File

@ -625,6 +625,8 @@ rcu_scale_shutdown(void *arg)
torture_param(int, kfree_nthreads, -1, "Number of threads running loops of kfree_rcu().");
torture_param(int, kfree_alloc_num, 8000, "Number of allocations and frees done in an iteration.");
torture_param(int, kfree_loops, 10, "Number of loops doing kfree_alloc_num allocations and frees.");
torture_param(bool, kfree_rcu_test_double, false, "Do we run a kfree_rcu() double-argument scale test?");
torture_param(bool, kfree_rcu_test_single, false, "Do we run a kfree_rcu() single-argument scale test?");
static struct task_struct **kfree_reader_tasks;
static int kfree_nrealthreads;
@ -644,10 +646,13 @@ kfree_scale_thread(void *arg)
struct kfree_obj *alloc_ptr;
u64 start_time, end_time;
long long mem_begin, mem_during = 0;
bool kfree_rcu_test_both;
DEFINE_TORTURE_RANDOM(tr);
VERBOSE_SCALEOUT_STRING("kfree_scale_thread task started");
set_cpus_allowed_ptr(current, cpumask_of(me % nr_cpu_ids));
set_user_nice(current, MAX_NICE);
kfree_rcu_test_both = (kfree_rcu_test_single == kfree_rcu_test_double);
start_time = ktime_get_mono_fast_ns();
@ -670,7 +675,15 @@ kfree_scale_thread(void *arg)
if (!alloc_ptr)
return -ENOMEM;
kfree_rcu(alloc_ptr, rh);
// By default kfree_rcu_test_single and kfree_rcu_test_double are
// initialized to false. If both have the same value (false or true)
// both are randomly tested, otherwise only the one with value true
// is tested.
if ((kfree_rcu_test_single && !kfree_rcu_test_double) ||
(kfree_rcu_test_both && torture_random(&tr) & 0x800))
kfree_rcu(alloc_ptr);
else
kfree_rcu(alloc_ptr, rh);
}
cond_resched();

93
kernel/rcu/rcutorture.c
View File

@ -245,11 +245,11 @@ static const char *rcu_torture_writer_state_getname(void)
return rcu_torture_writer_state_names[i];
}
#if defined(CONFIG_RCU_BOOST) && !defined(CONFIG_HOTPLUG_CPU)
#define rcu_can_boost() 1
#else /* #if defined(CONFIG_RCU_BOOST) && !defined(CONFIG_HOTPLUG_CPU) */
#define rcu_can_boost() 0
#endif /* #else #if defined(CONFIG_RCU_BOOST) && !defined(CONFIG_HOTPLUG_CPU) */
#if defined(CONFIG_RCU_BOOST) && defined(CONFIG_PREEMPT_RT)
# define rcu_can_boost() 1
#else
# define rcu_can_boost() 0
#endif
#ifdef CONFIG_RCU_TRACE
static u64 notrace rcu_trace_clock_local(void)
@ -494,6 +494,8 @@ static struct rcu_torture_ops rcu_ops = {
.sync = synchronize_rcu,
.exp_sync = synchronize_rcu_expedited,
.get_gp_state = get_state_synchronize_rcu,
.start_gp_poll = start_poll_synchronize_rcu,
.poll_gp_state = poll_state_synchronize_rcu,
.cond_sync = cond_synchronize_rcu,
.call = call_rcu,
.cb_barrier = rcu_barrier,
@ -923,9 +925,13 @@ static void rcu_torture_enable_rt_throttle(void)
static bool rcu_torture_boost_failed(unsigned long start, unsigned long end)
{
static int dbg_done;
if (end - start > test_boost_duration * HZ - HZ / 2) {
VERBOSE_TOROUT_STRING("rcu_torture_boost boosting failed");
n_rcu_torture_boost_failure++;
if (!xchg(&dbg_done, 1) && cur_ops->gp_kthread_dbg)
cur_ops->gp_kthread_dbg();
return true; /* failed */
}
@ -948,8 +954,8 @@ static int rcu_torture_boost(void *arg)
init_rcu_head_on_stack(&rbi.rcu);
/* Each pass through the following loop does one boost-test cycle. */
do {
/* Track if the test failed already in this test interval? */
bool failed = false;
bool failed = false; // Test failed already in this test interval
bool firsttime = true;
/* Increment n_rcu_torture_boosts once per boost-test */
while (!kthread_should_stop()) {
@ -975,18 +981,17 @@ static int rcu_torture_boost(void *arg)
/* Do one boost-test interval. */
endtime = oldstarttime + test_boost_duration * HZ;
call_rcu_time = jiffies;
while (time_before(jiffies, endtime)) {
/* If we don't have a callback in flight, post one. */
if (!smp_load_acquire(&rbi.inflight)) {
/* RCU core before ->inflight = 1. */
smp_store_release(&rbi.inflight, 1);
call_rcu(&rbi.rcu, rcu_torture_boost_cb);
cur_ops->call(&rbi.rcu, rcu_torture_boost_cb);
/* Check if the boost test failed */
failed = failed ||
rcu_torture_boost_failed(call_rcu_time,
jiffies);
if (!firsttime && !failed)
failed = rcu_torture_boost_failed(call_rcu_time, jiffies);
call_rcu_time = jiffies;
firsttime = false;
}
if (stutter_wait("rcu_torture_boost"))
sched_set_fifo_low(current);
@ -999,7 +1004,7 @@ static int rcu_torture_boost(void *arg)
* this case the boost check would never happen in the above
* loop so do another one here.
*/
if (!failed && smp_load_acquire(&rbi.inflight))
if (!firsttime && !failed && smp_load_acquire(&rbi.inflight))
rcu_torture_boost_failed(call_rcu_time, jiffies);
/*
@ -1025,6 +1030,9 @@ checkwait: if (stutter_wait("rcu_torture_boost"))
sched_set_fifo_low(current);
} while (!torture_must_stop());
while (smp_load_acquire(&rbi.inflight))
schedule_timeout_uninterruptible(1); // rcu_barrier() deadlocks.
/* Clean up and exit. */
while (!kthread_should_stop() || smp_load_acquire(&rbi.inflight)) {
torture_shutdown_absorb("rcu_torture_boost");
@ -1223,14 +1231,6 @@ rcu_torture_writer(void *arg)
WARN_ON_ONCE(1);
break;
}
if (cur_ops->get_gp_state && cur_ops->poll_gp_state)
WARN_ONCE(rcu_torture_writer_state != RTWS_DEF_FREE &&
!cur_ops->poll_gp_state(cookie),
"%s: Cookie check 2 failed %s(%d) %lu->%lu\n",
__func__,
rcu_torture_writer_state_getname(),
rcu_torture_writer_state,
cookie, cur_ops->get_gp_state());
}
WRITE_ONCE(rcu_torture_current_version,
rcu_torture_current_version + 1);
@ -1589,7 +1589,7 @@ static bool rcu_torture_one_read(struct torture_random_state *trsp, long myid)
preempt_enable();
if (cur_ops->get_gp_state && cur_ops->poll_gp_state)
WARN_ONCE(cur_ops->poll_gp_state(cookie),
"%s: Cookie check 3 failed %s(%d) %lu->%lu\n",
"%s: Cookie check 2 failed %s(%d) %lu->%lu\n",
__func__,
rcu_torture_writer_state_getname(),
rcu_torture_writer_state,
@ -1797,7 +1797,7 @@ rcu_torture_stats_print(void)
WARN_ON_ONCE(n_rcu_torture_barrier_error); // rcu_barrier()
WARN_ON_ONCE(n_rcu_torture_boost_ktrerror); // no boost kthread
WARN_ON_ONCE(n_rcu_torture_boost_rterror); // can't set RT prio
WARN_ON_ONCE(n_rcu_torture_boost_failure); // RCU boost failed
WARN_ON_ONCE(n_rcu_torture_boost_failure); // boost failed (TIMER_SOFTIRQ RT prio?)
WARN_ON_ONCE(i > 1); // Too-short grace period
}
pr_cont("Reader Pipe: ");
@ -1861,6 +1861,45 @@ rcu_torture_stats(void *arg)
torture_shutdown_absorb("rcu_torture_stats");
} while (!torture_must_stop());
torture_kthread_stopping("rcu_torture_stats");
{
struct rcu_head *rhp;
struct kmem_cache *kcp;
static int z;
kcp = kmem_cache_create("rcuscale", 136, 8, SLAB_STORE_USER, NULL);
rhp = kmem_cache_alloc(kcp, GFP_KERNEL);
pr_alert("mem_dump_obj() slab test: rcu_torture_stats = %px, &rhp = %px, rhp = %px, &z = %px\n", stats_task, &rhp, rhp, &z);
pr_alert("mem_dump_obj(ZERO_SIZE_PTR):");
mem_dump_obj(ZERO_SIZE_PTR);
pr_alert("mem_dump_obj(NULL):");
mem_dump_obj(NULL);
pr_alert("mem_dump_obj(%px):", &rhp);
mem_dump_obj(&rhp);
pr_alert("mem_dump_obj(%px):", rhp);
mem_dump_obj(rhp);
pr_alert("mem_dump_obj(%px):", &rhp->func);
mem_dump_obj(&rhp->func);
pr_alert("mem_dump_obj(%px):", &z);
mem_dump_obj(&z);
kmem_cache_free(kcp, rhp);
kmem_cache_destroy(kcp);
rhp = kmalloc(sizeof(*rhp), GFP_KERNEL);
pr_alert("mem_dump_obj() kmalloc test: rcu_torture_stats = %px, &rhp = %px, rhp = %px\n", stats_task, &rhp, rhp);
pr_alert("mem_dump_obj(kmalloc %px):", rhp);
mem_dump_obj(rhp);
pr_alert("mem_dump_obj(kmalloc %px):", &rhp->func);
mem_dump_obj(&rhp->func);
kfree(rhp);
rhp = vmalloc(4096);
pr_alert("mem_dump_obj() vmalloc test: rcu_torture_stats = %px, &rhp = %px, rhp = %px\n", stats_task, &rhp, rhp);
pr_alert("mem_dump_obj(vmalloc %px):", rhp);
mem_dump_obj(rhp);
pr_alert("mem_dump_obj(vmalloc %px):", &rhp->func);
mem_dump_obj(&rhp->func);
vfree(rhp);
}
return 0;
}
@ -1971,8 +2010,8 @@ static int rcu_torture_stall(void *args)
local_irq_disable();
else if (!stall_cpu_block)
preempt_disable();
pr_alert("rcu_torture_stall start on CPU %d.\n",
raw_smp_processor_id());
pr_alert("%s start on CPU %d.\n",
__func__, raw_smp_processor_id());
while (ULONG_CMP_LT((unsigned long)ktime_get_seconds(),
stop_at))
if (stall_cpu_block)
@ -1983,7 +2022,7 @@ static int rcu_torture_stall(void *args)
preempt_enable();
cur_ops->readunlock(idx);
}
pr_alert("rcu_torture_stall end.\n");
pr_alert("%s end.\n", __func__);
torture_shutdown_absorb("rcu_torture_stall");
while (!kthread_should_stop())
schedule_timeout_interruptible(10 * HZ);
@ -2595,6 +2634,8 @@ static bool rcu_torture_can_boost(void)
if (!(test_boost == 1 && cur_ops->can_boost) && test_boost != 2)
return false;
if (!cur_ops->call)
return false;
prio = rcu_get_gp_kthreads_prio();
if (!prio)

40
kernel/rcu/tasks.h
View File

@ -20,7 +20,7 @@ typedef void (*holdouts_func_t)(struct list_head *hop, bool ndrpt, bool *frptp);
typedef void (*postgp_func_t)(struct rcu_tasks *rtp);
/**
* Definition for a Tasks-RCU-like mechanism.
* struct rcu_tasks - Definition for a Tasks-RCU-like mechanism.
* @cbs_head: Head of callback list.
* @cbs_tail: Tail pointer for callback list.
* @cbs_wq: Wait queue allowning new callback to get kthread's attention.
@ -38,7 +38,7 @@ typedef void (*postgp_func_t)(struct rcu_tasks *rtp);
* @pregp_func: This flavor's pre-grace-period function (optional).
* @pertask_func: This flavor's per-task scan function (optional).
* @postscan_func: This flavor's post-task scan function (optional).
* @holdout_func: This flavor's holdout-list scan function (optional).
* @holdouts_func: This flavor's holdout-list scan function (optional).
* @postgp_func: This flavor's post-grace-period function (optional).
* @call_func: This flavor's call_rcu()-equivalent function.
* @name: This flavor's textual name.
@ -726,6 +726,42 @@ EXPORT_SYMBOL_GPL(show_rcu_tasks_rude_gp_kthread);
// flavors, rcu_preempt and rcu_sched. The fact that RCU Tasks Trace
// readers can operate from idle, offline, and exception entry/exit in no
// way allows rcu_preempt and rcu_sched readers to also do so.
//
// The implementation uses rcu_tasks_wait_gp(), which relies on function
// pointers in the rcu_tasks structure. The rcu_spawn_tasks_trace_kthread()
// function sets these function pointers up so that rcu_tasks_wait_gp()
// invokes these functions in this order:
//
// rcu_tasks_trace_pregp_step():
// Initialize the count of readers and block CPU-hotplug operations.
// rcu_tasks_trace_pertask(), invoked on every non-idle task:
// Initialize per-task state and attempt to identify an immediate
// quiescent state for that task, or, failing that, attempt to
// set that task's .need_qs flag so that task's next outermost
// rcu_read_unlock_trace() will report the quiescent state (in which
// case the count of readers is incremented). If both attempts fail,
// the task is added to a "holdout" list.
// rcu_tasks_trace_postscan():
// Initialize state and attempt to identify an immediate quiescent
// state as above (but only for idle tasks), unblock CPU-hotplug
// operations, and wait for an RCU grace period to avoid races with
// tasks that are in the process of exiting.
// check_all_holdout_tasks_trace(), repeatedly until holdout list is empty:
// Scans the holdout list, attempting to identify a quiescent state
// for each task on the list. If there is a quiescent state, the
// corresponding task is removed from the holdout list.
// rcu_tasks_trace_postgp():
// Wait for the count of readers do drop to zero, reporting any stalls.
// Also execute full memory barriers to maintain ordering with code
// executing after the grace period.
//
// The exit_tasks_rcu_finish_trace() synchronizes with exiting tasks.
//
// Pre-grace-period update-side code is ordered before the grace
// period via the ->cbs_lock and barriers in rcu_tasks_kthread().
// Pre-grace-period read-side code is ordered before the grace period by
// atomic_dec_and_test() of the count of readers (for IPIed readers) and by
// scheduler context-switch ordering (for locked-down non-running readers).
// The lockdep state must be outside of #ifdef to be useful.
#ifdef CONFIG_DEBUG_LOCK_ALLOC

40
kernel/rcu/tiny.c
View File

@ -32,12 +32,14 @@ struct rcu_ctrlblk {
struct rcu_head *rcucblist; /* List of pending callbacks (CBs). */
struct rcu_head **donetail; /* ->next pointer of last "done" CB. */
struct rcu_head **curtail; /* ->next pointer of last CB. */
unsigned long gp_seq; /* Grace-period counter. */
};
/* Definition for rcupdate control block. */
static struct rcu_ctrlblk rcu_ctrlblk = {
.donetail = &rcu_ctrlblk.rcucblist,
.curtail = &rcu_ctrlblk.rcucblist,
.gp_seq = 0 - 300UL,
};
void rcu_barrier(void)
@ -56,6 +58,7 @@ void rcu_qs(void)
rcu_ctrlblk.donetail = rcu_ctrlblk.curtail;
raise_softirq_irqoff(RCU_SOFTIRQ);
}
WRITE_ONCE(rcu_ctrlblk.gp_seq, rcu_ctrlblk.gp_seq + 1);
local_irq_restore(flags);
}
@ -177,6 +180,43 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func)
}
EXPORT_SYMBOL_GPL(call_rcu);
/*
* Return a grace-period-counter "cookie". For more information,
* see the Tree RCU header comment.
*/
unsigned long get_state_synchronize_rcu(void)
{
return READ_ONCE(rcu_ctrlblk.gp_seq);
}
EXPORT_SYMBOL_GPL(get_state_synchronize_rcu);
/*
* Return a grace-period-counter "cookie" and ensure that a future grace
* period completes. For more information, see the Tree RCU header comment.
*/
unsigned long start_poll_synchronize_rcu(void)
{
unsigned long gp_seq = get_state_synchronize_rcu();
if (unlikely(is_idle_task(current))) {
/* force scheduling for rcu_qs() */
resched_cpu(0);
}
return gp_seq;
}
EXPORT_SYMBOL_GPL(start_poll_synchronize_rcu);
/*
* Return true if the grace period corresponding to oldstate has completed
* and false otherwise. For more information, see the Tree RCU header
* comment.
*/
bool poll_state_synchronize_rcu(unsigned long oldstate)
{
return READ_ONCE(rcu_ctrlblk.gp_seq) != oldstate;
}
EXPORT_SYMBOL_GPL(poll_state_synchronize_rcu);
void __init rcu_init(void)
{
open_softirq(RCU_SOFTIRQ, rcu_process_callbacks);

169
kernel/rcu/tree.c
View File

@ -156,6 +156,7 @@ static void invoke_rcu_core(void);
static void rcu_report_exp_rdp(struct rcu_data *rdp);
static void sync_sched_exp_online_cleanup(int cpu);
static void check_cb_ovld_locked(struct rcu_data *rdp, struct rcu_node *rnp);
static bool rcu_rdp_is_offloaded(struct rcu_data *rdp);
/* rcuc/rcub kthread realtime priority */
static int kthread_prio = IS_ENABLED(CONFIG_RCU_BOOST) ? 1 : 0;
@ -648,7 +649,6 @@ static noinstr void rcu_eqs_enter(bool user)
instrumentation_begin();
trace_rcu_dyntick(TPS("Start"), rdp->dynticks_nesting, 0, atomic_read(&rdp->dynticks));
WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current));
rdp = this_cpu_ptr(&rcu_data);
rcu_prepare_for_idle();
rcu_preempt_deferred_qs(current);
@ -1077,7 +1077,6 @@ noinstr void rcu_nmi_enter(void)
} else if (!in_nmi()) {
instrumentation_begin();
rcu_irq_enter_check_tick();
instrumentation_end();
} else {
instrumentation_begin();
}
@ -1672,7 +1671,7 @@ static bool __note_gp_changes(struct rcu_node *rnp, struct rcu_data *rdp)
{
bool ret = false;
bool need_qs;
const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
const bool offloaded = rcu_rdp_is_offloaded(rdp);
raw_lockdep_assert_held_rcu_node(rnp);
@ -2128,7 +2127,7 @@ static void rcu_gp_cleanup(void)
needgp = true;
}
/* Advance CBs to reduce false positives below. */
offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
offloaded = rcu_rdp_is_offloaded(rdp);
if ((offloaded || !rcu_accelerate_cbs(rnp, rdp)) && needgp) {
WRITE_ONCE(rcu_state.gp_flags, RCU_GP_FLAG_INIT);
WRITE_ONCE(rcu_state.gp_req_activity, jiffies);
@ -2327,7 +2326,7 @@ rcu_report_qs_rdp(struct rcu_data *rdp)
unsigned long flags;
unsigned long mask;
bool needwake = false;
const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
const bool offloaded = rcu_rdp_is_offloaded(rdp);
struct rcu_node *rnp;
WARN_ON_ONCE(rdp->cpu != smp_processor_id());
@ -2414,7 +2413,7 @@ int rcutree_dying_cpu(unsigned int cpu)
blkd = !!(rnp->qsmask & rdp->grpmask);
trace_rcu_grace_period(rcu_state.name, READ_ONCE(rnp->gp_seq),
blkd ? TPS("cpuofl") : TPS("cpuofl-bgp"));
blkd ? TPS("cpuofl-bgp") : TPS("cpuofl"));
return 0;
}
@ -2497,7 +2496,7 @@ static void rcu_do_batch(struct rcu_data *rdp)
int div;
bool __maybe_unused empty;
unsigned long flags;
const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
const bool offloaded = rcu_rdp_is_offloaded(rdp);
struct rcu_head *rhp;
struct rcu_cblist rcl = RCU_CBLIST_INITIALIZER(rcl);
long bl, count = 0;
@ -3066,7 +3065,7 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func)
trace_rcu_segcb_stats(&rdp->cblist, TPS("SegCBQueued"));
/* Go handle any RCU core processing required. */
if (unlikely(rcu_segcblist_is_offloaded(&rdp->cblist))) {
if (unlikely(rcu_rdp_is_offloaded(rdp))) {
__call_rcu_nocb_wake(rdp, was_alldone, flags); /* unlocks */
} else {
__call_rcu_core(rdp, head, flags);
@ -3229,8 +3228,7 @@ krc_this_cpu_lock(unsigned long *flags)
static inline void
krc_this_cpu_unlock(struct kfree_rcu_cpu *krcp, unsigned long flags)
{
raw_spin_unlock(&krcp->lock);
local_irq_restore(flags);
raw_spin_unlock_irqrestore(&krcp->lock, flags);
}
static inline struct kvfree_rcu_bulk_data *
@ -3464,7 +3462,7 @@ static void fill_page_cache_func(struct work_struct *work)
for (i = 0; i < rcu_min_cached_objs; i++) {
bnode = (struct kvfree_rcu_bulk_data *)
__get_free_page(GFP_KERNEL | __GFP_NOWARN);
__get_free_page(GFP_KERNEL | __GFP_NORETRY | __GFP_NOMEMALLOC | __GFP_NOWARN);
if (bnode) {
raw_spin_lock_irqsave(&krcp->lock, flags);
@ -3493,37 +3491,62 @@ run_page_cache_worker(struct kfree_rcu_cpu *krcp)
}
}
// Record ptr in a page managed by krcp, with the pre-krc_this_cpu_lock()
// state specified by flags. If can_alloc is true, the caller must
// be schedulable and not be holding any locks or mutexes that might be
// acquired by the memory allocator or anything that it might invoke.
// Returns true if ptr was successfully recorded, else the caller must
// use a fallback.
static inline bool
kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr)
add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp,
unsigned long *flags, void *ptr, bool can_alloc)
{
struct kvfree_rcu_bulk_data *bnode;
int idx;
if (unlikely(!krcp->initialized))
*krcp = krc_this_cpu_lock(flags);
if (unlikely(!(*krcp)->initialized))
return false;
lockdep_assert_held(&krcp->lock);
idx = !!is_vmalloc_addr(ptr);
/* Check if a new block is required. */
if (!krcp->bkvhead[idx] ||
krcp->bkvhead[idx]->nr_records == KVFREE_BULK_MAX_ENTR) {
bnode = get_cached_bnode(krcp);
/* Switch to emergency path. */
if (!(*krcp)->bkvhead[idx] ||
(*krcp)->bkvhead[idx]->nr_records == KVFREE_BULK_MAX_ENTR) {
bnode = get_cached_bnode(*krcp);
if (!bnode && can_alloc) {
krc_this_cpu_unlock(*krcp, *flags);
// __GFP_NORETRY - allows a light-weight direct reclaim
// what is OK from minimizing of fallback hitting point of
// view. Apart of that it forbids any OOM invoking what is
// also beneficial since we are about to release memory soon.
//
// __GFP_NOMEMALLOC - prevents from consuming of all the
// memory reserves. Please note we have a fallback path.
//
// __GFP_NOWARN - it is supposed that an allocation can
// be failed under low memory or high memory pressure
// scenarios.
bnode = (struct kvfree_rcu_bulk_data *)
__get_free_page(GFP_KERNEL | __GFP_NORETRY | __GFP_NOMEMALLOC | __GFP_NOWARN);
*krcp = krc_this_cpu_lock(flags);
}
if (!bnode)
return false;
/* Initialize the new block. */
bnode->nr_records = 0;
bnode->next = krcp->bkvhead[idx];
bnode->next = (*krcp)->bkvhead[idx];
/* Attach it to the head. */
krcp->bkvhead[idx] = bnode;
(*krcp)->bkvhead[idx] = bnode;
}
/* Finally insert. */
krcp->bkvhead[idx]->records
[krcp->bkvhead[idx]->nr_records++] = ptr;
(*krcp)->bkvhead[idx]->records
[(*krcp)->bkvhead[idx]->nr_records++] = ptr;
return true;
}
@ -3561,8 +3584,6 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
ptr = (unsigned long *) func;
}
krcp = krc_this_cpu_lock(&flags);
// Queue the object but don't yet schedule the batch.
if (debug_rcu_head_queue(ptr)) {
// Probable double kfree_rcu(), just leak.
@ -3570,12 +3591,11 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
__func__, head);
// Mark as success and leave.
success = true;
goto unlock_return;
return;
}
kasan_record_aux_stack(ptr);
success = kvfree_call_rcu_add_ptr_to_bulk(krcp, ptr);
success = add_ptr_to_bulk_krc_lock(&krcp, &flags, ptr, !head);
if (!success) {
run_page_cache_worker(krcp);
@ -3774,8 +3794,8 @@ EXPORT_SYMBOL_GPL(synchronize_rcu);
* get_state_synchronize_rcu - Snapshot current RCU state
*
* Returns a cookie that is used by a later call to cond_synchronize_rcu()
* to determine whether or not a full grace period has elapsed in the
* meantime.
* or poll_state_synchronize_rcu() to determine whether or not a full
* grace period has elapsed in the meantime.
*/
unsigned long get_state_synchronize_rcu(void)
{
@ -3788,14 +3808,77 @@ unsigned long get_state_synchronize_rcu(void)
}
EXPORT_SYMBOL_GPL(get_state_synchronize_rcu);
/**
* start_poll_synchronize_rcu - Snapshot and start RCU grace period
*
* Returns a cookie that is used by a later call to cond_synchronize_rcu()
* or poll_state_synchronize_rcu() to determine whether or not a full
* grace period has elapsed in the meantime. If the needed grace period
* is not already slated to start, notifies RCU core of the need for that
* grace period.
*
* Interrupts must be enabled for the case where it is necessary to awaken
* the grace-period kthread.
*/
unsigned long start_poll_synchronize_rcu(void)
{
unsigned long flags;
unsigned long gp_seq = get_state_synchronize_rcu();
bool needwake;
struct rcu_data *rdp;
struct rcu_node *rnp;
lockdep_assert_irqs_enabled();
local_irq_save(flags);
rdp = this_cpu_ptr(&rcu_data);
rnp = rdp->mynode;
raw_spin_lock_rcu_node(rnp); // irqs already disabled.
needwake = rcu_start_this_gp(rnp, rdp, gp_seq);
raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
if (needwake)
rcu_gp_kthread_wake();
return gp_seq;
}
EXPORT_SYMBOL_GPL(start_poll_synchronize_rcu);
/**
* poll_state_synchronize_rcu - Conditionally wait for an RCU grace period
*
* @oldstate: return from call to get_state_synchronize_rcu() or start_poll_synchronize_rcu()
*
* If a full RCU grace period has elapsed since the earlier call from
* which oldstate was obtained, return @true, otherwise return @false.
* If @false is returned, it is the caller's responsibilty to invoke this
* function later on until it does return @true. Alternatively, the caller
* can explicitly wait for a grace period, for example, by passing @oldstate
* to cond_synchronize_rcu() or by directly invoking synchronize_rcu().
*
* Yes, this function does not take counter wrap into account.
* But counter wrap is harmless. If the counter wraps, we have waited for
* more than 2 billion grace periods (and way more on a 64-bit system!).
* Those needing to keep oldstate values for very long time periods
* (many hours even on 32-bit systems) should check them occasionally
* and either refresh them or set a flag indicating that the grace period
* has completed.
*/
bool poll_state_synchronize_rcu(unsigned long oldstate)
{
if (rcu_seq_done(&rcu_state.gp_seq, oldstate)) {
smp_mb(); /* Ensure GP ends before subsequent accesses. */
return true;
}
return false;
}
EXPORT_SYMBOL_GPL(poll_state_synchronize_rcu);
/**
* cond_synchronize_rcu - Conditionally wait for an RCU grace period
*
* @oldstate: return value from earlier call to get_state_synchronize_rcu()
*
* If a full RCU grace period has elapsed since the earlier call to
* get_state_synchronize_rcu(), just return. Otherwise, invoke
* synchronize_rcu() to wait for a full grace period.
* get_state_synchronize_rcu() or start_poll_synchronize_rcu(), just return.
* Otherwise, invoke synchronize_rcu() to wait for a full grace period.
*
* Yes, this function does not take counter wrap into account. But
* counter wrap is harmless. If the counter wraps, we have waited for
@ -3804,10 +3887,8 @@ EXPORT_SYMBOL_GPL(get_state_synchronize_rcu);
*/
void cond_synchronize_rcu(unsigned long oldstate)
{
if (!rcu_seq_done(&rcu_state.gp_seq, oldstate))
if (!poll_state_synchronize_rcu(oldstate))
synchronize_rcu();
else
smp_mb(); /* Ensure GP ends before subsequent accesses. */
}
EXPORT_SYMBOL_GPL(cond_synchronize_rcu);
@ -3843,13 +3924,13 @@ static int rcu_pending(int user)
return 1;
/* Does this CPU have callbacks ready to invoke? */
if (!rcu_segcblist_is_offloaded(&rdp->cblist) &&
if (!rcu_rdp_is_offloaded(rdp) &&
rcu_segcblist_ready_cbs(&rdp->cblist))
return 1;
/* Has RCU gone idle with this CPU needing another grace period? */
if (!gp_in_progress && rcu_segcblist_is_enabled(&rdp->cblist) &&
!rcu_segcblist_is_offloaded(&rdp->cblist) &&
!rcu_rdp_is_offloaded(rdp) &&
!rcu_segcblist_restempty(&rdp->cblist, RCU_NEXT_READY_TAIL))
return 1;
@ -3968,7 +4049,7 @@ void rcu_barrier(void)
for_each_possible_cpu(cpu) {
rdp = per_cpu_ptr(&rcu_data, cpu);
if (cpu_is_offline(cpu) &&
!rcu_segcblist_is_offloaded(&rdp->cblist))
!rcu_rdp_is_offloaded(rdp))
continue;
if (rcu_segcblist_n_cbs(&rdp->cblist) && cpu_online(cpu)) {
rcu_barrier_trace(TPS("OnlineQ"), cpu,
@ -4083,15 +4164,13 @@ int rcutree_prepare_cpu(unsigned int cpu)
rdp->dynticks_nesting = 1; /* CPU not up, no tearing. */
rcu_dynticks_eqs_online();
raw_spin_unlock_rcu_node(rnp); /* irqs remain disabled. */
/*
* Lock in case the CB/GP kthreads are still around handling
* old callbacks (longer term we should flush all callbacks
* before completing CPU offline)
* Only non-NOCB CPUs that didn't have early-boot callbacks need to be
* (re-)initialized.
*/
rcu_nocb_lock(rdp);
if (rcu_segcblist_empty(&rdp->cblist)) /* No early-boot CBs? */
if (!rcu_segcblist_is_enabled(&rdp->cblist))
rcu_segcblist_init(&rdp->cblist); /* Re-enable callbacks. */
rcu_nocb_unlock(rdp);
/*
* Add CPU to leaf rcu_node pending-online bitmask. Any needed
@ -4291,7 +4370,7 @@ void rcutree_migrate_callbacks(int cpu)
struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);
bool needwake;
if (rcu_segcblist_is_offloaded(&rdp->cblist) ||
if (rcu_rdp_is_offloaded(rdp) ||
rcu_segcblist_empty(&rdp->cblist))
return; /* No callbacks to migrate. */
@ -4309,7 +4388,7 @@ void rcutree_migrate_callbacks(int cpu)
rcu_segcblist_disable(&rdp->cblist);
WARN_ON_ONCE(rcu_segcblist_empty(&my_rdp->cblist) !=
!rcu_segcblist_n_cbs(&my_rdp->cblist));
if (rcu_segcblist_is_offloaded(&my_rdp->cblist)) {
if (rcu_rdp_is_offloaded(my_rdp)) {
raw_spin_unlock_rcu_node(my_rnp); /* irqs remain disabled. */
__call_rcu_nocb_wake(my_rdp, true, flags);
} else {

1
kernel/rcu/tree_exp.h
View File

@ -521,6 +521,7 @@ static void synchronize_rcu_expedited_wait(void)
if (rcu_stall_is_suppressed())
continue;
panic_on_rcu_stall();
trace_rcu_stall_warning(rcu_state.name, TPS("ExpeditedStall"));
pr_err("INFO: %s detected expedited stalls on CPUs/tasks: {",
rcu_state.name);
ndetected = 0;

254
kernel/rcu/tree_plugin.h
View File

@ -16,8 +16,70 @@
#ifdef CONFIG_RCU_NOCB_CPU
static cpumask_var_t rcu_nocb_mask; /* CPUs to have callbacks offloaded. */
static bool __read_mostly rcu_nocb_poll; /* Offload kthread are to poll. */
static inline int rcu_lockdep_is_held_nocb(struct rcu_data *rdp)
{
return lockdep_is_held(&rdp->nocb_lock);
}
static inline bool rcu_current_is_nocb_kthread(struct rcu_data *rdp)
{
/* Race on early boot between thread creation and assignment */
if (!rdp->nocb_cb_kthread || !rdp->nocb_gp_kthread)
return true;
if (current == rdp->nocb_cb_kthread || current == rdp->nocb_gp_kthread)
if (in_task())
return true;
return false;
}
static inline bool rcu_running_nocb_timer(struct rcu_data *rdp)
{
return (timer_curr_running(&rdp->nocb_timer) && !in_irq());
}
#else
static inline int rcu_lockdep_is_held_nocb(struct rcu_data *rdp)
{
return 0;
}
static inline bool rcu_current_is_nocb_kthread(struct rcu_data *rdp)
{
return false;
}
static inline bool rcu_running_nocb_timer(struct rcu_data *rdp)
{
return false;
}
#endif /* #ifdef CONFIG_RCU_NOCB_CPU */
static bool rcu_rdp_is_offloaded(struct rcu_data *rdp)
{
/*
* In order to read the offloaded state of an rdp is a safe
* and stable way and prevent from its value to be changed
* under us, we must either hold the barrier mutex, the cpu
* hotplug lock (read or write) or the nocb lock. Local
* non-preemptible reads are also safe. NOCB kthreads and
* timers have their own means of synchronization against the
* offloaded state updaters.
*/
RCU_LOCKDEP_WARN(
!(lockdep_is_held(&rcu_state.barrier_mutex) ||
(IS_ENABLED(CONFIG_HOTPLUG_CPU) && lockdep_is_cpus_held()) ||
rcu_lockdep_is_held_nocb(rdp) ||
(rdp == this_cpu_ptr(&rcu_data) &&
!(IS_ENABLED(CONFIG_PREEMPT_COUNT) && preemptible())) ||
rcu_current_is_nocb_kthread(rdp) ||
rcu_running_nocb_timer(rdp)),
"Unsafe read of RCU_NOCB offloaded state"
);
return rcu_segcblist_is_offloaded(&rdp->cblist);
}
/*
* Check the RCU kernel configuration parameters and print informative
* messages about anything out of the ordinary.
@ -393,8 +455,9 @@ void __rcu_read_unlock(void)
{
struct task_struct *t = current;
barrier(); // critical section before exit code.
if (rcu_preempt_read_exit() == 0) {
barrier(); /* critical section before exit code. */
barrier(); // critical-section exit before .s check.
if (unlikely(READ_ONCE(t->rcu_read_unlock_special.s)))
rcu_read_unlock_special(t);
}
@ -598,9 +661,9 @@ static void rcu_preempt_deferred_qs_handler(struct irq_work *iwp)
static void rcu_read_unlock_special(struct task_struct *t)
{
unsigned long flags;
bool irqs_were_disabled;
bool preempt_bh_were_disabled =
!!(preempt_count() & (PREEMPT_MASK | SOFTIRQ_MASK));
bool irqs_were_disabled;
/* NMI handlers cannot block and cannot safely manipulate state. */
if (in_nmi())
@ -609,30 +672,33 @@ static void rcu_read_unlock_special(struct task_struct *t)
local_irq_save(flags);
irqs_were_disabled = irqs_disabled_flags(flags);
if (preempt_bh_were_disabled || irqs_were_disabled) {
bool exp;
bool expboost; // Expedited GP in flight or possible boosting.
struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
struct rcu_node *rnp = rdp->mynode;
exp = (t->rcu_blocked_node &&
READ_ONCE(t->rcu_blocked_node->exp_tasks)) ||
(rdp->grpmask & READ_ONCE(rnp->expmask));
expboost = (t->rcu_blocked_node && READ_ONCE(t->rcu_blocked_node->exp_tasks)) ||
(rdp->grpmask & READ_ONCE(rnp->expmask)) ||
IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) ||
(IS_ENABLED(CONFIG_RCU_BOOST) && irqs_were_disabled &&
t->rcu_blocked_node);
// Need to defer quiescent state until everything is enabled.
if (use_softirq && (in_irq() || (exp && !irqs_were_disabled))) {
if (use_softirq && (in_irq() || (expboost && !irqs_were_disabled))) {
// Using softirq, safe to awaken, and either the
// wakeup is free or there is an expedited GP.
// wakeup is free or there is either an expedited
// GP in flight or a potential need to deboost.
raise_softirq_irqoff(RCU_SOFTIRQ);
} else {
// Enabling BH or preempt does reschedule, so...
// Also if no expediting, slow is OK.
// Plus nohz_full CPUs eventually get tick enabled.
// Also if no expediting and no possible deboosting,
// slow is OK. Plus nohz_full CPUs eventually get
// tick enabled.
set_tsk_need_resched(current);
set_preempt_need_resched();
if (IS_ENABLED(CONFIG_IRQ_WORK) && irqs_were_disabled &&
!rdp->defer_qs_iw_pending && exp && cpu_online(rdp->cpu)) {
expboost && !rdp->defer_qs_iw_pending && cpu_online(rdp->cpu)) {
// Get scheduler to re-evaluate and call hooks.
// If !IRQ_WORK, FQS scan will eventually IPI.
init_irq_work(&rdp->defer_qs_iw,
rcu_preempt_deferred_qs_handler);
init_irq_work(&rdp->defer_qs_iw, rcu_preempt_deferred_qs_handler);
rdp->defer_qs_iw_pending = true;
irq_work_queue_on(&rdp->defer_qs_iw, rdp->cpu);
}
@ -1257,7 +1323,7 @@ int rcu_needs_cpu(u64 basemono, u64 *nextevt)
{
*nextevt = KTIME_MAX;
return !rcu_segcblist_empty(&this_cpu_ptr(&rcu_data)->cblist) &&
!rcu_segcblist_is_offloaded(&this_cpu_ptr(&rcu_data)->cblist);
!rcu_rdp_is_offloaded(this_cpu_ptr(&rcu_data));
}
/*
@ -1352,7 +1418,7 @@ int rcu_needs_cpu(u64 basemono, u64 *nextevt)
/* If no non-offloaded callbacks, RCU doesn't need the CPU. */
if (rcu_segcblist_empty(&rdp->cblist) ||
rcu_segcblist_is_offloaded(&this_cpu_ptr(&rcu_data)->cblist)) {
rcu_rdp_is_offloaded(rdp)) {
*nextevt = KTIME_MAX;
return 0;
}
@ -1388,7 +1454,7 @@ static void rcu_prepare_for_idle(void)
int tne;
lockdep_assert_irqs_disabled();
if (rcu_segcblist_is_offloaded(&rdp->cblist))
if (rcu_rdp_is_offloaded(rdp))
return;
/* Handle nohz enablement switches conservatively. */
@ -1429,7 +1495,7 @@ static void rcu_cleanup_after_idle(void)
struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
lockdep_assert_irqs_disabled();
if (rcu_segcblist_is_offloaded(&rdp->cblist))
if (rcu_rdp_is_offloaded(rdp))
return;
if (rcu_try_advance_all_cbs())
invoke_rcu_core();
@ -1464,14 +1530,12 @@ static void rcu_cleanup_after_idle(void)
/*
* Parse the boot-time rcu_nocb_mask CPU list from the kernel parameters.
* The string after the "rcu_nocbs=" is either "all" for all CPUs, or a
* comma-separated list of CPUs and/or CPU ranges. If an invalid list is
* given, a warning is emitted and all CPUs are offloaded.
* If the list is invalid, a warning is emitted and all CPUs are offloaded.
*/
static int __init rcu_nocb_setup(char *str)
{
alloc_bootmem_cpumask_var(&rcu_nocb_mask);
if (!strcasecmp(str, "all"))
if (!strcasecmp(str, "all")) /* legacy: use "0-N" instead */
cpumask_setall(rcu_nocb_mask);
else
if (cpulist_parse(str, rcu_nocb_mask)) {
@ -1494,7 +1558,7 @@ early_param("rcu_nocb_poll", parse_rcu_nocb_poll);
* After all, the main point of bypassing is to avoid lock contention
* on ->nocb_lock, which only can happen at high call_rcu() rates.
*/
int nocb_nobypass_lim_per_jiffy = 16 * 1000 / HZ;
static int nocb_nobypass_lim_per_jiffy = 16 * 1000 / HZ;
module_param(nocb_nobypass_lim_per_jiffy, int, 0);
/*
@ -1560,7 +1624,7 @@ static void rcu_nocb_bypass_unlock(struct rcu_data *rdp)
static void rcu_nocb_lock(struct rcu_data *rdp)
{
lockdep_assert_irqs_disabled();
if (!rcu_segcblist_is_offloaded(&rdp->cblist))
if (!rcu_rdp_is_offloaded(rdp))
return;
raw_spin_lock(&rdp->nocb_lock);
}
@ -1571,7 +1635,7 @@ static void rcu_nocb_lock(struct rcu_data *rdp)
*/
static void rcu_nocb_unlock(struct rcu_data *rdp)
{
if (rcu_segcblist_is_offloaded(&rdp->cblist)) {
if (rcu_rdp_is_offloaded(rdp)) {
lockdep_assert_irqs_disabled();
raw_spin_unlock(&rdp->nocb_lock);
}
@ -1584,7 +1648,7 @@ static void rcu_nocb_unlock(struct rcu_data *rdp)
static void rcu_nocb_unlock_irqrestore(struct rcu_data *rdp,
unsigned long flags)
{
if (rcu_segcblist_is_offloaded(&rdp->cblist)) {
if (rcu_rdp_is_offloaded(rdp)) {
lockdep_assert_irqs_disabled();
raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
} else {
@ -1596,7 +1660,7 @@ static void rcu_nocb_unlock_irqrestore(struct rcu_data *rdp,
static void rcu_lockdep_assert_cblist_protected(struct rcu_data *rdp)
{
lockdep_assert_irqs_disabled();
if (rcu_segcblist_is_offloaded(&rdp->cblist))
if (rcu_rdp_is_offloaded(rdp))
lockdep_assert_held(&rdp->nocb_lock);
}
@ -1641,12 +1705,16 @@ static bool wake_nocb_gp(struct rcu_data *rdp, bool force,
lockdep_assert_held(&rdp->nocb_lock);
if (!READ_ONCE(rdp_gp->nocb_gp_kthread)) {
rcu_nocb_unlock_irqrestore(rdp, flags);
trace_rcu_nocb_wake(rcu_state.name, rdp->cpu,
TPS("AlreadyAwake"));
rcu_nocb_unlock_irqrestore(rdp, flags);
return false;
}
del_timer(&rdp->nocb_timer);
if (READ_ONCE(rdp->nocb_defer_wakeup) > RCU_NOCB_WAKE_NOT) {
WRITE_ONCE(rdp->nocb_defer_wakeup, RCU_NOCB_WAKE_NOT);
del_timer(&rdp->nocb_timer);
}
rcu_nocb_unlock_irqrestore(rdp, flags);
raw_spin_lock_irqsave(&rdp_gp->nocb_gp_lock, flags);
if (force || READ_ONCE(rdp_gp->nocb_gp_sleep)) {
@ -1690,7 +1758,7 @@ static bool rcu_nocb_do_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp,
{
struct rcu_cblist rcl;
WARN_ON_ONCE(!rcu_segcblist_is_offloaded(&rdp->cblist));
WARN_ON_ONCE(!rcu_rdp_is_offloaded(rdp));
rcu_lockdep_assert_cblist_protected(rdp);
lockdep_assert_held(&rdp->nocb_bypass_lock);
if (rhp && !rcu_cblist_n_cbs(&rdp->nocb_bypass)) {
@ -1718,7 +1786,7 @@ static bool rcu_nocb_do_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp,
static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp,
unsigned long j)
{
if (!rcu_segcblist_is_offloaded(&rdp->cblist))
if (!rcu_rdp_is_offloaded(rdp))
return true;
rcu_lockdep_assert_cblist_protected(rdp);
rcu_nocb_bypass_lock(rdp);
@ -1732,7 +1800,7 @@ static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp,
static void rcu_nocb_try_flush_bypass(struct rcu_data *rdp, unsigned long j)
{
rcu_lockdep_assert_cblist_protected(rdp);
if (!rcu_segcblist_is_offloaded(&rdp->cblist) ||
if (!rcu_rdp_is_offloaded(rdp) ||
!rcu_nocb_bypass_trylock(rdp))
return;
WARN_ON_ONCE(!rcu_nocb_do_flush_bypass(rdp, NULL, j));
@ -1764,11 +1832,22 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp,
unsigned long j = jiffies;
long ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass);
if (!rcu_segcblist_is_offloaded(&rdp->cblist)) {
lockdep_assert_irqs_disabled();
// Pure softirq/rcuc based processing: no bypassing, no
// locking.
if (!rcu_rdp_is_offloaded(rdp)) {
*was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist);
return false;
}
// In the process of (de-)offloading: no bypassing, but
// locking.
if (!rcu_segcblist_completely_offloaded(&rdp->cblist)) {
rcu_nocb_lock(rdp);
*was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist);
return false; /* Not offloaded, no bypassing. */
}
lockdep_assert_irqs_disabled();
// Don't use ->nocb_bypass during early boot.
if (rcu_scheduler_active != RCU_SCHEDULER_RUNNING) {
@ -1878,9 +1957,9 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone,
// If we are being polled or there is no kthread, just leave.
t = READ_ONCE(rdp->nocb_gp_kthread);
if (rcu_nocb_poll || !t) {
rcu_nocb_unlock_irqrestore(rdp, flags);
trace_rcu_nocb_wake(rcu_state.name, rdp->cpu,
TPS("WakeNotPoll"));
rcu_nocb_unlock_irqrestore(rdp, flags);
return;
}
// Need to actually to a wakeup.
@ -1915,8 +1994,8 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone,
TPS("WakeOvfIsDeferred"));
rcu_nocb_unlock_irqrestore(rdp, flags);
} else {
trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WakeNot"));
rcu_nocb_unlock_irqrestore(rdp, flags);
trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WakeNot"));
}
return;
}
@ -1954,7 +2033,8 @@ static inline bool nocb_gp_enabled_cb(struct rcu_data *rdp)
return rcu_segcblist_test_flags(&rdp->cblist, flags);
}
static inline bool nocb_gp_update_state(struct rcu_data *rdp, bool *needwake_state)
static inline bool nocb_gp_update_state_deoffloading(struct rcu_data *rdp,
bool *needwake_state)
{
struct rcu_segcblist *cblist = &rdp->cblist;
@ -1964,7 +2044,7 @@ static inline bool nocb_gp_update_state(struct rcu_data *rdp, bool *needwake_sta
if (rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB))
*needwake_state = true;
}
return true;
return false;
}
/*
@ -1975,7 +2055,7 @@ static inline bool nocb_gp_update_state(struct rcu_data *rdp, bool *needwake_sta
rcu_segcblist_clear_flags(cblist, SEGCBLIST_KTHREAD_GP);
if (!rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB))
*needwake_state = true;
return false;
return true;
}
@ -2013,7 +2093,7 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
continue;
trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Check"));
rcu_nocb_lock_irqsave(rdp, flags);
if (!nocb_gp_update_state(rdp, &needwake_state)) {
if (nocb_gp_update_state_deoffloading(rdp, &needwake_state)) {
rcu_nocb_unlock_irqrestore(rdp, flags);
if (needwake_state)
swake_up_one(&rdp->nocb_state_wq);
@ -2168,11 +2248,18 @@ static void nocb_cb_wait(struct rcu_data *rdp)
unsigned long flags;
bool needwake_state = false;
bool needwake_gp = false;
bool can_sleep = true;
struct rcu_node *rnp = rdp->mynode;
local_irq_save(flags);
rcu_momentary_dyntick_idle();
local_irq_restore(flags);
/*
* Disable BH to provide the expected environment. Also, when
* transitioning to/from NOCB mode, a self-requeuing callback might
* be invoked from softirq. A short grace period could cause both
* instances of this callback would execute concurrently.
*/
local_bh_disable();
rcu_do_batch(rdp);
local_bh_enable();
@ -2185,8 +2272,6 @@ static void nocb_cb_wait(struct rcu_data *rdp)
raw_spin_unlock_rcu_node(rnp); /* irqs remain disabled. */
}
WRITE_ONCE(rdp->nocb_cb_sleep, true);
if (rcu_segcblist_test_flags(cblist, SEGCBLIST_OFFLOADED)) {
if (!rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB)) {
rcu_segcblist_set_flags(cblist, SEGCBLIST_KTHREAD_CB);
@ -2194,7 +2279,7 @@ static void nocb_cb_wait(struct rcu_data *rdp)
needwake_state = true;
}
if (rcu_segcblist_ready_cbs(cblist))
WRITE_ONCE(rdp->nocb_cb_sleep, false);
can_sleep = false;
} else {
/*
* De-offloading. Clear our flag and notify the de-offload worker.
@ -2207,6 +2292,8 @@ static void nocb_cb_wait(struct rcu_data *rdp)
needwake_state = true;
}
WRITE_ONCE(rdp->nocb_cb_sleep, can_sleep);
if (rdp->nocb_cb_sleep)
trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("CBSleep"));
@ -2265,7 +2352,6 @@ static bool do_nocb_deferred_wakeup_common(struct rcu_data *rdp)
return false;
}
ndw = READ_ONCE(rdp->nocb_defer_wakeup);
WRITE_ONCE(rdp->nocb_defer_wakeup, RCU_NOCB_WAKE_NOT);
ret = wake_nocb_gp(rdp, ndw == RCU_NOCB_WAKE_FORCE, flags);
trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("DeferredWake"));
@ -2331,24 +2417,28 @@ static int rdp_offload_toggle(struct rcu_data *rdp,
return 0;
}
static int __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)
static long rcu_nocb_rdp_deoffload(void *arg)
{
struct rcu_data *rdp = arg;
struct rcu_segcblist *cblist = &rdp->cblist;
unsigned long flags;
int ret;
WARN_ON_ONCE(rdp->cpu != raw_smp_processor_id());
pr_info("De-offloading %d\n", rdp->cpu);
rcu_nocb_lock_irqsave(rdp, flags);
/*
* If there are still pending work offloaded, the offline
* CPU won't help much handling them.
* Flush once and for all now. This suffices because we are
* running on the target CPU holding ->nocb_lock (thus having
* interrupts disabled), and because rdp_offload_toggle()
* invokes rcu_segcblist_offload(), which clears SEGCBLIST_OFFLOADED.
* Thus future calls to rcu_segcblist_completely_offloaded() will
* return false, which means that future calls to rcu_nocb_try_bypass()
* will refuse to put anything into the bypass.
*/
if (cpu_is_offline(rdp->cpu) && !rcu_segcblist_empty(&rdp->cblist)) {
rcu_nocb_unlock_irqrestore(rdp, flags);
return -EBUSY;
}
WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies));
ret = rdp_offload_toggle(rdp, false, flags);
swait_event_exclusive(rdp->nocb_state_wq,
!rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB |
@ -2360,32 +2450,24 @@ static int __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)
del_timer_sync(&rdp->nocb_timer);
/*
* Flush bypass. While IRQs are disabled and once we set
* SEGCBLIST_SOFTIRQ_ONLY, no callback is supposed to be
* enqueued on bypass.
* Theoretically we could set SEGCBLIST_SOFTIRQ_ONLY with CB unlocked
* and IRQs disabled but let's be paranoid.
*/
rcu_nocb_lock_irqsave(rdp, flags);
rcu_nocb_flush_bypass(rdp, NULL, jiffies);
rcu_segcblist_set_flags(cblist, SEGCBLIST_SOFTIRQ_ONLY);
/*
* With SEGCBLIST_SOFTIRQ_ONLY, we can't use
* rcu_nocb_unlock_irqrestore() anymore. Theoretically we
* could set SEGCBLIST_SOFTIRQ_ONLY with cb unlocked and IRQs
* disabled now, but let's be paranoid.
* rcu_nocb_unlock_irqrestore() anymore.
*/
raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
/* Sanity check */
WARN_ON_ONCE(rcu_cblist_n_cbs(&rdp->nocb_bypass));
return ret;
}
static long rcu_nocb_rdp_deoffload(void *arg)
{
struct rcu_data *rdp = arg;
WARN_ON_ONCE(rdp->cpu != raw_smp_processor_id());
return __rcu_nocb_rdp_deoffload(rdp);
}
int rcu_nocb_cpu_deoffload(int cpu)
{
struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);
@ -2397,13 +2479,15 @@ int rcu_nocb_cpu_deoffload(int cpu)
}
mutex_lock(&rcu_state.barrier_mutex);
cpus_read_lock();
if (rcu_segcblist_is_offloaded(&rdp->cblist)) {
if (cpu_online(cpu))
if (rcu_rdp_is_offloaded(rdp)) {
if (cpu_online(cpu)) {
ret = work_on_cpu(cpu, rcu_nocb_rdp_deoffload, rdp);
else
ret = __rcu_nocb_rdp_deoffload(rdp);
if (!ret)
cpumask_clear_cpu(cpu, rcu_nocb_mask);
if (!ret)
cpumask_clear_cpu(cpu, rcu_nocb_mask);
} else {
pr_info("NOCB: Can't CB-deoffload an offline CPU\n");
ret = -EINVAL;
}
}
cpus_read_unlock();
mutex_unlock(&rcu_state.barrier_mutex);
@ -2412,12 +2496,14 @@ int rcu_nocb_cpu_deoffload(int cpu)
}
EXPORT_SYMBOL_GPL(rcu_nocb_cpu_deoffload);
static int __rcu_nocb_rdp_offload(struct rcu_data *rdp)
static long rcu_nocb_rdp_offload(void *arg)
{
struct rcu_data *rdp = arg;
struct rcu_segcblist *cblist = &rdp->cblist;
unsigned long flags;
int ret;
WARN_ON_ONCE(rdp->cpu != raw_smp_processor_id());
/*
* For now we only support re-offload, ie: the rdp must have been
* offloaded on boot first.
@ -2457,14 +2543,6 @@ static int __rcu_nocb_rdp_offload(struct rcu_data *rdp)
return ret;
}
static long rcu_nocb_rdp_offload(void *arg)
{
struct rcu_data *rdp = arg;
WARN_ON_ONCE(rdp->cpu != raw_smp_processor_id());
return __rcu_nocb_rdp_offload(rdp);
}
int rcu_nocb_cpu_offload(int cpu)
{
struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);
@ -2472,13 +2550,15 @@ int rcu_nocb_cpu_offload(int cpu)
mutex_lock(&rcu_state.barrier_mutex);
cpus_read_lock();
if (!rcu_segcblist_is_offloaded(&rdp->cblist)) {
if (cpu_online(cpu))
if (!rcu_rdp_is_offloaded(rdp)) {
if (cpu_online(cpu)) {
ret = work_on_cpu(cpu, rcu_nocb_rdp_offload, rdp);
else
ret = __rcu_nocb_rdp_offload(rdp);
if (!ret)
cpumask_set_cpu(cpu, rcu_nocb_mask);
if (!ret)
cpumask_set_cpu(cpu, rcu_nocb_mask);
} else {
pr_info("NOCB: Can't CB-offload an offline CPU\n");
ret = -EINVAL;
}
}
cpus_read_unlock();
mutex_unlock(&rcu_state.barrier_mutex);

2
kernel/rcu/tree_stall.h
View File

@ -536,6 +536,7 @@ static void print_other_cpu_stall(unsigned long gp_seq, unsigned long gps)
* See Documentation/RCU/stallwarn.rst for info on how to debug
* RCU CPU stall warnings.
*/
trace_rcu_stall_warning(rcu_state.name, TPS("StallDetected"));
pr_err("INFO: %s detected stalls on CPUs/tasks:\n", rcu_state.name);
rcu_for_each_leaf_node(rnp) {
raw_spin_lock_irqsave_rcu_node(rnp, flags);
@ -606,6 +607,7 @@ static void print_cpu_stall(unsigned long gps)
* See Documentation/RCU/stallwarn.rst for info on how to debug
* RCU CPU stall warnings.
*/
trace_rcu_stall_warning(rcu_state.name, TPS("SelfDetected"));
pr_err("INFO: %s self-detected stall on CPU\n", rcu_state.name);
raw_spin_lock_irqsave_rcu_node(rdp->mynode, flags);
print_cpu_stall_info(smp_processor_id());

2
kernel/softirq.c
View File

@ -423,7 +423,7 @@ static inline void invoke_softirq(void)
if (ksoftirqd_running(local_softirq_pending()))
return;
if (!force_irqthreads) {
if (!force_irqthreads || !__this_cpu_read(ksoftirqd)) {
#ifdef CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK
/*
* We can safely execute softirq on the current stack if

6
kernel/torture.c
View File

@ -816,9 +816,9 @@ bool torture_init_begin(char *ttype, int v)
{
mutex_lock(&fullstop_mutex);
if (torture_type != NULL) {
pr_alert("torture_init_begin: Refusing %s init: %s running.\n",
ttype, torture_type);
pr_alert("torture_init_begin: One torture test at a time!\n");
pr_alert("%s: Refusing %s init: %s running.\n",
__func__, ttype, torture_type);
pr_alert("%s: One torture test at a time!\n", __func__);
mutex_unlock(&fullstop_mutex);
return false;
}

49
lib/bitmap.c
View File

@ -487,30 +487,25 @@ EXPORT_SYMBOL(bitmap_print_to_pagebuf);
/*
* Region 9-38:4/10 describes the following bitmap structure:
* 0 9 12 18 38
* .........****......****......****......
* ^ ^ ^ ^
* start off group_len end
* 0 9 12 18 38 N
* .........****......****......****..................
* ^ ^ ^ ^ ^
* start off group_len end nbits
*/
struct region {
unsigned int start;
unsigned int off;
unsigned int group_len;
unsigned int end;
unsigned int nbits;
};
static int bitmap_set_region(const struct region *r,
unsigned long *bitmap, int nbits)
static void bitmap_set_region(const struct region *r, unsigned long *bitmap)
{
unsigned int start;
if (r->end >= nbits)
return -ERANGE;
for (start = r->start; start <= r->end; start += r->group_len)
bitmap_set(bitmap, start, min(r->end - start + 1, r->off));
return 0;
}
static int bitmap_check_region(const struct region *r)
@ -518,14 +513,23 @@ static int bitmap_check_region(const struct region *r)
if (r->start > r->end || r->group_len == 0 || r->off > r->group_len)
return -EINVAL;
if (r->end >= r->nbits)
return -ERANGE;
return 0;
}
static const char *bitmap_getnum(const char *str, unsigned int *num)
static const char *bitmap_getnum(const char *str, unsigned int *num,
unsigned int lastbit)
{
unsigned long long n;
unsigned int len;
if (str[0] == 'N') {
*num = lastbit;
return str + 1;
}
len = _parse_integer(str, 10, &n);
if (!len)
return ERR_PTR(-EINVAL);
@ -573,7 +577,9 @@ static const char *bitmap_find_region_reverse(const char *start, const char *end
static const char *bitmap_parse_region(const char *str, struct region *r)
{
str = bitmap_getnum(str, &r->start);
unsigned int lastbit = r->nbits - 1;
str = bitmap_getnum(str, &r->start, lastbit);
if (IS_ERR(str))
return str;
@ -583,7 +589,7 @@ static const char *bitmap_parse_region(const char *str, struct region *r)
if (*str != '-')
return ERR_PTR(-EINVAL);
str = bitmap_getnum(str + 1, &r->end);
str = bitmap_getnum(str + 1, &r->end, lastbit);
if (IS_ERR(str))
return str;
@ -593,14 +599,14 @@ static const char *bitmap_parse_region(const char *str, struct region *r)
if (*str != ':')
return ERR_PTR(-EINVAL);
str = bitmap_getnum(str + 1, &r->off);
str = bitmap_getnum(str + 1, &r->off, lastbit);
if (IS_ERR(str))
return str;
if (*str != '/')
return ERR_PTR(-EINVAL);
return bitmap_getnum(str + 1, &r->group_len);
return bitmap_getnum(str + 1, &r->group_len, lastbit);
no_end:
r->end = r->start;
@ -627,6 +633,10 @@ no_pattern:
* From each group will be used only defined amount of bits.
* Syntax: range:used_size/group_size
* Example: 0-1023:2/256 ==> 0,1,256,257,512,513,768,769
* The value 'N' can be used as a dynamically substituted token for the
* maximum allowed value; i.e (nmaskbits - 1). Keep in mind that it is
* dynamic, so if system changes cause the bitmap width to change, such
* as more cores in a CPU list, then any ranges using N will also change.
*
* Returns: 0 on success, -errno on invalid input strings. Error values:
*
@ -640,7 +650,8 @@ int bitmap_parselist(const char *buf, unsigned long *maskp, int nmaskbits)
struct region r;
long ret;
bitmap_zero(maskp, nmaskbits);
r.nbits = nmaskbits;
bitmap_zero(maskp, r.nbits);
while (buf) {
buf = bitmap_find_region(buf);
@ -655,9 +666,7 @@ int bitmap_parselist(const char *buf, unsigned long *maskp, int nmaskbits)
if (ret)
return ret;
ret = bitmap_set_region(&r, maskp, nmaskbits);
if (ret)
return ret;
bitmap_set_region(&r, maskp);
}
return 0;

46
lib/test_bitmap.c
View File

@ -34,6 +34,8 @@ static const unsigned long exp1[] __initconst = {
BITMAP_FROM_U64(0x3333333311111111ULL),
BITMAP_FROM_U64(0xffffffff77777777ULL),
BITMAP_FROM_U64(0),
BITMAP_FROM_U64(0x00008000),
BITMAP_FROM_U64(0x80000000),
};
static const unsigned long exp2[] __initconst = {
@ -334,15 +336,47 @@ static const struct test_bitmap_parselist parselist_tests[] __initconst = {
{0, " , ,, , , ", &exp1[12 * step], 8, 0},
{0, " , ,, , , \n", &exp1[12 * step], 8, 0},
{0, "0-0", &exp1[0], 32, 0},
{0, "1-1", &exp1[1 * step], 32, 0},
{0, "15-15", &exp1[13 * step], 32, 0},
{0, "31-31", &exp1[14 * step], 32, 0},
{0, "0-0:0/1", &exp1[12 * step], 32, 0},
{0, "0-0:1/1", &exp1[0], 32, 0},
{0, "0-0:1/31", &exp1[0], 32, 0},
{0, "0-0:31/31", &exp1[0], 32, 0},
{0, "1-1:1/1", &exp1[1 * step], 32, 0},
{0, "0-15:16/31", &exp1[2 * step], 32, 0},
{0, "15-15:1/2", &exp1[13 * step], 32, 0},
{0, "15-15:31/31", &exp1[13 * step], 32, 0},
{0, "15-31:1/31", &exp1[13 * step], 32, 0},
{0, "16-31:16/31", &exp1[3 * step], 32, 0},
{0, "31-31:31/31", &exp1[14 * step], 32, 0},
{0, "N-N", &exp1[14 * step], 32, 0},
{0, "0-0:1/N", &exp1[0], 32, 0},
{0, "0-0:N/N", &exp1[0], 32, 0},
{0, "0-15:16/N", &exp1[2 * step], 32, 0},
{0, "15-15:N/N", &exp1[13 * step], 32, 0},
{0, "15-N:1/N", &exp1[13 * step], 32, 0},
{0, "16-N:16/N", &exp1[3 * step], 32, 0},
{0, "N-N:N/N", &exp1[14 * step], 32, 0},
{0, "0-N:1/3,1-N:1/3,2-N:1/3", &exp1[8 * step], 32, 0},
{0, "0-31:1/3,1-31:1/3,2-31:1/3", &exp1[8 * step], 32, 0},
{0, "1-10:8/12,8-31:24/29,0-31:0/3", &exp1[9 * step], 32, 0},
{-EINVAL, "-1", NULL, 8, 0},
{-EINVAL, "-0", NULL, 8, 0},
{-EINVAL, "10-1", NULL, 8, 0},
{-EINVAL, "0-31:", NULL, 8, 0},
{-EINVAL, "0-31:0", NULL, 8, 0},
{-EINVAL, "0-31:0/", NULL, 8, 0},
{-EINVAL, "0-31:0/0", NULL, 8, 0},
{-EINVAL, "0-31:1/0", NULL, 8, 0},
{-EINVAL, "0-31:10/1", NULL, 8, 0},
{-ERANGE, "8-8", NULL, 8, 0},
{-ERANGE, "0-31", NULL, 8, 0},
{-EINVAL, "0-31:", NULL, 32, 0},
{-EINVAL, "0-31:0", NULL, 32, 0},
{-EINVAL, "0-31:0/", NULL, 32, 0},
{-EINVAL, "0-31:0/0", NULL, 32, 0},
{-EINVAL, "0-31:1/0", NULL, 32, 0},
{-EINVAL, "0-31:10/1", NULL, 32, 0},
{-EOVERFLOW, "0-98765432123456789:10/1", NULL, 8, 0},
{-EINVAL, "a-31", NULL, 8, 0},

2
mm/slab.c
View File

@ -3651,6 +3651,7 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t flags,
EXPORT_SYMBOL(__kmalloc_node_track_caller);
#endif /* CONFIG_NUMA */
#ifdef CONFIG_PRINTK
void kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct page *page)
{
struct kmem_cache *cachep;
@ -3670,6 +3671,7 @@ void kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct page *page)
if (DEBUG && cachep->flags & SLAB_STORE_USER)
kpp->kp_ret = *dbg_userword(cachep, objp);
}
#endif
/**
* __do_kmalloc - allocate memory

2
mm/slab.h
View File

@ -621,6 +621,7 @@ static inline bool slab_want_init_on_free(struct kmem_cache *c)
return false;
}
#ifdef CONFIG_PRINTK
#define KS_ADDRS_COUNT 16
struct kmem_obj_info {
void *kp_ptr;
@ -632,5 +633,6 @@ struct kmem_obj_info {
void *kp_stack[KS_ADDRS_COUNT];
};
void kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct page *page);
#endif
#endif /* MM_SLAB_H */

4
mm/slab_common.c
View File

@ -526,6 +526,7 @@ bool slab_is_available(void)
return slab_state >= UP;
}
#ifdef CONFIG_PRINTK
/**
* kmem_valid_obj - does the pointer reference a valid slab object?
* @object: pointer to query.
@ -544,6 +545,7 @@ bool kmem_valid_obj(void *object)
page = virt_to_head_page(object);
return PageSlab(page);
}
EXPORT_SYMBOL_GPL(kmem_valid_obj);
/**
* kmem_dump_obj - Print available slab provenance information
@ -600,6 +602,8 @@ void kmem_dump_obj(void *object)
pr_info(" %pS\n", kp.kp_stack[i]);
}
}
EXPORT_SYMBOL_GPL(kmem_dump_obj);
#endif
#ifndef CONFIG_SLOB
/* Create a cache during boot when no slab services are available yet */

2
mm/slob.c
View File

@ -461,11 +461,13 @@ out:
spin_unlock_irqrestore(&slob_lock, flags);
}
#ifdef CONFIG_PRINTK
void kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct page *page)
{
kpp->kp_ptr = object;
kpp->kp_page = page;
}
#endif
/*
* End of slob allocator proper. Begin kmem_cache_alloc and kmalloc frontend.

2
mm/slub.c
View File

@ -3964,6 +3964,7 @@ int __kmem_cache_shutdown(struct kmem_cache *s)
return 0;
}
#ifdef CONFIG_PRINTK
void kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct page *page)
{
void *base;
@ -4003,6 +4004,7 @@ void kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct page *page)
#endif
#endif
}
#endif
/********************************************************************
* Kmalloc subsystem

3
mm/util.c
View File

@ -983,6 +983,7 @@ int __weak memcmp_pages(struct page *page1, struct page *page2)
return ret;
}
#ifdef CONFIG_PRINTK
/**
* mem_dump_obj - Print available provenance information
* @object: object for which to find provenance information.
@ -1013,3 +1014,5 @@ void mem_dump_obj(void *object)
}
pr_cont(" non-slab/vmalloc memory.\n");
}
EXPORT_SYMBOL_GPL(mem_dump_obj);
#endif

2
mm/vmalloc.c
View File

@ -3450,6 +3450,7 @@ void pcpu_free_vm_areas(struct vm_struct **vms, int nr_vms)
}
#endif /* CONFIG_SMP */
#ifdef CONFIG_PRINTK
bool vmalloc_dump_obj(void *object)
{
struct vm_struct *vm;
@ -3462,6 +3463,7 @@ bool vmalloc_dump_obj(void *object)
vm->nr_pages, (unsigned long)vm->addr, vm->caller);
return true;
}
#endif
#ifdef CONFIG_PROC_FS
static void *s_start(struct seq_file *m, loff_t *pos)

1
tools/testing/selftests/rcutorture/bin/cpus2use.sh
View File

@ -21,7 +21,6 @@ then
awk -v ncpus=$ncpus '{ print ncpus * ($7 + $NF) / 100 }'`
else
# No mpstat command, so use all available CPUs.
echo The mpstat command is not available, so greedily using all CPUs.
idlecpus=$ncpus
fi
awk -v ncpus=$ncpus -v idlecpus=$idlecpus < /dev/null '

20
tools/testing/selftests/rcutorture/bin/jitter.sh
View File

@ -5,10 +5,11 @@
# of this script is to inflict random OS jitter on a concurrently running
# test.
#
# Usage: jitter.sh me duration [ sleepmax [ spinmax ] ]
# Usage: jitter.sh me jittering-path duration [ sleepmax [ spinmax ] ]
#
# me: Random-number-generator seed salt.
# duration: Time to run in seconds.
# jittering-path: Path to file whose removal will stop this script.
# sleepmax: Maximum microseconds to sleep, defaults to one second.
# spinmax: Maximum microseconds to spin, defaults to one millisecond.
#
@ -17,9 +18,10 @@
# Authors: Paul E. McKenney <paulmck@linux.ibm.com>
me=$(($1 * 1000))
duration=$2
sleepmax=${3-1000000}
spinmax=${4-1000}
jittering=$2
duration=$3
sleepmax=${4-1000000}
spinmax=${5-1000}
n=1
@ -47,7 +49,7 @@ do
fi
# Check for stop request.
if test -f "$TORTURE_STOPFILE"
if ! test -f "$jittering"
then
exit 1;
fi
@ -67,10 +69,10 @@ do
srand(n + me + systime());
ncpus = split(cpus, ca);
curcpu = ca[int(rand() * ncpus + 1)];
mask = lshift(1, curcpu);
if (mask + 0 <= 0)
mask = 1;
printf("%#x\n", mask);
z = "";
for (i = 1; 4 * i <= curcpu; i++)
z = z "0";
print "0x" 2 ^ (curcpu % 4) z;
}' < /dev/null`
n=$(($n+1))
if ! taskset -p $cpumask $$ > /dev/null 2>&1

37
tools/testing/selftests/rcutorture/bin/jitterstart.sh Normal file
View File

@ -0,0 +1,37 @@
#!/bin/bash
# SPDX-License-Identifier: GPL-2.0+
#
# Start up the specified number of jitter.sh scripts in the background.
#
# Usage: . jitterstart.sh n jittering-dir duration [ sleepmax [ spinmax ] ]
#
# n: Number of jitter.sh scripts to start up.
# jittering-dir: Directory in which to put "jittering" file.
# duration: Time to run in seconds.
# sleepmax: Maximum microseconds to sleep, defaults to one second.
# spinmax: Maximum microseconds to spin, defaults to one millisecond.
#
# Copyright (C) 2021 Facebook, Inc.
#
# Authors: Paul E. McKenney <paulmck@kernel.org>
jitter_n=$1
if test -z "$jitter_n"
then
echo jitterstart.sh: Missing count of jitter.sh scripts to start.
exit 33
fi
jittering_dir=$2
if test -z "$jittering_dir"
then
echo jitterstart.sh: Missing directory in which to place jittering file.
exit 34
fi
shift
shift
touch ${jittering_dir}/jittering
for ((jitter_i = 1; jitter_i <= $jitter_n; jitter_i++))
do
jitter.sh $jitter_i "${jittering_dir}/jittering" "$@" &
done

23
tools/testing/selftests/rcutorture/bin/jitterstop.sh Normal file
View File

@ -0,0 +1,23 @@
#!/bin/bash
# SPDX-License-Identifier: GPL-2.0+
#
# Remove the "jittering" file, signaling the jitter.sh scripts to stop,
# then wait for them to terminate.
#
# Usage: . jitterstop.sh jittering-dir
#
# jittering-dir: Directory containing "jittering" file.
#
# Copyright (C) 2021 Facebook, Inc.
#
# Authors: Paul E. McKenney <paulmck@kernel.org>
jittering_dir=$1
if test -z "$jittering_dir"
then
echo jitterstop.sh: Missing directory in which to place jittering file.
exit 34
fi
rm -f ${jittering_dir}/jittering
wait

199
tools/testing/selftests/rcutorture/bin/kvm-again.sh Executable file
View File

@ -0,0 +1,199 @@
#!/bin/bash
# SPDX-License-Identifier: GPL-2.0+
#
# Rerun a series of tests under KVM.
#
# Usage: kvm-again.sh /path/to/old/run [ options ]
#
# Copyright (C) 2021 Facebook, Inc.
#
# Authors: Paul E. McKenney <paulmck@kernel.org>
scriptname=$0
args="$*"
T=${TMPDIR-/tmp}/kvm-again.sh.$$
trap 'rm -rf $T' 0
mkdir $T
if ! test -d tools/testing/selftests/rcutorture/bin
then
echo $scriptname must be run from top-level directory of kernel source tree.
exit 1
fi
oldrun=$1
shift
if ! test -d "$oldrun"
then
echo "Usage: $scriptname /path/to/old/run [ options ]"
exit 1
fi
if ! cp "$oldrun/batches" $T/batches.oldrun
then
# Later on, can reconstitute this from console.log files.
echo Prior run batches file does not exist: $oldrun/batches
exit 1
fi
if test -f "$oldrun/torture_suite"
then
torture_suite="`cat $oldrun/torture_suite`"
elif test -f "$oldrun/TORTURE_SUITE"
then
torture_suite="`cat $oldrun/TORTURE_SUITE`"
else
echo "Prior run torture_suite file does not exist: $oldrun/{torture_suite,TORTURE_SUITE}"
exit 1
fi
KVM="`pwd`/tools/testing/selftests/rcutorture"; export KVM
PATH=${KVM}/bin:$PATH; export PATH
. functions.sh
dryrun=
dur=
default_link="cp -R"
rundir="`pwd`/tools/testing/selftests/rcutorture/res/`date +%Y.%m.%d-%H.%M.%S-again`"
startdate="`date`"
starttime="`get_starttime`"
usage () {
echo "Usage: $scriptname $oldrun [ arguments ]:"
echo " --dryrun"
echo " --duration minutes | <seconds>s | <hours>h | <days>d"
echo " --link hard|soft|copy"
echo " --remote"
echo " --rundir /new/res/path"
exit 1
}
while test $# -gt 0
do
case "$1" in
--dryrun)
dryrun=1
;;
--duration)
checkarg --duration "(minutes)" $# "$2" '^[0-9][0-9]*\(s\|m\|h\|d\|\)$' '^error'
mult=60
if echo "$2" | grep -q 's$'
then
mult=1
elif echo "$2" | grep -q 'h$'
then
mult=3600
elif echo "$2" | grep -q 'd$'
then
mult=86400
fi
ts=`echo $2 | sed -e 's/[smhd]$//'`
dur=$(($ts*mult))
shift
;;
--link)
checkarg --link "hard|soft|copy" "$#" "$2" 'hard\|soft\|copy' '^--'
case "$2" in
copy)
arg_link="cp -R"
;;
hard)
arg_link="cp -Rl"
;;
soft)
arg_link="cp -Rs"
;;
esac
shift
;;
--remote)
arg_remote=1
default_link="cp -as"
;;
--rundir)
checkarg --rundir "(absolute pathname)" "$#" "$2" '^/' '^error'
rundir=$2
if test -e "$rundir"
then
echo "--rundir $2: Already exists."
usage
fi
shift
;;
*)
echo Unknown argument $1
usage
;;
esac
shift
done
if test -z "$arg_link"
then
arg_link="$default_link"
fi
echo ---- Re-run results directory: $rundir
# Copy old run directory tree over and adjust.
mkdir -p "`dirname "$rundir"`"
if ! $arg_link "$oldrun" "$rundir"
then
echo "Cannot copy from $oldrun to $rundir."
usage
fi
rm -f "$rundir"/*/{console.log,console.log.diags,qemu_pid,qemu-retval,Warnings,kvm-test-1-run.sh.out,kvm-test-1-run-qemu.sh.out,vmlinux} "$rundir"/log
echo $oldrun > "$rundir/re-run"
if ! test -d "$rundir/../../bin"
then
$arg_link "$oldrun/../../bin" "$rundir/../.."
fi
for i in $rundir/*/qemu-cmd
do
cp "$i" $T
qemu_cmd_dir="`dirname "$i"`"
kernel_dir="`echo $qemu_cmd_dir | sed -e 's/\.[0-9]\+$//'`"
jitter_dir="`dirname "$kernel_dir"`"
kvm-transform.sh "$kernel_dir/bzImage" "$qemu_cmd_dir/console.log" "$jitter_dir" $dur < $T/qemu-cmd > $i
if test -n "$arg_remote"
then
echo "# TORTURE_KCONFIG_GDB_ARG=''" >> $i
fi
done
# Extract settings from the last qemu-cmd file transformed above.
grep '^#' $i | sed -e 's/^# //' > $T/qemu-cmd-settings
. $T/qemu-cmd-settings
grep -v '^#' $T/batches.oldrun | awk '
BEGIN {
oldbatch = 1;
}
{
if (oldbatch != $1) {
print "kvm-test-1-run-batch.sh" curbatch;
curbatch = "";
oldbatch = $1;
}
curbatch = curbatch " " $2;
}
END {
print "kvm-test-1-run-batch.sh" curbatch
}' > $T/runbatches.sh
if test -n "$dryrun"
then
echo ---- Dryrun complete, directory: $rundir | tee -a "$rundir/log"
else
( cd "$rundir"; sh $T/runbatches.sh )
kcsan-collapse.sh "$rundir" | tee -a "$rundir/log"
echo | tee -a "$rundir/log"
echo ---- Results directory: $rundir | tee -a "$rundir/log"
kvm-recheck.sh "$rundir" > $T/kvm-recheck.sh.out 2>&1
ret=$?
cat $T/kvm-recheck.sh.out | tee -a "$rundir/log"
echo " --- Done at `date` (`get_starttime_duration $starttime`) exitcode $ret" | tee -a "$rundir/log"
exit $ret
fi

2
tools/testing/selftests/rcutorture/bin/kvm-recheck.sh
View File

@ -30,7 +30,7 @@ do
resdir=`echo $i | sed -e 's,/$,,' -e 's,/[^/]*$,,'`
head -1 $resdir/log
fi
TORTURE_SUITE="`cat $i/../TORTURE_SUITE`"
TORTURE_SUITE="`cat $i/../torture_suite`"
configfile=`echo $i | sed -e 's,^.*/,,'`
rm -f $i/console.log.*.diags
kvm-recheck-${TORTURE_SUITE}.sh $i

67
tools/testing/selftests/rcutorture/bin/kvm-test-1-run-batch.sh Executable file
View File

@ -0,0 +1,67 @@
#!/bin/bash
# SPDX-License-Identifier: GPL-2.0+
#
# Carry out a kvm-based run for the specified batch of scenarios, which
# might have been built by --build-only kvm.sh run.
#
# Usage: kvm-test-1-run-batch.sh SCENARIO [ SCENARIO ... ]
#
# Each SCENARIO is the name of a directory in the current directory
# containing a ready-to-run qemu-cmd file.
#
# Copyright (C) 2021 Facebook, Inc.
#
# Authors: Paul E. McKenney <paulmck@kernel.org>
T=${TMPDIR-/tmp}/kvm-test-1-run-batch.sh.$$
trap 'rm -rf $T' 0
mkdir $T
echo ---- Running batch $*
# Check arguments
runfiles=
for i in "$@"
do
if ! echo $i | grep -q '^[^/.a-z]\+\(\.[0-9]\+\)\?$'
then
echo Bad scenario name: \"$i\" 1>&2
exit 1
fi
if ! test -d "$i"
then
echo Scenario name not a directory: \"$i\" 1>&2
exit 2
fi
if ! test -f "$i/qemu-cmd"
then
echo Scenario lacks a command file: \"$i/qemu-cmd\" 1>&2
exit 3
fi
rm -f $i/build.*
touch $i/build.run
runfiles="$runfiles $i/build.run"
done
# Extract settings from the qemu-cmd file.
grep '^#' $1/qemu-cmd | sed -e 's/^# //' > $T/qemu-cmd-settings
. $T/qemu-cmd-settings
# Start up jitter, start each scenario, wait, end jitter.
echo ---- System running test: `uname -a`
echo ---- Starting kernels. `date` | tee -a log
$TORTURE_JITTER_START
for i in "$@"
do
echo ---- System running test: `uname -a` > $i/kvm-test-1-run-qemu.sh.out
echo > $i/kvm-test-1-run-qemu.sh.out
kvm-test-1-run-qemu.sh $i >> $i/kvm-test-1-run-qemu.sh.out 2>&1 &
done
for i in $runfiles
do
while ls $i > /dev/null 2>&1
do
:
done
done
echo ---- All kernel runs complete. `date` | tee -a log
$TORTURE_JITTER_STOP

176
tools/testing/selftests/rcutorture/bin/kvm-test-1-run-qemu.sh Executable file
View File

@ -0,0 +1,176 @@
#!/bin/bash
# SPDX-License-Identifier: GPL-2.0+
#
# Carry out a kvm-based run for the specified qemu-cmd file, which might
# have been generated by --build-only kvm.sh run.
#
# Usage: kvm-test-1-run-qemu.sh qemu-cmd-dir
#
# qemu-cmd-dir provides the directory containing qemu-cmd file.
# This is assumed to be of the form prefix/ds/scenario, where
# "ds" is the top-level date-stamped directory and "scenario"
# is the scenario name. Any required adjustments to this file
# must have been made by the caller. The shell-command comments
# at the end of the qemu-cmd file are not optional.
#
# Copyright (C) 2021 Facebook, Inc.
#
# Authors: Paul E. McKenney <paulmck@kernel.org>
T=${TMPDIR-/tmp}/kvm-test-1-run-qemu.sh.$$
trap 'rm -rf $T' 0
mkdir $T
resdir="$1"
if ! test -d "$resdir"
then
echo $0: Nonexistent directory: $resdir
exit 1
fi
if ! test -f "$resdir/qemu-cmd"
then
echo $0: Nonexistent qemu-cmd file: $resdir/qemu-cmd
exit 1
fi
echo ' ---' `date`: Starting kernel, PID $$
# Obtain settings from the qemu-cmd file.
grep '^#' $resdir/qemu-cmd | sed -e 's/^# //' > $T/qemu-cmd-settings
. $T/qemu-cmd-settings
# Decorate qemu-cmd with redirection, backgrounding, and PID capture
sed -e 's/$/ 2>\&1 \&/' < $resdir/qemu-cmd > $T/qemu-cmd
echo 'echo $! > $resdir/qemu_pid' >> $T/qemu-cmd
# In case qemu refuses to run...
echo "NOTE: $QEMU either did not run or was interactive" > $resdir/console.log
# Attempt to run qemu
kstarttime=`gawk 'BEGIN { print systime() }' < /dev/null`
( . $T/qemu-cmd; wait `cat $resdir/qemu_pid`; echo $? > $resdir/qemu-retval ) &
commandcompleted=0
if test -z "$TORTURE_KCONFIG_GDB_ARG"
then
sleep 10 # Give qemu's pid a chance to reach the file
if test -s "$resdir/qemu_pid"
then
qemu_pid=`cat "$resdir/qemu_pid"`
echo Monitoring qemu job at pid $qemu_pid
else
qemu_pid=""
echo Monitoring qemu job at yet-as-unknown pid
fi
fi
if test -n "$TORTURE_KCONFIG_GDB_ARG"
then
base_resdir=`echo $resdir | sed -e 's/\.[0-9]\+$//'`
if ! test -f $base_resdir/vmlinux
then
base_resdir="`cat re-run`/$resdir"
if ! test -f $base_resdir/vmlinux
then
base_resdir=/path/to
fi
fi
echo Waiting for you to attach a debug session, for example: > /dev/tty
echo " gdb $base_resdir/vmlinux" > /dev/tty
echo 'After symbols load and the "(gdb)" prompt appears:' > /dev/tty
echo " target remote :1234" > /dev/tty
echo " continue" > /dev/tty
kstarttime=`gawk 'BEGIN { print systime() }' < /dev/null`
fi
while :
do
if test -z "$qemu_pid" -a -s "$resdir/qemu_pid"
then
qemu_pid=`cat "$resdir/qemu_pid"`
fi
kruntime=`gawk 'BEGIN { print systime() - '"$kstarttime"' }' < /dev/null`
if test -z "$qemu_pid" || kill -0 "$qemu_pid" > /dev/null 2>&1
then
if test -n "$TORTURE_KCONFIG_GDB_ARG"
then
:
elif test $kruntime -ge $seconds || test -f "$resdir/../STOP.1"
then
break;
fi
sleep 1
else
commandcompleted=1
if test $kruntime -lt $seconds
then
echo Completed in $kruntime vs. $seconds >> $resdir/Warnings 2>&1
grep "^(qemu) qemu:" $resdir/kvm-test-1-run.sh.out >> $resdir/Warnings 2>&1
killpid="`sed -n "s/^(qemu) qemu: terminating on signal [0-9]* from pid \([0-9]*\).*$/\1/p" $resdir/Warnings`"
if test -n "$killpid"
then
echo "ps -fp $killpid" >> $resdir/Warnings 2>&1
ps -fp $killpid >> $resdir/Warnings 2>&1
fi
else
echo ' ---' `date`: "Kernel done"
fi
break
fi
done
if test -z "$qemu_pid" -a -s "$resdir/qemu_pid"
then
qemu_pid=`cat "$resdir/qemu_pid"`
fi
if test $commandcompleted -eq 0 -a -n "$qemu_pid"
then
if ! test -f "$resdir/../STOP.1"
then
echo Grace period for qemu job at pid $qemu_pid
fi
oldline="`tail $resdir/console.log`"
while :
do
if test -f "$resdir/../STOP.1"
then
echo "PID $qemu_pid killed due to run STOP.1 request" >> $resdir/Warnings 2>&1
kill -KILL $qemu_pid
break
fi
kruntime=`gawk 'BEGIN { print systime() - '"$kstarttime"' }' < /dev/null`
if kill -0 $qemu_pid > /dev/null 2>&1
then
:
else
break
fi
must_continue=no
newline="`tail $resdir/console.log`"
if test "$newline" != "$oldline" && echo $newline | grep -q ' [0-9]\+us : '
then
must_continue=yes
fi
last_ts="`tail $resdir/console.log | grep '^\[ *[0-9]\+\.[0-9]\+]' | tail -1 | sed -e 's/^\[ *//' -e 's/\..*$//'`"
if test -z "$last_ts"
then
last_ts=0
fi
if test "$newline" != "$oldline" -a "$last_ts" -lt $((seconds + $TORTURE_SHUTDOWN_GRACE))
then
must_continue=yes
fi
if test $must_continue = no -a $kruntime -ge $((seconds + $TORTURE_SHUTDOWN_GRACE))
then
echo "!!! PID $qemu_pid hung at $kruntime vs. $seconds seconds" >> $resdir/Warnings 2>&1
kill -KILL $qemu_pid
break
fi
oldline=$newline
sleep 10
done
elif test -z "$qemu_pid"
then
echo Unknown PID, cannot kill qemu command
fi
# Tell the script that this run is done.
rm -f $resdir/build.run
parse-console.sh $resdir/console.log $title

218
tools/testing/selftests/rcutorture/bin/kvm-test-1-run.sh
View File

@ -7,15 +7,15 @@
# Execute this in the source tree. Do not run it as a background task
# because qemu does not seem to like that much.
#
# Usage: kvm-test-1-run.sh config builddir resdir seconds qemu-args boot_args
# Usage: kvm-test-1-run.sh config resdir seconds qemu-args boot_args_in
#
# qemu-args defaults to "-enable-kvm -nographic", along with arguments
# specifying the number of CPUs and other options
# generated from the underlying CPU architecture.
# boot_args defaults to value returned by the per_version_boot_params
# boot_args_in defaults to value returned by the per_version_boot_params
# shell function.
#
# Anything you specify for either qemu-args or boot_args is appended to
# Anything you specify for either qemu-args or boot_args_in is appended to
# the default values. The "-smp" value is deduced from the contents of
# the config fragment.
#
@ -35,14 +35,13 @@ mkdir $T
config_template=${1}
config_dir=`echo $config_template | sed -e 's,/[^/]*$,,'`
title=`echo $config_template | sed -e 's/^.*\///'`
builddir=${2}
resdir=${3}
resdir=${2}
if test -z "$resdir" -o ! -d "$resdir" -o ! -w "$resdir"
then
echo "kvm-test-1-run.sh :$resdir: Not a writable directory, cannot store results into it"
exit 1
fi
echo ' ---' `date`: Starting build
echo ' ---' `date`: Starting build, PID $$
echo ' ---' Kconfig fragment at: $config_template >> $resdir/log
touch $resdir/ConfigFragment.input
@ -73,7 +72,7 @@ config_override_param "--kconfig argument" KcList "$TORTURE_KCONFIG_ARG"
cp $T/KcList $resdir/ConfigFragment
base_resdir=`echo $resdir | sed -e 's/\.[0-9]\+$//'`
if test "$base_resdir" != "$resdir" -a -f $base_resdir/bzImage -a -f $base_resdir/vmlinux
if test "$base_resdir" != "$resdir" && test -f $base_resdir/bzImage && test -f $base_resdir/vmlinux
then
# Rerunning previous test, so use that test's kernel.
QEMU="`identify_qemu $base_resdir/vmlinux`"
@ -83,6 +82,17 @@ then
ln -s $base_resdir/.config $resdir # for kvm-recheck.sh
# Arch-independent indicator
touch $resdir/builtkernel
elif test "$base_resdir" != "$resdir"
then
# Rerunning previous test for which build failed
ln -s $base_resdir/Make*.out $resdir # for kvm-recheck.sh
ln -s $base_resdir/.config $resdir # for kvm-recheck.sh
echo Initial build failed, not running KVM, see $resdir.
if test -f $resdir/build.wait
then
mv $resdir/build.wait $resdir/build.ready
fi
exit 1
elif kvm-build.sh $T/KcList $resdir
then
# Had to build a kernel for this test.
@ -107,23 +117,23 @@ else
# Build failed.
cp .config $resdir || :
echo Build failed, not running KVM, see $resdir.
if test -f $builddir.wait
if test -f $resdir/build.wait
then
mv $builddir.wait $builddir.ready
mv $resdir/build.wait $resdir/build.ready
fi
exit 1
fi
if test -f $builddir.wait
if test -f $resdir/build.wait
then
mv $builddir.wait $builddir.ready
mv $resdir/build.wait $resdir/build.ready
fi
while test -f $builddir.ready
while test -f $resdir/build.ready
do
sleep 1
done
seconds=$4
qemu_args=$5
boot_args=$6
seconds=$3
qemu_args=$4
boot_args_in=$5
if test -z "$TORTURE_BUILDONLY"
then
@ -133,7 +143,7 @@ fi
# Generate -smp qemu argument.
qemu_args="-enable-kvm -nographic $qemu_args"
cpu_count=`configNR_CPUS.sh $resdir/ConfigFragment`
cpu_count=`configfrag_boot_cpus "$boot_args" "$config_template" "$cpu_count"`
cpu_count=`configfrag_boot_cpus "$boot_args_in" "$config_template" "$cpu_count"`
if test "$cpu_count" -gt "$TORTURE_ALLOTED_CPUS"
then
echo CPU count limited from $cpu_count to $TORTURE_ALLOTED_CPUS | tee -a $resdir/Warnings
@ -149,16 +159,52 @@ qemu_args="$qemu_args `identify_qemu_args "$QEMU" "$resdir/console.log"`"
qemu_append="`identify_qemu_append "$QEMU"`"
# Pull in Kconfig-fragment boot parameters
boot_args="`configfrag_boot_params "$boot_args" "$config_template"`"
boot_args="`configfrag_boot_params "$boot_args_in" "$config_template"`"
# Generate kernel-version-specific boot parameters
boot_args="`per_version_boot_params "$boot_args" $resdir/.config $seconds`"
if test -n "$TORTURE_BOOT_GDB_ARG"
then
boot_args="$boot_args $TORTURE_BOOT_GDB_ARG"
fi
# Give bare-metal advice
modprobe_args="`echo $boot_args | tr -s ' ' '\012' | grep "^$TORTURE_MOD\." | sed -e "s/$TORTURE_MOD\.//g"`"
kboot_args="`echo $boot_args | tr -s ' ' '\012' | grep -v "^$TORTURE_MOD\."`"
testid_txt="`dirname $resdir`/testid.txt"
touch $resdir/bare-metal
echo To run this scenario on bare metal: >> $resdir/bare-metal
echo >> $resdir/bare-metal
echo " 1." Set your bare-metal build tree to the state shown in this file: >> $resdir/bare-metal
echo " " $testid_txt >> $resdir/bare-metal
echo " 2." Update your bare-metal build tree"'"s .config based on this file: >> $resdir/bare-metal
echo " " $resdir/ConfigFragment >> $resdir/bare-metal
echo " 3." Make the bare-metal kernel"'"s build system aware of your .config updates: >> $resdir/bare-metal
echo " " $ 'yes "" | make oldconfig' >> $resdir/bare-metal
echo " 4." Build your bare-metal kernel. >> $resdir/bare-metal
echo " 5." Boot your bare-metal kernel with the following parameters: >> $resdir/bare-metal
echo " " $kboot_args >> $resdir/bare-metal
echo " 6." Start the test with the following command: >> $resdir/bare-metal
echo " " $ modprobe $TORTURE_MOD $modprobe_args >> $resdir/bare-metal
echo " 7." After some time, end the test with the following command: >> $resdir/bare-metal
echo " " $ rmmod $TORTURE_MOD >> $resdir/bare-metal
echo " 8." Copy your bare-metal kernel"'"s .config file, overwriting this file: >> $resdir/bare-metal
echo " " $resdir/.config >> $resdir/bare-metal
echo " 9." Copy the console output from just before the modprobe to just after >> $resdir/bare-metal
echo " " the rmmod into this file: >> $resdir/bare-metal
echo " " $resdir/console.log >> $resdir/bare-metal
echo "10." Check for runtime errors using the following command: >> $resdir/bare-metal
echo " " $ tools/testing/selftests/rcutorture/bin/kvm-recheck.sh `dirname $resdir` >> $resdir/bare-metal
echo >> $resdir/bare-metal
echo Some of the above steps may be skipped if you build your bare-metal >> $resdir/bare-metal
echo kernel here: `head -n 1 $testid_txt | sed -e 's/^Build directory: //'` >> $resdir/bare-metal
echo $QEMU $qemu_args -m $TORTURE_QEMU_MEM -kernel $KERNEL -append \"$qemu_append $boot_args\" $TORTURE_QEMU_GDB_ARG > $resdir/qemu-cmd
echo "# TORTURE_SHUTDOWN_GRACE=$TORTURE_SHUTDOWN_GRACE" >> $resdir/qemu-cmd
echo "# seconds=$seconds" >> $resdir/qemu-cmd
echo "# TORTURE_KCONFIG_GDB_ARG=\"$TORTURE_KCONFIG_GDB_ARG\"" >> $resdir/qemu-cmd
echo "# TORTURE_JITTER_START=\"$TORTURE_JITTER_START\"" >> $resdir/qemu-cmd
echo "# TORTURE_JITTER_STOP=\"$TORTURE_JITTER_STOP\"" >> $resdir/qemu-cmd
echo "# TORTURE_TRUST_MAKE=\"$TORTURE_TRUST_MAKE\"; export TORTURE_TRUST_MAKE" >> $resdir/qemu-cmd
if test -n "$TORTURE_BUILDONLY"
then
@ -167,140 +213,4 @@ then
exit 0
fi
# Decorate qemu-cmd with redirection, backgrounding, and PID capture
sed -e 's/$/ 2>\&1 \&/' < $resdir/qemu-cmd > $T/qemu-cmd
echo 'echo $! > $resdir/qemu_pid' >> $T/qemu-cmd
# In case qemu refuses to run...
echo "NOTE: $QEMU either did not run or was interactive" > $resdir/console.log
# Attempt to run qemu
kstarttime=`gawk 'BEGIN { print systime() }' < /dev/null`
( . $T/qemu-cmd; wait `cat $resdir/qemu_pid`; echo $? > $resdir/qemu-retval ) &
commandcompleted=0
if test -z "$TORTURE_KCONFIG_GDB_ARG"
then
sleep 10 # Give qemu's pid a chance to reach the file
if test -s "$resdir/qemu_pid"
then
qemu_pid=`cat "$resdir/qemu_pid"`
echo Monitoring qemu job at pid $qemu_pid
else
qemu_pid=""
echo Monitoring qemu job at yet-as-unknown pid
fi
fi
if test -n "$TORTURE_KCONFIG_GDB_ARG"
then
echo Waiting for you to attach a debug session, for example: > /dev/tty
echo " gdb $base_resdir/vmlinux" > /dev/tty
echo 'After symbols load and the "(gdb)" prompt appears:' > /dev/tty
echo " target remote :1234" > /dev/tty
echo " continue" > /dev/tty
kstarttime=`gawk 'BEGIN { print systime() }' < /dev/null`
fi
while :
do
if test -z "$qemu_pid" -a -s "$resdir/qemu_pid"
then
qemu_pid=`cat "$resdir/qemu_pid"`
fi
kruntime=`gawk 'BEGIN { print systime() - '"$kstarttime"' }' < /dev/null`
if test -z "$qemu_pid" || kill -0 "$qemu_pid" > /dev/null 2>&1
then
if test -n "$TORTURE_KCONFIG_GDB_ARG"
then
:
elif test $kruntime -ge $seconds || test -f "$resdir/../STOP.1"
then
break;
fi
sleep 1
else
commandcompleted=1
if test $kruntime -lt $seconds
then
echo Completed in $kruntime vs. $seconds >> $resdir/Warnings 2>&1
grep "^(qemu) qemu:" $resdir/kvm-test-1-run.sh.out >> $resdir/Warnings 2>&1
killpid="`sed -n "s/^(qemu) qemu: terminating on signal [0-9]* from pid \([0-9]*\).*$/\1/p" $resdir/Warnings`"
if test -n "$killpid"
then
echo "ps -fp $killpid" >> $resdir/Warnings 2>&1
ps -fp $killpid >> $resdir/Warnings 2>&1
fi
# Reduce probability of PID reuse by allowing a one-minute buffer
if test $((kruntime + 60)) -lt $seconds && test -s "$resdir/../jitter_pids"
then
awk < "$resdir/../jitter_pids" '
NF > 0 {
pidlist = pidlist " " $1;
n++;
}
END {
if (n > 0) {
print "kill " pidlist;
}
}' | sh
fi
else
echo ' ---' `date`: "Kernel done"
fi
break
fi
done
if test -z "$qemu_pid" -a -s "$resdir/qemu_pid"
then
qemu_pid=`cat "$resdir/qemu_pid"`
fi
if test $commandcompleted -eq 0 -a -n "$qemu_pid"
then
if ! test -f "$resdir/../STOP.1"
then
echo Grace period for qemu job at pid $qemu_pid
fi
oldline="`tail $resdir/console.log`"
while :
do
if test -f "$resdir/../STOP.1"
then
echo "PID $qemu_pid killed due to run STOP.1 request" >> $resdir/Warnings 2>&1
kill -KILL $qemu_pid
break
fi
kruntime=`gawk 'BEGIN { print systime() - '"$kstarttime"' }' < /dev/null`
if kill -0 $qemu_pid > /dev/null 2>&1
then
:
else
break
fi
must_continue=no
newline="`tail $resdir/console.log`"
if test "$newline" != "$oldline" && echo $newline | grep -q ' [0-9]\+us : '
then
must_continue=yes
fi
last_ts="`tail $resdir/console.log | grep '^\[ *[0-9]\+\.[0-9]\+]' | tail -1 | sed -e 's/^\[ *//' -e 's/\..*$//'`"
if test -z "$last_ts"
then
last_ts=0
fi
if test "$newline" != "$oldline" -a "$last_ts" -lt $((seconds + $TORTURE_SHUTDOWN_GRACE))
then
must_continue=yes
fi
if test $must_continue = no -a $kruntime -ge $((seconds + $TORTURE_SHUTDOWN_GRACE))
then
echo "!!! PID $qemu_pid hung at $kruntime vs. $seconds seconds" >> $resdir/Warnings 2>&1
kill -KILL $qemu_pid
break
fi
oldline=$newline
sleep 10
done
elif test -z "$qemu_pid"
then
echo Unknown PID, cannot kill qemu command
fi
parse-console.sh $resdir/console.log $title
kvm-test-1-run-qemu.sh $resdir

54
tools/testing/selftests/rcutorture/bin/kvm-transform.sh
View File

@ -3,7 +3,7 @@
#
# Transform a qemu-cmd file to allow reuse.
#
# Usage: kvm-transform.sh bzImage console.log < qemu-cmd-in > qemu-cmd-out
# Usage: kvm-transform.sh bzImage console.log jitter_dir [ seconds ] < qemu-cmd-in > qemu-cmd-out
#
# bzImage: Kernel and initrd from the same prior kvm.sh run.
# console.log: File into which to place console output.
@ -29,20 +29,62 @@ then
echo "Need console log file name."
exit 1
fi
jitter_dir="$3"
if test -z "$jitter_dir" || ! test -d "$jitter_dir"
then
echo "Need valid jitter directory: '$jitter_dir'"
exit 1
fi
seconds="$4"
if test -n "$seconds" && echo $seconds | grep -q '[^0-9]'
then
echo "Invalid duration, should be numeric in seconds: '$seconds'"
exit 1
fi
awk -v image="$image" -v consolelog="$consolelog" -v jitter_dir="$jitter_dir" \
-v seconds="$seconds" '
/^# seconds=/ {
if (seconds == "")
print $0;
else
print "# seconds=" seconds;
next;
}
/^# TORTURE_JITTER_START=/ {
print "# TORTURE_JITTER_START=\". jitterstart.sh " $4 " " jitter_dir " " $6 " " $7;
next;
}
/^# TORTURE_JITTER_STOP=/ {
print "# TORTURE_JITTER_STOP=\". jitterstop.sh " " " jitter_dir " " $5;
next;
}
/^#/ {
print $0;
next;
}
awk -v image="$image" -v consolelog="$consolelog" '
{
line = "";
for (i = 1; i <= NF; i++) {
if (line == "")
if ("" seconds != "" && $i ~ /\.shutdown_secs=[0-9]*$/) {
sub(/[0-9]*$/, seconds, $i);
if (line == "")
line = $i;
else
line = line " " $i;
} else if (line == "") {
line = $i;
else
} else {
line = line " " $i;
}
if ($i == "-serial") {
i++;
line = line " file:" consolelog;
}
if ($i == "-kernel") {
} else if ($i == "-kernel") {
i++;
line = line " " image;
}

142
tools/testing/selftests/rcutorture/bin/kvm.sh
View File

@ -29,17 +29,21 @@ PATH=${KVM}/bin:$PATH; export PATH
TORTURE_ALLOTED_CPUS="`identify_qemu_vcpus`"
TORTURE_DEFCONFIG=defconfig
TORTURE_BOOT_IMAGE=""
TORTURE_BUILDONLY=
TORTURE_INITRD="$KVM/initrd"; export TORTURE_INITRD
TORTURE_KCONFIG_ARG=""
TORTURE_KCONFIG_GDB_ARG=""
TORTURE_BOOT_GDB_ARG=""
TORTURE_QEMU_GDB_ARG=""
TORTURE_JITTER_START=""
TORTURE_JITTER_STOP=""
TORTURE_KCONFIG_KASAN_ARG=""
TORTURE_KCONFIG_KCSAN_ARG=""
TORTURE_KMAKE_ARG=""
TORTURE_QEMU_MEM=512
TORTURE_SHUTDOWN_GRACE=180
TORTURE_SUITE=rcu
TORTURE_MOD=rcutorture
TORTURE_TRUST_MAKE=""
resdir=""
configs=""
@ -100,7 +104,7 @@ do
TORTURE_BUILDONLY=1
;;
--configs|--config)
checkarg --configs "(list of config files)" "$#" "$2" '^[^/]\+$' '^--'
checkarg --configs "(list of config files)" "$#" "$2" '^[^/.a-z]\+$' '^--'
configs="$configs $2"
shift
;;
@ -116,7 +120,7 @@ do
shift
;;
--datestamp)
checkarg --datestamp "(relative pathname)" "$#" "$2" '^[a-zA-Z0-9._-/]*$' '^--'
checkarg --datestamp "(relative pathname)" "$#" "$2" '^[a-zA-Z0-9._/-]*$' '^--'
ds=$2
shift
;;
@ -215,6 +219,7 @@ do
--torture)
checkarg --torture "(suite name)" "$#" "$2" '^\(lock\|rcu\|rcuscale\|refscale\|scf\)$' '^--'
TORTURE_SUITE=$2
TORTURE_MOD="`echo $TORTURE_SUITE | sed -e 's/^\(lock\|rcu\|scf\)$/\1torture/'`"
shift
if test "$TORTURE_SUITE" = rcuscale || test "$TORTURE_SUITE" = refscale
then
@ -381,6 +386,7 @@ TORTURE_QEMU_GDB_ARG="$TORTURE_QEMU_GDB_ARG"; export TORTURE_QEMU_GDB_ARG
TORTURE_KCONFIG_KASAN_ARG="$TORTURE_KCONFIG_KASAN_ARG"; export TORTURE_KCONFIG_KASAN_ARG
TORTURE_KCONFIG_KCSAN_ARG="$TORTURE_KCONFIG_KCSAN_ARG"; export TORTURE_KCONFIG_KCSAN_ARG
TORTURE_KMAKE_ARG="$TORTURE_KMAKE_ARG"; export TORTURE_KMAKE_ARG
TORTURE_MOD="$TORTURE_MOD"; export TORTURE_MOD
TORTURE_QEMU_CMD="$TORTURE_QEMU_CMD"; export TORTURE_QEMU_CMD
TORTURE_QEMU_INTERACTIVE="$TORTURE_QEMU_INTERACTIVE"; export TORTURE_QEMU_INTERACTIVE
TORTURE_QEMU_MAC="$TORTURE_QEMU_MAC"; export TORTURE_QEMU_MAC
@ -399,12 +405,17 @@ echo Results directory: $resdir/$ds
echo $scriptname $args
touch $resdir/$ds/log
echo $scriptname $args >> $resdir/$ds/log
echo ${TORTURE_SUITE} > $resdir/$ds/TORTURE_SUITE
pwd > $resdir/$ds/testid.txt
echo ${TORTURE_SUITE} > $resdir/$ds/torture_suite
echo Build directory: `pwd` > $resdir/$ds/testid.txt
if test -d .git
then
echo Current commit: `git rev-parse HEAD` >> $resdir/$ds/testid.txt
echo >> $resdir/$ds/testid.txt
echo ' ---' Output of "'"git status"'": >> $resdir/$ds/testid.txt
git status >> $resdir/$ds/testid.txt
git rev-parse HEAD >> $resdir/$ds/testid.txt
echo >> $resdir/$ds/testid.txt
echo >> $resdir/$ds/testid.txt
echo ' ---' Output of "'"git diff HEAD"'": >> $resdir/$ds/testid.txt
git diff HEAD >> $resdir/$ds/testid.txt
fi
___EOF___
@ -434,42 +445,6 @@ function dump(first, pastlast, batchnum)
print "echo ----Start batch " batchnum ": `date` | tee -a " rd "log";
print "needqemurun="
jn=1
for (j = first; j < pastlast; j++) {
builddir=KVM "/b" j - first + 1
cpusr[jn] = cpus[j];
if (cfrep[cf[j]] == "") {
cfr[jn] = cf[j];
cfrep[cf[j]] = 1;
} else {
cfrep[cf[j]]++;
cfr[jn] = cf[j] "." cfrep[cf[j]];
}
if (cpusr[jn] > ncpus && ncpus != 0)
ovf = "-ovf";
else
ovf = "";
print "echo ", cfr[jn], cpusr[jn] ovf ": Starting build. `date` | tee -a " rd "log";
print "rm -f " builddir ".*";
print "touch " builddir ".wait";
print "mkdir " rd cfr[jn] " || :";
print "kvm-test-1-run.sh " CONFIGDIR cf[j], builddir, rd cfr[jn], dur " \"" TORTURE_QEMU_ARG "\" \"" TORTURE_BOOTARGS "\" > " rd cfr[jn] "/kvm-test-1-run.sh.out 2>&1 &"
print "echo ", cfr[jn], cpusr[jn] ovf ": Waiting for build to complete. `date` | tee -a " rd "log";
print "while test -f " builddir ".wait"
print "do"
print "\tsleep 1"
print "done"
print "echo ", cfr[jn], cpusr[jn] ovf ": Build complete. `date` | tee -a " rd "log";
jn++;
}
for (j = 1; j < jn; j++) {
builddir=KVM "/b" j
print "rm -f " builddir ".ready"
print "if test -f \"" rd cfr[j] "/builtkernel\""
print "then"
print "\techo ----", cfr[j], cpusr[j] ovf ": Kernel present. `date` | tee -a " rd "log";
print "\tneedqemurun=1"
print "fi"
}
njitter = 0;
split(jitter, ja);
if (ja[1] == -1 && ncpus == 0)
@ -478,6 +453,49 @@ function dump(first, pastlast, batchnum)
njitter = ncpus;
else
njitter = ja[1];
print "TORTURE_JITTER_START=\". jitterstart.sh " njitter " " rd " " dur " " ja[2] " " ja[3] "\"; export TORTURE_JITTER_START";
print "TORTURE_JITTER_STOP=\". jitterstop.sh " rd " \"; export TORTURE_JITTER_STOP"
for (j = first; j < pastlast; j++) {
cpusr[jn] = cpus[j];
if (cfrep[cf[j]] == "") {
cfr[jn] = cf[j];
cfrep[cf[j]] = 1;
} else {
cfrep[cf[j]]++;
cfr[jn] = cf[j] "." cfrep[cf[j]];
}
builddir=rd cfr[jn] "/build";
if (cpusr[jn] > ncpus && ncpus != 0)
ovf = "-ovf";
else
ovf = "";
print "echo ", cfr[jn], cpusr[jn] ovf ": Starting build. `date` | tee -a " rd "log";
print "mkdir " rd cfr[jn] " || :";
print "touch " builddir ".wait";
print "kvm-test-1-run.sh " CONFIGDIR cf[j], rd cfr[jn], dur " \"" TORTURE_QEMU_ARG "\" \"" TORTURE_BOOTARGS "\" > " rd cfr[jn] "/kvm-test-1-run.sh.out 2>&1 &"
print "echo ", cfr[jn], cpusr[jn] ovf ": Waiting for build to complete. `date` | tee -a " rd "log";
print "while test -f " builddir ".wait"
print "do"
print "\tsleep 1"
print "done"
print "echo ", cfr[jn], cpusr[jn] ovf ": Build complete. `date` | tee -a " rd "log";
jn++;
}
print "runfiles="
for (j = 1; j < jn; j++) {
builddir=rd cfr[j] "/build";
if (TORTURE_BUILDONLY)
print "rm -f " builddir ".ready"
else
print "mv " builddir ".ready " builddir ".run"
print "runfiles=\"$runfiles " builddir ".run\""
fi
print "if test -f \"" rd cfr[j] "/builtkernel\""
print "then"
print "\techo ----", cfr[j], cpusr[j] ovf ": Kernel present. `date` | tee -a " rd "log";
print "\tneedqemurun=1"
print "fi"
}
if (TORTURE_BUILDONLY && njitter != 0) {
njitter = 0;
print "echo Build-only run, so suppressing jitter | tee -a " rd "log"
@ -488,19 +506,18 @@ function dump(first, pastlast, batchnum)
print "if test -n \"$needqemurun\""
print "then"
print "\techo ---- Starting kernels. `date` | tee -a " rd "log";
print "\techo > " rd "jitter_pids"
for (j = 0; j < njitter; j++) {
print "\tjitter.sh " j " " dur " " ja[2] " " ja[3] "&"
print "\techo $! >> " rd "jitter_pids"
}
print "\twait"
print "\t$TORTURE_JITTER_START";
print "\twhile ls $runfiles > /dev/null 2>&1"
print "\tdo"
print "\t\t:"
print "\tdone"
print "\t$TORTURE_JITTER_STOP";
print "\techo ---- All kernel runs complete. `date` | tee -a " rd "log";
print "else"
print "\twait"
print "\techo ---- No kernel runs. `date` | tee -a " rd "log";
print "fi"
for (j = 1; j < jn; j++) {
builddir=KVM "/b" j
print "echo ----", cfr[j], cpusr[j] ovf ": Build/run results: | tee -a " rd "log";
print "cat " rd cfr[j] "/kvm-test-1-run.sh.out | tee -a " rd "log";
}
@ -548,6 +565,18 @@ echo 'ret=$?' >> $T/script
echo "cat $T/kvm-recheck.sh.out | tee -a $resdir/$ds/log" >> $T/script
echo 'exit $ret' >> $T/script
# Extract the tests and their batches from the script.
egrep 'Start batch|Starting build\.' $T/script | grep -v ">>" |
sed -e 's/:.*$//' -e 's/^echo //' -e 's/-ovf//' |
awk '
/^----Start/ {
batchno = $3;
next;
}
{
print batchno, $1, $2
}' > $T/batches
if test "$dryrun" = script
then
cat $T/script
@ -566,21 +595,14 @@ then
exit 0
elif test "$dryrun" = batches
then
# Extract the tests and their batches from the script.
egrep 'Start batch|Starting build\.' $T/script | grep -v ">>" |
sed -e 's/:.*$//' -e 's/^echo //' -e 's/-ovf//' |
awk '
/^----Start/ {
batchno = $3;
next;
}
{
print batchno, $1, $2
}'
cat $T/batches
exit 0
else
# Not a dryrun, so run the script.
# Not a dryrun. Record the batches and the number of CPUs, then run the script.
bash $T/script
ret=$?
cp $T/batches $resdir/$ds/batches
echo '#' cpus=$cpus >> $resdir/$ds/batches
echo " --- Done at `date` (`get_starttime_duration $starttime`) exitcode $ret" | tee -a $resdir/$ds/log
exit $ret
fi

2
tools/testing/selftests/rcutorture/bin/torture.sh
View File

@ -374,7 +374,7 @@ done
if test "$do_kvfree" = "yes"
then
torture_bootargs="rcuscale.kfree_rcu_test=1 rcuscale.kfree_nthreads=16 rcuscale.holdoff=20 rcuscale.kfree_loops=10000 torture.disable_onoff_at_boot"
torture_set "rcuscale-kvfree" tools/testing/selftests/rcutorture/bin/kvm.sh --torture rcuscale --allcpus --duration 10 --kconfig "CONFIG_NR_CPUS=$HALF_ALLOTED_CPUS" --trust-make
torture_set "rcuscale-kvfree" tools/testing/selftests/rcutorture/bin/kvm.sh --torture rcuscale --allcpus --duration 10 --kconfig "CONFIG_NR_CPUS=$HALF_ALLOTED_CPUS" --memory 1G --trust-make
fi
echo " --- " $scriptname $args

4
tools/testing/selftests/rcutorture/configs/rcu/CFLIST
View File

@ -7,8 +7,8 @@ TREE07
TREE09
SRCU-N
SRCU-P
SRCU-t
SRCU-u
SRCU-T
SRCU-U
TINY01
TINY02
TASKS01

0
tools/testing/selftests/rcutorture/configs/rcu/SRCU-t → tools/testing/selftests/rcutorture/configs/rcu/SRCU-T
View File

0
tools/testing/selftests/rcutorture/configs/rcu/SRCU-t.boot → tools/testing/selftests/rcutorture/configs/rcu/SRCU-T.boot
View File

0
tools/testing/selftests/rcutorture/configs/rcu/SRCU-u → tools/testing/selftests/rcutorture/configs/rcu/SRCU-U
View File

0
tools/testing/selftests/rcutorture/configs/rcu/SRCU-u.boot → tools/testing/selftests/rcutorture/configs/rcu/SRCU-U.boot
View File

1
tools/testing/selftests/rcutorture/configs/rcu/TREE03.boot
View File

@ -4,3 +4,4 @@ rcutree.gp_init_delay=3
rcutree.gp_cleanup_delay=3
rcutree.kthread_prio=2
threadirqs
tree.use_softirq=0

2
tools/testing/selftests/rcutorture/configs/rcu/TREE04.boot
View File

@ -1 +1 @@
rcutree.rcu_fanout_leaf=4 nohz_full=1-7
rcutree.rcu_fanout_leaf=4 nohz_full=1-N

2
tools/testing/selftests/rcutorture/configs/rcu/TREE08.boot
View File

@ -1,3 +1,3 @@
rcupdate.rcu_self_test=1
rcutree.rcu_fanout_exact=1
rcu_nocbs=0-7
rcu_nocbs=all

2
tools/testing/selftests/rcutorture/configs/rcuscale/ver_functions.sh
View File

@ -12,5 +12,5 @@
# Adds per-version torture-module parameters to kernels supporting them.
per_version_boot_params () {
echo $1 rcuscale.shutdown=1 \
rcuscale.verbose=1
rcuscale.verbose=0
}

2
tools/testing/selftests/rcutorture/configs/refscale/ver_functions.sh
View File

@ -12,5 +12,5 @@
# Adds per-version torture-module parameters to kernels supporting them.
per_version_boot_params () {
echo $1 refscale.shutdown=1 \
refscale.verbose=1
refscale.verbose=0
}