linux-stable/tools/perf/Documentation/perf-lock.txt

205 lines
4.9 KiB
Text
Raw Normal View History

perf-lock(1)
============
NAME
----
perf-lock - Analyze lock events
SYNOPSIS
--------
[verse]
'perf lock' {record|report|script|info|contention}
DESCRIPTION
-----------
You can analyze various lock behaviours
and statistics with this 'perf lock' command.
'perf lock record <command>' records lock events
between start and end <command>. And this command
produces the file "perf.data" which contains tracing
results of lock events.
'perf lock report' reports statistical data.
'perf lock script' shows raw lock events.
'perf lock info' shows metadata like threads or addresses
of lock instances.
'perf lock contention' shows contention statistics.
COMMON OPTIONS
--------------
-i::
--input=<file>::
Input file name. (default: perf.data unless stdin is a fifo)
-v::
--verbose::
Be more verbose (show symbol address, etc).
-q::
--quiet::
Do not show any warnings or messages. (Suppress -v)
-D::
--dump-raw-trace::
Dump raw trace in ASCII.
-f::
--force::
Don't complain, do it.
--vmlinux=<file>::
vmlinux pathname
--kallsyms=<file>::
kallsyms pathname
REPORT OPTIONS
--------------
-k::
--key=<value>::
Sorting key. Possible values: acquired (default), contended,
avg_wait, wait_total, wait_max, wait_min.
-F::
--field=<value>::
Output fields. By default it shows all the fields but users can
customize that using this. Possible values: acquired, contended,
avg_wait, wait_total, wait_max, wait_min.
2022-01-27 00:00:49 +00:00
-c::
--combine-locks::
Merge lock instances in the same class (based on name).
-t::
--threads::
The -t option is to show per-thread lock stat like below:
$ perf lock report -t -F acquired,contended,avg_wait
Name acquired contended avg wait (ns)
perf 240569 9 5784
swapper 106610 19 543
:15789 17370 2 14538
ContainerMgr 8981 6 874
sleep 5275 1 11281
ContainerThread 4416 4 944
RootPressureThr 3215 5 1215
rcu_preempt 2954 0 0
ContainerMgr 2560 0 0
unnamed 1873 0 0
EventManager_De 1845 1 636
futex-default-S 1609 0 0
-E::
--entries=<value>::
Display this many entries.
INFO OPTIONS
------------
-t::
--threads::
dump thread list in perf.data
-m::
--map::
dump map of lock instances (address:name table)
CONTENTION OPTIONS
--------------
-k::
--key=<value>::
Sorting key. Possible values: contended, wait_total (default),
wait_max, wait_min, avg_wait.
-F::
--field=<value>::
Output fields. By default it shows all but the wait_min fields
and users can customize that using this. Possible values:
contended, wait_total, wait_max, wait_min, avg_wait.
-t::
--threads::
Show per-thread lock contention stat
-b::
--use-bpf::
Use BPF program to collect lock contention stats instead of
using the input data.
-a::
--all-cpus::
System-wide collection from all CPUs.
-C::
perf lock contention: Add -Y/--type-filter option The -Y/--type-filter option is to filter the result for specific lock types only. It can accept comma-separated values. Note that it would accept type names like one in the output. spinlock, mutex, rwsem:R and so on. For RW-variant lock types, it converts the name to the both variants. In other words, "rwsem" is same as "rwsem:R,rwsem:W". Also note that "mutex" has two different encoding - one for sleeping wait, another for optimistic spinning. Add "mutex-spin" entry for the lock_type_table so that we can add it for "mutex" under the table. $ sudo ./perf lock record -a -- ./perf bench sched messaging $ sudo ./perf lock con -E 5 -Y spinlock contended total wait max wait avg wait type caller 802 1.26 ms 11.73 us 1.58 us spinlock __wake_up_common_lock+0x62 13 787.16 us 105.44 us 60.55 us spinlock remove_wait_queue+0x14 12 612.96 us 78.70 us 51.08 us spinlock prepare_to_wait+0x27 114 340.68 us 12.61 us 2.99 us spinlock try_to_wake_up+0x1f5 83 226.38 us 9.15 us 2.73 us spinlock folio_lruvec_lock_irqsave+0x5e Committer notes: Make get_type_flag() return UINT_MAX for error instad of -1UL, as that function returns 'unsigned int' and we store the value on a 'unsigned int' 'flags' variable which makes clang unhappy: 35 98.23 fedora:37 : FAIL clang version 15.0.6 (Fedora 15.0.6-1.fc37) builtin-lock.c:2012:14: error: result of comparison of constant 18446744073709551615 with expression of type 'unsigned int' is always true [-Werror,-Wtautological-constant-out-of-range-compare] if (flags != -1UL) { ~~~~~ ^ ~~~~ builtin-lock.c:2021:14: error: result of comparison of constant 18446744073709551615 with expression of type 'unsigned int' is always true [-Werror,-Wtautological-constant-out-of-range-compare] if (flags != -1UL) { ~~~~~ ^ ~~~~ builtin-lock.c:2037:14: error: result of comparison of constant 18446744073709551615 with expression of type 'unsigned int' is always true [-Werror,-Wtautological-constant-out-of-range-compare] if (flags != -1UL) { ~~~~~ ^ ~~~~ 3 errors generated. Signed-off-by: Namhyung Kim <namhyung@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Blake Jones <blakejones@google.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Song Liu <song@kernel.org> Cc: bpf@vger.kernel.org Link: https://lore.kernel.org/r/20221219201732.460111-3-namhyung@kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2022-12-19 20:17:28 +00:00
--cpu=<value>::
Collect samples only on the list of CPUs provided. Multiple CPUs can be
provided as a comma-separated list with no space: 0,1. Ranges of CPUs
are specified with -: 0-2. Default is to monitor all CPUs.
-p::
perf lock contention: Add -Y/--type-filter option The -Y/--type-filter option is to filter the result for specific lock types only. It can accept comma-separated values. Note that it would accept type names like one in the output. spinlock, mutex, rwsem:R and so on. For RW-variant lock types, it converts the name to the both variants. In other words, "rwsem" is same as "rwsem:R,rwsem:W". Also note that "mutex" has two different encoding - one for sleeping wait, another for optimistic spinning. Add "mutex-spin" entry for the lock_type_table so that we can add it for "mutex" under the table. $ sudo ./perf lock record -a -- ./perf bench sched messaging $ sudo ./perf lock con -E 5 -Y spinlock contended total wait max wait avg wait type caller 802 1.26 ms 11.73 us 1.58 us spinlock __wake_up_common_lock+0x62 13 787.16 us 105.44 us 60.55 us spinlock remove_wait_queue+0x14 12 612.96 us 78.70 us 51.08 us spinlock prepare_to_wait+0x27 114 340.68 us 12.61 us 2.99 us spinlock try_to_wake_up+0x1f5 83 226.38 us 9.15 us 2.73 us spinlock folio_lruvec_lock_irqsave+0x5e Committer notes: Make get_type_flag() return UINT_MAX for error instad of -1UL, as that function returns 'unsigned int' and we store the value on a 'unsigned int' 'flags' variable which makes clang unhappy: 35 98.23 fedora:37 : FAIL clang version 15.0.6 (Fedora 15.0.6-1.fc37) builtin-lock.c:2012:14: error: result of comparison of constant 18446744073709551615 with expression of type 'unsigned int' is always true [-Werror,-Wtautological-constant-out-of-range-compare] if (flags != -1UL) { ~~~~~ ^ ~~~~ builtin-lock.c:2021:14: error: result of comparison of constant 18446744073709551615 with expression of type 'unsigned int' is always true [-Werror,-Wtautological-constant-out-of-range-compare] if (flags != -1UL) { ~~~~~ ^ ~~~~ builtin-lock.c:2037:14: error: result of comparison of constant 18446744073709551615 with expression of type 'unsigned int' is always true [-Werror,-Wtautological-constant-out-of-range-compare] if (flags != -1UL) { ~~~~~ ^ ~~~~ 3 errors generated. Signed-off-by: Namhyung Kim <namhyung@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Blake Jones <blakejones@google.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Song Liu <song@kernel.org> Cc: bpf@vger.kernel.org Link: https://lore.kernel.org/r/20221219201732.460111-3-namhyung@kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2022-12-19 20:17:28 +00:00
--pid=<value>::
Record events on existing process ID (comma separated list).
perf lock contention: Add -Y/--type-filter option The -Y/--type-filter option is to filter the result for specific lock types only. It can accept comma-separated values. Note that it would accept type names like one in the output. spinlock, mutex, rwsem:R and so on. For RW-variant lock types, it converts the name to the both variants. In other words, "rwsem" is same as "rwsem:R,rwsem:W". Also note that "mutex" has two different encoding - one for sleeping wait, another for optimistic spinning. Add "mutex-spin" entry for the lock_type_table so that we can add it for "mutex" under the table. $ sudo ./perf lock record -a -- ./perf bench sched messaging $ sudo ./perf lock con -E 5 -Y spinlock contended total wait max wait avg wait type caller 802 1.26 ms 11.73 us 1.58 us spinlock __wake_up_common_lock+0x62 13 787.16 us 105.44 us 60.55 us spinlock remove_wait_queue+0x14 12 612.96 us 78.70 us 51.08 us spinlock prepare_to_wait+0x27 114 340.68 us 12.61 us 2.99 us spinlock try_to_wake_up+0x1f5 83 226.38 us 9.15 us 2.73 us spinlock folio_lruvec_lock_irqsave+0x5e Committer notes: Make get_type_flag() return UINT_MAX for error instad of -1UL, as that function returns 'unsigned int' and we store the value on a 'unsigned int' 'flags' variable which makes clang unhappy: 35 98.23 fedora:37 : FAIL clang version 15.0.6 (Fedora 15.0.6-1.fc37) builtin-lock.c:2012:14: error: result of comparison of constant 18446744073709551615 with expression of type 'unsigned int' is always true [-Werror,-Wtautological-constant-out-of-range-compare] if (flags != -1UL) { ~~~~~ ^ ~~~~ builtin-lock.c:2021:14: error: result of comparison of constant 18446744073709551615 with expression of type 'unsigned int' is always true [-Werror,-Wtautological-constant-out-of-range-compare] if (flags != -1UL) { ~~~~~ ^ ~~~~ builtin-lock.c:2037:14: error: result of comparison of constant 18446744073709551615 with expression of type 'unsigned int' is always true [-Werror,-Wtautological-constant-out-of-range-compare] if (flags != -1UL) { ~~~~~ ^ ~~~~ 3 errors generated. Signed-off-by: Namhyung Kim <namhyung@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Blake Jones <blakejones@google.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Song Liu <song@kernel.org> Cc: bpf@vger.kernel.org Link: https://lore.kernel.org/r/20221219201732.460111-3-namhyung@kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2022-12-19 20:17:28 +00:00
--tid=<value>::
Record events on existing thread ID (comma separated list).
perf lock contention: Add -Y/--type-filter option The -Y/--type-filter option is to filter the result for specific lock types only. It can accept comma-separated values. Note that it would accept type names like one in the output. spinlock, mutex, rwsem:R and so on. For RW-variant lock types, it converts the name to the both variants. In other words, "rwsem" is same as "rwsem:R,rwsem:W". Also note that "mutex" has two different encoding - one for sleeping wait, another for optimistic spinning. Add "mutex-spin" entry for the lock_type_table so that we can add it for "mutex" under the table. $ sudo ./perf lock record -a -- ./perf bench sched messaging $ sudo ./perf lock con -E 5 -Y spinlock contended total wait max wait avg wait type caller 802 1.26 ms 11.73 us 1.58 us spinlock __wake_up_common_lock+0x62 13 787.16 us 105.44 us 60.55 us spinlock remove_wait_queue+0x14 12 612.96 us 78.70 us 51.08 us spinlock prepare_to_wait+0x27 114 340.68 us 12.61 us 2.99 us spinlock try_to_wake_up+0x1f5 83 226.38 us 9.15 us 2.73 us spinlock folio_lruvec_lock_irqsave+0x5e Committer notes: Make get_type_flag() return UINT_MAX for error instad of -1UL, as that function returns 'unsigned int' and we store the value on a 'unsigned int' 'flags' variable which makes clang unhappy: 35 98.23 fedora:37 : FAIL clang version 15.0.6 (Fedora 15.0.6-1.fc37) builtin-lock.c:2012:14: error: result of comparison of constant 18446744073709551615 with expression of type 'unsigned int' is always true [-Werror,-Wtautological-constant-out-of-range-compare] if (flags != -1UL) { ~~~~~ ^ ~~~~ builtin-lock.c:2021:14: error: result of comparison of constant 18446744073709551615 with expression of type 'unsigned int' is always true [-Werror,-Wtautological-constant-out-of-range-compare] if (flags != -1UL) { ~~~~~ ^ ~~~~ builtin-lock.c:2037:14: error: result of comparison of constant 18446744073709551615 with expression of type 'unsigned int' is always true [-Werror,-Wtautological-constant-out-of-range-compare] if (flags != -1UL) { ~~~~~ ^ ~~~~ 3 errors generated. Signed-off-by: Namhyung Kim <namhyung@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Blake Jones <blakejones@google.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Song Liu <song@kernel.org> Cc: bpf@vger.kernel.org Link: https://lore.kernel.org/r/20221219201732.460111-3-namhyung@kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2022-12-19 20:17:28 +00:00
--map-nr-entries=<value>::
Maximum number of BPF map entries (default: 10240).
perf lock contention: Add -Y/--type-filter option The -Y/--type-filter option is to filter the result for specific lock types only. It can accept comma-separated values. Note that it would accept type names like one in the output. spinlock, mutex, rwsem:R and so on. For RW-variant lock types, it converts the name to the both variants. In other words, "rwsem" is same as "rwsem:R,rwsem:W". Also note that "mutex" has two different encoding - one for sleeping wait, another for optimistic spinning. Add "mutex-spin" entry for the lock_type_table so that we can add it for "mutex" under the table. $ sudo ./perf lock record -a -- ./perf bench sched messaging $ sudo ./perf lock con -E 5 -Y spinlock contended total wait max wait avg wait type caller 802 1.26 ms 11.73 us 1.58 us spinlock __wake_up_common_lock+0x62 13 787.16 us 105.44 us 60.55 us spinlock remove_wait_queue+0x14 12 612.96 us 78.70 us 51.08 us spinlock prepare_to_wait+0x27 114 340.68 us 12.61 us 2.99 us spinlock try_to_wake_up+0x1f5 83 226.38 us 9.15 us 2.73 us spinlock folio_lruvec_lock_irqsave+0x5e Committer notes: Make get_type_flag() return UINT_MAX for error instad of -1UL, as that function returns 'unsigned int' and we store the value on a 'unsigned int' 'flags' variable which makes clang unhappy: 35 98.23 fedora:37 : FAIL clang version 15.0.6 (Fedora 15.0.6-1.fc37) builtin-lock.c:2012:14: error: result of comparison of constant 18446744073709551615 with expression of type 'unsigned int' is always true [-Werror,-Wtautological-constant-out-of-range-compare] if (flags != -1UL) { ~~~~~ ^ ~~~~ builtin-lock.c:2021:14: error: result of comparison of constant 18446744073709551615 with expression of type 'unsigned int' is always true [-Werror,-Wtautological-constant-out-of-range-compare] if (flags != -1UL) { ~~~~~ ^ ~~~~ builtin-lock.c:2037:14: error: result of comparison of constant 18446744073709551615 with expression of type 'unsigned int' is always true [-Werror,-Wtautological-constant-out-of-range-compare] if (flags != -1UL) { ~~~~~ ^ ~~~~ 3 errors generated. Signed-off-by: Namhyung Kim <namhyung@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Blake Jones <blakejones@google.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Song Liu <song@kernel.org> Cc: bpf@vger.kernel.org Link: https://lore.kernel.org/r/20221219201732.460111-3-namhyung@kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2022-12-19 20:17:28 +00:00
--max-stack=<value>::
perf lock contention: Allow to change stack depth and skip It needs stack traces to find callers of locks. To minimize the performance overhead it only collects up to 8 entries for each stack trace. And it skips first 3 entries as they came from BPF, tracepoint and lock functions which are not interested for most users. But it turned out that those numbers are different in some configuration. Using fixed number can result in non meaningful caller names. Let's make them adjustable with --stack-depth and --skip-stack options. On my setup, the default output is like below: # /perf lock con -ab -F contended,wait_total sleep 3 contended total wait type caller 28 4.55 ms rwlock:W __bpf_trace_contention_begin+0xb 33 1.67 ms rwlock:W __bpf_trace_contention_begin+0xb 12 580.28 us spinlock __bpf_trace_contention_begin+0xb 60 240.54 us rwsem:R __bpf_trace_contention_begin+0xb 27 64.45 us spinlock __bpf_trace_contention_begin+0xb If I change the stack skip to 5, the result will be like: # perf lock con -ab -F contended,wait_total --stack-skip 5 sleep 3 contended total wait type caller 32 715.45 us spinlock folio_lruvec_lock_irqsave+0x61 26 550.22 us spinlock folio_lruvec_lock_irqsave+0x61 15 486.93 us rwsem:R mmap_read_lock+0x13 12 139.66 us rwsem:W vm_mmap_pgoff+0x93 1 7.04 us spinlock tick_do_update_jiffies64+0x25 Signed-off-by: Namhyung Kim <namhyung@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Song Liu <songliubraving@fb.com> Cc: bpf@vger.kernel.org Link: https://lore.kernel.org/r/20220912055314.744552-4-namhyung@kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2022-09-12 05:53:13 +00:00
Maximum stack depth when collecting lock contention (default: 8).
perf lock contention: Add -Y/--type-filter option The -Y/--type-filter option is to filter the result for specific lock types only. It can accept comma-separated values. Note that it would accept type names like one in the output. spinlock, mutex, rwsem:R and so on. For RW-variant lock types, it converts the name to the both variants. In other words, "rwsem" is same as "rwsem:R,rwsem:W". Also note that "mutex" has two different encoding - one for sleeping wait, another for optimistic spinning. Add "mutex-spin" entry for the lock_type_table so that we can add it for "mutex" under the table. $ sudo ./perf lock record -a -- ./perf bench sched messaging $ sudo ./perf lock con -E 5 -Y spinlock contended total wait max wait avg wait type caller 802 1.26 ms 11.73 us 1.58 us spinlock __wake_up_common_lock+0x62 13 787.16 us 105.44 us 60.55 us spinlock remove_wait_queue+0x14 12 612.96 us 78.70 us 51.08 us spinlock prepare_to_wait+0x27 114 340.68 us 12.61 us 2.99 us spinlock try_to_wake_up+0x1f5 83 226.38 us 9.15 us 2.73 us spinlock folio_lruvec_lock_irqsave+0x5e Committer notes: Make get_type_flag() return UINT_MAX for error instad of -1UL, as that function returns 'unsigned int' and we store the value on a 'unsigned int' 'flags' variable which makes clang unhappy: 35 98.23 fedora:37 : FAIL clang version 15.0.6 (Fedora 15.0.6-1.fc37) builtin-lock.c:2012:14: error: result of comparison of constant 18446744073709551615 with expression of type 'unsigned int' is always true [-Werror,-Wtautological-constant-out-of-range-compare] if (flags != -1UL) { ~~~~~ ^ ~~~~ builtin-lock.c:2021:14: error: result of comparison of constant 18446744073709551615 with expression of type 'unsigned int' is always true [-Werror,-Wtautological-constant-out-of-range-compare] if (flags != -1UL) { ~~~~~ ^ ~~~~ builtin-lock.c:2037:14: error: result of comparison of constant 18446744073709551615 with expression of type 'unsigned int' is always true [-Werror,-Wtautological-constant-out-of-range-compare] if (flags != -1UL) { ~~~~~ ^ ~~~~ 3 errors generated. Signed-off-by: Namhyung Kim <namhyung@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Blake Jones <blakejones@google.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Song Liu <song@kernel.org> Cc: bpf@vger.kernel.org Link: https://lore.kernel.org/r/20221219201732.460111-3-namhyung@kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2022-12-19 20:17:28 +00:00
--stack-skip=<value>::
perf lock contention: Allow to change stack depth and skip It needs stack traces to find callers of locks. To minimize the performance overhead it only collects up to 8 entries for each stack trace. And it skips first 3 entries as they came from BPF, tracepoint and lock functions which are not interested for most users. But it turned out that those numbers are different in some configuration. Using fixed number can result in non meaningful caller names. Let's make them adjustable with --stack-depth and --skip-stack options. On my setup, the default output is like below: # /perf lock con -ab -F contended,wait_total sleep 3 contended total wait type caller 28 4.55 ms rwlock:W __bpf_trace_contention_begin+0xb 33 1.67 ms rwlock:W __bpf_trace_contention_begin+0xb 12 580.28 us spinlock __bpf_trace_contention_begin+0xb 60 240.54 us rwsem:R __bpf_trace_contention_begin+0xb 27 64.45 us spinlock __bpf_trace_contention_begin+0xb If I change the stack skip to 5, the result will be like: # perf lock con -ab -F contended,wait_total --stack-skip 5 sleep 3 contended total wait type caller 32 715.45 us spinlock folio_lruvec_lock_irqsave+0x61 26 550.22 us spinlock folio_lruvec_lock_irqsave+0x61 15 486.93 us rwsem:R mmap_read_lock+0x13 12 139.66 us rwsem:W vm_mmap_pgoff+0x93 1 7.04 us spinlock tick_do_update_jiffies64+0x25 Signed-off-by: Namhyung Kim <namhyung@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Song Liu <songliubraving@fb.com> Cc: bpf@vger.kernel.org Link: https://lore.kernel.org/r/20220912055314.744552-4-namhyung@kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2022-09-12 05:53:13 +00:00
Number of stack depth to skip when finding a lock caller (default: 3).
-E::
--entries=<value>::
Display this many entries.
-l::
--lock-addr::
Show lock contention stat by address
perf lock contention: Add -o/--lock-owner option When there're many lock contentions in the system, people sometimes want to know who caused the contention, IOW who's the owner of the locks. The -o/--lock-owner option tries to follow the lock owners for the contended mutexes and rwsems from BPF, and then attributes the contention time to the owner instead of the waiter. It's a best effort approach to get the owner info at the time of the contention and doesn't guarantee to have the precise tracking of owners if it's changing over time. Currently it only handles mutex and rwsem that have owner field in their struct and it basically points to a task_struct that owns the lock at the moment. Technically its type is atomic_long_t and it comes with some LSB bits used for other meanings. So it needs to clear them when casting it to a pointer to task_struct. Also the atomic_long_t is a typedef of the atomic 32 or 64 bit types depending on arch which is a wrapper struct for the counter value. I'm not aware of proper ways to access those kernel atomic types from BPF so I just read the internal counter value directly. Please let me know if there's a better way. When -o/--lock-owner option is used, it goes to the task aggregation mode like -t/--threads option does. However it cannot get the owner for other lock types like spinlock and sometimes even for mutex. $ sudo ./perf lock con -abo -- ./perf bench sched pipe # Running 'sched/pipe' benchmark: # Executed 1000000 pipe operations between two processes Total time: 4.766 [sec] 4.766540 usecs/op 209795 ops/sec contended total wait max wait avg wait pid owner 403 565.32 us 26.81 us 1.40 us -1 Unknown 4 27.99 us 8.57 us 7.00 us 1583145 sched-pipe 1 8.25 us 8.25 us 8.25 us 1583144 sched-pipe 1 2.03 us 2.03 us 2.03 us 5068 chrome As you can see, the owner is unknown for the most cases. But if we filter only for the mutex locks, it'd more likely get the onwers. $ sudo ./perf lock con -abo -Y mutex -- ./perf bench sched pipe # Running 'sched/pipe' benchmark: # Executed 1000000 pipe operations between two processes Total time: 4.910 [sec] 4.910435 usecs/op 203647 ops/sec contended total wait max wait avg wait pid owner 2 15.50 us 8.29 us 7.75 us 1582852 sched-pipe 7 7.20 us 2.47 us 1.03 us -1 Unknown 1 6.74 us 6.74 us 6.74 us 1582851 sched-pipe Signed-off-by: Namhyung Kim <namhyung@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Davidlohr Bueso <dave@stgolabs.net> Cc: Hao Luo <haoluo@google.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Song Liu <song@kernel.org> Cc: Waiman Long <longman@redhat.com> Cc: Will Deacon <will@kernel.org> Cc: bpf@vger.kernel.org Link: https://lore.kernel.org/r/20230207002403.63590-3-namhyung@kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2023-02-07 00:24:02 +00:00
-o::
--lock-owner::
Show lock contention stat by owners. Implies --threads and
requires --use-bpf.
perf lock contention: Add -Y/--type-filter option The -Y/--type-filter option is to filter the result for specific lock types only. It can accept comma-separated values. Note that it would accept type names like one in the output. spinlock, mutex, rwsem:R and so on. For RW-variant lock types, it converts the name to the both variants. In other words, "rwsem" is same as "rwsem:R,rwsem:W". Also note that "mutex" has two different encoding - one for sleeping wait, another for optimistic spinning. Add "mutex-spin" entry for the lock_type_table so that we can add it for "mutex" under the table. $ sudo ./perf lock record -a -- ./perf bench sched messaging $ sudo ./perf lock con -E 5 -Y spinlock contended total wait max wait avg wait type caller 802 1.26 ms 11.73 us 1.58 us spinlock __wake_up_common_lock+0x62 13 787.16 us 105.44 us 60.55 us spinlock remove_wait_queue+0x14 12 612.96 us 78.70 us 51.08 us spinlock prepare_to_wait+0x27 114 340.68 us 12.61 us 2.99 us spinlock try_to_wake_up+0x1f5 83 226.38 us 9.15 us 2.73 us spinlock folio_lruvec_lock_irqsave+0x5e Committer notes: Make get_type_flag() return UINT_MAX for error instad of -1UL, as that function returns 'unsigned int' and we store the value on a 'unsigned int' 'flags' variable which makes clang unhappy: 35 98.23 fedora:37 : FAIL clang version 15.0.6 (Fedora 15.0.6-1.fc37) builtin-lock.c:2012:14: error: result of comparison of constant 18446744073709551615 with expression of type 'unsigned int' is always true [-Werror,-Wtautological-constant-out-of-range-compare] if (flags != -1UL) { ~~~~~ ^ ~~~~ builtin-lock.c:2021:14: error: result of comparison of constant 18446744073709551615 with expression of type 'unsigned int' is always true [-Werror,-Wtautological-constant-out-of-range-compare] if (flags != -1UL) { ~~~~~ ^ ~~~~ builtin-lock.c:2037:14: error: result of comparison of constant 18446744073709551615 with expression of type 'unsigned int' is always true [-Werror,-Wtautological-constant-out-of-range-compare] if (flags != -1UL) { ~~~~~ ^ ~~~~ 3 errors generated. Signed-off-by: Namhyung Kim <namhyung@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Blake Jones <blakejones@google.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Song Liu <song@kernel.org> Cc: bpf@vger.kernel.org Link: https://lore.kernel.org/r/20221219201732.460111-3-namhyung@kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2022-12-19 20:17:28 +00:00
-Y::
--type-filter=<value>::
Show lock contention only for given lock types (comma separated list).
Available values are:
semaphore, spinlock, rwlock, rwlock:R, rwlock:W, rwsem, rwsem:R, rwsem:W,
rtmutex, rwlock-rt, rwlock-rt:R, rwlock-rt:W, pcpu-sem, pcpu-sem:R, pcpu-sem:W,
mutex
Note that RW-variant of locks have :R and :W suffix. Names without the
suffix are shortcuts for the both variants. Ex) rwsem = rwsem:R + rwsem:W.
perf lock contention: Add -L/--lock-filter option The -L/--lock-filter option is to filter only given locks. The locks can be specified by address or name (if exists). $ sudo ./perf lock record -a sleep 1 $ sudo ./perf lock con -l contended total wait max wait avg wait address symbol 57 1.11 ms 42.83 us 19.54 us ffff9f4140059000 15 280.88 us 23.51 us 18.73 us ffffffff9d007a40 jiffies_lock 1 20.49 us 20.49 us 20.49 us ffffffff9d0d50c0 rcu_state 1 9.02 us 9.02 us 9.02 us ffff9f41759e9ba0 $ sudo ./perf lock con -L jiffies_lock,rcu_state contended total wait max wait avg wait type caller 15 280.88 us 23.51 us 18.73 us spinlock tick_sched_do_timer+0x93 1 20.49 us 20.49 us 20.49 us spinlock __softirqentry_text_start+0xeb $ sudo ./perf lock con -L ffff9f4140059000 contended total wait max wait avg wait type caller 38 779.40 us 42.83 us 20.51 us spinlock worker_thread+0x50 11 216.30 us 39.87 us 19.66 us spinlock queue_work_on+0x39 8 118.13 us 20.51 us 14.77 us spinlock kthread+0xe5 Committer testing: # uname -a Linux quaco 6.0.12-200.fc36.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Dec 8 17:15:53 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux # perf lock record ^C[ perf record: Woken up 1 times to write data ] # perf lock con -L jiffies_lock,rcu_state contended total wait max wait avg wait type caller # perf lock con contended total wait max wait avg wait type caller 1 9.06 us 9.06 us 9.06 us spinlock call_timer_fn+0x24 # perf lock con -L call ignore unknown symbol: call contended total wait max wait avg wait type caller 1 9.06 us 9.06 us 9.06 us spinlock call_timer_fn+0x24 # Signed-off-by: Namhyung Kim <namhyung@kernel.org> Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Blake Jones <blakejones@google.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Song Liu <song@kernel.org> Cc: bpf@vger.kernel.org Link: https://lore.kernel.org/r/20221219201732.460111-5-namhyung@kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2022-12-19 20:17:30 +00:00
-L::
--lock-filter=<value>::
Show lock contention only for given lock addresses or names (comma separated list).
-S::
--callstack-filter=<value>::
Show lock contention only if the callstack contains the given string.
Note that it matches the substring so 'rq' would match both 'raw_spin_rq_lock'
and 'irq_enter_rcu'.
SEE ALSO
--------
linkperf:perf[1]