perf test: Initial regression testing command
First an example with the first internal test:
[acme@doppio linux-2.6-tip]$ perf test
1: vmlinux symtab matches kallsyms: Ok
So it run just one test, that is "vmlinux symtab matches kallsyms", and it was
successful.
If we run it in verbose mode, we'll see details about errors and extra warnings
for non-fatal problems:
[acme@doppio linux-2.6-tip]$ perf test -v
1: vmlinux symtab matches kallsyms:
--- start ---
Looking at the vmlinux_path (5 entries long)
No build_id in vmlinux, ignoring it
No build_id in /boot/vmlinux, ignoring it
No build_id in /boot/vmlinux-2.6.34-rc4-tip+, ignoring it
Using /lib/modules/2.6.34-rc4-tip+/build/vmlinux for symbols
Maps only in vmlinux:
ffffffff81cb81b1-ffffffff81e1149b 0 [kernel].init.text
ffffffff81e1149c-ffffffff9fffffff 0 [kernel].exit.text
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2
Maps in vmlinux with a different name in kallsyms:
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0 in kallsyms as [kernel].0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn in kallsyms as:
*ffffffffff600100-ffffffffff60012f 0 [kernel].2
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1 in kallsyms as [kernel].6
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2 in kallsyms as [kernel].8
Maps only in kallsyms:
ffffffffff600130-ffffffffff6003ff 0 [kernel].4
---- end ----
vmlinux symtab matches kallsyms: Ok
[acme@doppio linux-2.6-tip]$
In the above case we only know the name of the non contiguous kernel ranges in
the address space when reading the symbol information from the ELF symtab in
vmlinux.
The /proc/kallsyms file lack this, we only notice they are separate because
there are modules after the kernel and after that more kernel functions, so we
need to have a module rbtree backed by the module .ko path to get symtabs in
the vmlinux case.
The tool uses it to match by address to emit appropriate warning, but don't
considers this fatal.
The .init.text and .exit.text ines, of course, aren't in kallsyms, so I left
these cases just as extra info in verbose mode.
The end of the sections also aren't in kallsyms, so we the symbols layer does
another pass and sets the end addresses as the next map start minus one, which
sometimes pads, causing harmless mismatches.
But at least the symbols match, tested it by copying /proc/kallsyms to
/tmp/kallsyms and doing changes to see if they were detected.
This first test also should serve as a first stab at documenting the
symbol library by providing a self contained example that exercises it
together with comments about what is being done.
More tests to check if actions done on a monitored app, like doing mmaps, etc,
makes the kernel generate the expected events should be added next.
Cc: Frédéric Weisbecker <fweisbec@gmail.com>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <new-submission>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2010-04-29 21:58:32 +00:00
|
|
|
/*
|
|
|
|
* builtin-test.c
|
|
|
|
*
|
|
|
|
* Builtin regression testing command: ever growing number of sanity tests
|
|
|
|
*/
|
|
|
|
#include "builtin.h"
|
|
|
|
|
|
|
|
#include "util/cache.h"
|
|
|
|
#include "util/debug.h"
|
2011-01-15 12:42:46 +00:00
|
|
|
#include "util/evlist.h"
|
perf test: Initial regression testing command
First an example with the first internal test:
[acme@doppio linux-2.6-tip]$ perf test
1: vmlinux symtab matches kallsyms: Ok
So it run just one test, that is "vmlinux symtab matches kallsyms", and it was
successful.
If we run it in verbose mode, we'll see details about errors and extra warnings
for non-fatal problems:
[acme@doppio linux-2.6-tip]$ perf test -v
1: vmlinux symtab matches kallsyms:
--- start ---
Looking at the vmlinux_path (5 entries long)
No build_id in vmlinux, ignoring it
No build_id in /boot/vmlinux, ignoring it
No build_id in /boot/vmlinux-2.6.34-rc4-tip+, ignoring it
Using /lib/modules/2.6.34-rc4-tip+/build/vmlinux for symbols
Maps only in vmlinux:
ffffffff81cb81b1-ffffffff81e1149b 0 [kernel].init.text
ffffffff81e1149c-ffffffff9fffffff 0 [kernel].exit.text
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2
Maps in vmlinux with a different name in kallsyms:
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0 in kallsyms as [kernel].0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn in kallsyms as:
*ffffffffff600100-ffffffffff60012f 0 [kernel].2
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1 in kallsyms as [kernel].6
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2 in kallsyms as [kernel].8
Maps only in kallsyms:
ffffffffff600130-ffffffffff6003ff 0 [kernel].4
---- end ----
vmlinux symtab matches kallsyms: Ok
[acme@doppio linux-2.6-tip]$
In the above case we only know the name of the non contiguous kernel ranges in
the address space when reading the symbol information from the ELF symtab in
vmlinux.
The /proc/kallsyms file lack this, we only notice they are separate because
there are modules after the kernel and after that more kernel functions, so we
need to have a module rbtree backed by the module .ko path to get symtabs in
the vmlinux case.
The tool uses it to match by address to emit appropriate warning, but don't
considers this fatal.
The .init.text and .exit.text ines, of course, aren't in kallsyms, so I left
these cases just as extra info in verbose mode.
The end of the sections also aren't in kallsyms, so we the symbols layer does
another pass and sets the end addresses as the next map start minus one, which
sometimes pads, causing harmless mismatches.
But at least the symbols match, tested it by copying /proc/kallsyms to
/tmp/kallsyms and doing changes to see if they were detected.
This first test also should serve as a first stab at documenting the
symbol library by providing a self contained example that exercises it
together with comments about what is being done.
More tests to check if actions done on a monitored app, like doing mmaps, etc,
makes the kernel generate the expected events should be added next.
Cc: Frédéric Weisbecker <fweisbec@gmail.com>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <new-submission>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2010-04-29 21:58:32 +00:00
|
|
|
#include "util/parse-options.h"
|
2011-01-15 12:42:46 +00:00
|
|
|
#include "util/parse-events.h"
|
perf test: Initial regression testing command
First an example with the first internal test:
[acme@doppio linux-2.6-tip]$ perf test
1: vmlinux symtab matches kallsyms: Ok
So it run just one test, that is "vmlinux symtab matches kallsyms", and it was
successful.
If we run it in verbose mode, we'll see details about errors and extra warnings
for non-fatal problems:
[acme@doppio linux-2.6-tip]$ perf test -v
1: vmlinux symtab matches kallsyms:
--- start ---
Looking at the vmlinux_path (5 entries long)
No build_id in vmlinux, ignoring it
No build_id in /boot/vmlinux, ignoring it
No build_id in /boot/vmlinux-2.6.34-rc4-tip+, ignoring it
Using /lib/modules/2.6.34-rc4-tip+/build/vmlinux for symbols
Maps only in vmlinux:
ffffffff81cb81b1-ffffffff81e1149b 0 [kernel].init.text
ffffffff81e1149c-ffffffff9fffffff 0 [kernel].exit.text
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2
Maps in vmlinux with a different name in kallsyms:
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0 in kallsyms as [kernel].0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn in kallsyms as:
*ffffffffff600100-ffffffffff60012f 0 [kernel].2
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1 in kallsyms as [kernel].6
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2 in kallsyms as [kernel].8
Maps only in kallsyms:
ffffffffff600130-ffffffffff6003ff 0 [kernel].4
---- end ----
vmlinux symtab matches kallsyms: Ok
[acme@doppio linux-2.6-tip]$
In the above case we only know the name of the non contiguous kernel ranges in
the address space when reading the symbol information from the ELF symtab in
vmlinux.
The /proc/kallsyms file lack this, we only notice they are separate because
there are modules after the kernel and after that more kernel functions, so we
need to have a module rbtree backed by the module .ko path to get symtabs in
the vmlinux case.
The tool uses it to match by address to emit appropriate warning, but don't
considers this fatal.
The .init.text and .exit.text ines, of course, aren't in kallsyms, so I left
these cases just as extra info in verbose mode.
The end of the sections also aren't in kallsyms, so we the symbols layer does
another pass and sets the end addresses as the next map start minus one, which
sometimes pads, causing harmless mismatches.
But at least the symbols match, tested it by copying /proc/kallsyms to
/tmp/kallsyms and doing changes to see if they were detected.
This first test also should serve as a first stab at documenting the
symbol library by providing a self contained example that exercises it
together with comments about what is being done.
More tests to check if actions done on a monitored app, like doing mmaps, etc,
makes the kernel generate the expected events should be added next.
Cc: Frédéric Weisbecker <fweisbec@gmail.com>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <new-submission>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2010-04-29 21:58:32 +00:00
|
|
|
#include "util/symbol.h"
|
2011-01-18 17:15:24 +00:00
|
|
|
#include "util/thread_map.h"
|
perf test: Initial regression testing command
First an example with the first internal test:
[acme@doppio linux-2.6-tip]$ perf test
1: vmlinux symtab matches kallsyms: Ok
So it run just one test, that is "vmlinux symtab matches kallsyms", and it was
successful.
If we run it in verbose mode, we'll see details about errors and extra warnings
for non-fatal problems:
[acme@doppio linux-2.6-tip]$ perf test -v
1: vmlinux symtab matches kallsyms:
--- start ---
Looking at the vmlinux_path (5 entries long)
No build_id in vmlinux, ignoring it
No build_id in /boot/vmlinux, ignoring it
No build_id in /boot/vmlinux-2.6.34-rc4-tip+, ignoring it
Using /lib/modules/2.6.34-rc4-tip+/build/vmlinux for symbols
Maps only in vmlinux:
ffffffff81cb81b1-ffffffff81e1149b 0 [kernel].init.text
ffffffff81e1149c-ffffffff9fffffff 0 [kernel].exit.text
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2
Maps in vmlinux with a different name in kallsyms:
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0 in kallsyms as [kernel].0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn in kallsyms as:
*ffffffffff600100-ffffffffff60012f 0 [kernel].2
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1 in kallsyms as [kernel].6
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2 in kallsyms as [kernel].8
Maps only in kallsyms:
ffffffffff600130-ffffffffff6003ff 0 [kernel].4
---- end ----
vmlinux symtab matches kallsyms: Ok
[acme@doppio linux-2.6-tip]$
In the above case we only know the name of the non contiguous kernel ranges in
the address space when reading the symbol information from the ELF symtab in
vmlinux.
The /proc/kallsyms file lack this, we only notice they are separate because
there are modules after the kernel and after that more kernel functions, so we
need to have a module rbtree backed by the module .ko path to get symtabs in
the vmlinux case.
The tool uses it to match by address to emit appropriate warning, but don't
considers this fatal.
The .init.text and .exit.text ines, of course, aren't in kallsyms, so I left
these cases just as extra info in verbose mode.
The end of the sections also aren't in kallsyms, so we the symbols layer does
another pass and sets the end addresses as the next map start minus one, which
sometimes pads, causing harmless mismatches.
But at least the symbols match, tested it by copying /proc/kallsyms to
/tmp/kallsyms and doing changes to see if they were detected.
This first test also should serve as a first stab at documenting the
symbol library by providing a self contained example that exercises it
together with comments about what is being done.
More tests to check if actions done on a monitored app, like doing mmaps, etc,
makes the kernel generate the expected events should be added next.
Cc: Frédéric Weisbecker <fweisbec@gmail.com>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <new-submission>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2010-04-29 21:58:32 +00:00
|
|
|
|
|
|
|
static long page_size;
|
|
|
|
|
|
|
|
static int vmlinux_matches_kallsyms_filter(struct map *map __used, struct symbol *sym)
|
|
|
|
{
|
|
|
|
bool *visited = symbol__priv(sym);
|
|
|
|
*visited = true;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int test__vmlinux_matches_kallsyms(void)
|
|
|
|
{
|
|
|
|
int err = -1;
|
|
|
|
struct rb_node *nd;
|
|
|
|
struct symbol *sym;
|
|
|
|
struct map *kallsyms_map, *vmlinux_map;
|
|
|
|
struct machine kallsyms, vmlinux;
|
|
|
|
enum map_type type = MAP__FUNCTION;
|
|
|
|
struct ref_reloc_sym ref_reloc_sym = { .name = "_stext", };
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Step 1:
|
|
|
|
*
|
|
|
|
* Init the machines that will hold kernel, modules obtained from
|
|
|
|
* both vmlinux + .ko files and from /proc/kallsyms split by modules.
|
|
|
|
*/
|
|
|
|
machine__init(&kallsyms, "", HOST_KERNEL_ID);
|
|
|
|
machine__init(&vmlinux, "", HOST_KERNEL_ID);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Step 2:
|
|
|
|
*
|
|
|
|
* Create the kernel maps for kallsyms and the DSO where we will then
|
|
|
|
* load /proc/kallsyms. Also create the modules maps from /proc/modules
|
|
|
|
* and find the .ko files that match them in /lib/modules/`uname -r`/.
|
|
|
|
*/
|
|
|
|
if (machine__create_kernel_maps(&kallsyms) < 0) {
|
|
|
|
pr_debug("machine__create_kernel_maps ");
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Step 3:
|
|
|
|
*
|
|
|
|
* Load and split /proc/kallsyms into multiple maps, one per module.
|
|
|
|
*/
|
|
|
|
if (machine__load_kallsyms(&kallsyms, "/proc/kallsyms", type, NULL) <= 0) {
|
|
|
|
pr_debug("dso__load_kallsyms ");
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Step 4:
|
|
|
|
*
|
|
|
|
* kallsyms will be internally on demand sorted by name so that we can
|
|
|
|
* find the reference relocation * symbol, i.e. the symbol we will use
|
|
|
|
* to see if the running kernel was relocated by checking if it has the
|
|
|
|
* same value in the vmlinux file we load.
|
|
|
|
*/
|
|
|
|
kallsyms_map = machine__kernel_map(&kallsyms, type);
|
|
|
|
|
|
|
|
sym = map__find_symbol_by_name(kallsyms_map, ref_reloc_sym.name, NULL);
|
|
|
|
if (sym == NULL) {
|
|
|
|
pr_debug("dso__find_symbol_by_name ");
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
ref_reloc_sym.addr = sym->start;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Step 5:
|
|
|
|
*
|
|
|
|
* Now repeat step 2, this time for the vmlinux file we'll auto-locate.
|
|
|
|
*/
|
|
|
|
if (machine__create_kernel_maps(&vmlinux) < 0) {
|
|
|
|
pr_debug("machine__create_kernel_maps ");
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
vmlinux_map = machine__kernel_map(&vmlinux, type);
|
|
|
|
map__kmap(vmlinux_map)->ref_reloc_sym = &ref_reloc_sym;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Step 6:
|
|
|
|
*
|
|
|
|
* Locate a vmlinux file in the vmlinux path that has a buildid that
|
|
|
|
* matches the one of the running kernel.
|
|
|
|
*
|
|
|
|
* While doing that look if we find the ref reloc symbol, if we find it
|
|
|
|
* we'll have its ref_reloc_symbol.unrelocated_addr and then
|
|
|
|
* maps__reloc_vmlinux will notice and set proper ->[un]map_ip routines
|
|
|
|
* to fixup the symbols.
|
|
|
|
*/
|
|
|
|
if (machine__load_vmlinux_path(&vmlinux, type,
|
|
|
|
vmlinux_matches_kallsyms_filter) <= 0) {
|
|
|
|
pr_debug("machine__load_vmlinux_path ");
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
err = 0;
|
|
|
|
/*
|
|
|
|
* Step 7:
|
|
|
|
*
|
|
|
|
* Now look at the symbols in the vmlinux DSO and check if we find all of them
|
|
|
|
* in the kallsyms dso. For the ones that are in both, check its names and
|
|
|
|
* end addresses too.
|
|
|
|
*/
|
|
|
|
for (nd = rb_first(&vmlinux_map->dso->symbols[type]); nd; nd = rb_next(nd)) {
|
2010-12-22 01:38:37 +00:00
|
|
|
struct symbol *pair, *first_pair;
|
|
|
|
bool backwards = true;
|
perf test: Initial regression testing command
First an example with the first internal test:
[acme@doppio linux-2.6-tip]$ perf test
1: vmlinux symtab matches kallsyms: Ok
So it run just one test, that is "vmlinux symtab matches kallsyms", and it was
successful.
If we run it in verbose mode, we'll see details about errors and extra warnings
for non-fatal problems:
[acme@doppio linux-2.6-tip]$ perf test -v
1: vmlinux symtab matches kallsyms:
--- start ---
Looking at the vmlinux_path (5 entries long)
No build_id in vmlinux, ignoring it
No build_id in /boot/vmlinux, ignoring it
No build_id in /boot/vmlinux-2.6.34-rc4-tip+, ignoring it
Using /lib/modules/2.6.34-rc4-tip+/build/vmlinux for symbols
Maps only in vmlinux:
ffffffff81cb81b1-ffffffff81e1149b 0 [kernel].init.text
ffffffff81e1149c-ffffffff9fffffff 0 [kernel].exit.text
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2
Maps in vmlinux with a different name in kallsyms:
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0 in kallsyms as [kernel].0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn in kallsyms as:
*ffffffffff600100-ffffffffff60012f 0 [kernel].2
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1 in kallsyms as [kernel].6
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2 in kallsyms as [kernel].8
Maps only in kallsyms:
ffffffffff600130-ffffffffff6003ff 0 [kernel].4
---- end ----
vmlinux symtab matches kallsyms: Ok
[acme@doppio linux-2.6-tip]$
In the above case we only know the name of the non contiguous kernel ranges in
the address space when reading the symbol information from the ELF symtab in
vmlinux.
The /proc/kallsyms file lack this, we only notice they are separate because
there are modules after the kernel and after that more kernel functions, so we
need to have a module rbtree backed by the module .ko path to get symtabs in
the vmlinux case.
The tool uses it to match by address to emit appropriate warning, but don't
considers this fatal.
The .init.text and .exit.text ines, of course, aren't in kallsyms, so I left
these cases just as extra info in verbose mode.
The end of the sections also aren't in kallsyms, so we the symbols layer does
another pass and sets the end addresses as the next map start minus one, which
sometimes pads, causing harmless mismatches.
But at least the symbols match, tested it by copying /proc/kallsyms to
/tmp/kallsyms and doing changes to see if they were detected.
This first test also should serve as a first stab at documenting the
symbol library by providing a self contained example that exercises it
together with comments about what is being done.
More tests to check if actions done on a monitored app, like doing mmaps, etc,
makes the kernel generate the expected events should be added next.
Cc: Frédéric Weisbecker <fweisbec@gmail.com>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <new-submission>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2010-04-29 21:58:32 +00:00
|
|
|
|
|
|
|
sym = rb_entry(nd, struct symbol, rb_node);
|
2010-12-22 01:38:37 +00:00
|
|
|
|
|
|
|
if (sym->start == sym->end)
|
|
|
|
continue;
|
|
|
|
|
|
|
|
first_pair = machine__find_kernel_symbol(&kallsyms, type, sym->start, NULL, NULL);
|
|
|
|
pair = first_pair;
|
perf test: Initial regression testing command
First an example with the first internal test:
[acme@doppio linux-2.6-tip]$ perf test
1: vmlinux symtab matches kallsyms: Ok
So it run just one test, that is "vmlinux symtab matches kallsyms", and it was
successful.
If we run it in verbose mode, we'll see details about errors and extra warnings
for non-fatal problems:
[acme@doppio linux-2.6-tip]$ perf test -v
1: vmlinux symtab matches kallsyms:
--- start ---
Looking at the vmlinux_path (5 entries long)
No build_id in vmlinux, ignoring it
No build_id in /boot/vmlinux, ignoring it
No build_id in /boot/vmlinux-2.6.34-rc4-tip+, ignoring it
Using /lib/modules/2.6.34-rc4-tip+/build/vmlinux for symbols
Maps only in vmlinux:
ffffffff81cb81b1-ffffffff81e1149b 0 [kernel].init.text
ffffffff81e1149c-ffffffff9fffffff 0 [kernel].exit.text
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2
Maps in vmlinux with a different name in kallsyms:
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0 in kallsyms as [kernel].0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn in kallsyms as:
*ffffffffff600100-ffffffffff60012f 0 [kernel].2
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1 in kallsyms as [kernel].6
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2 in kallsyms as [kernel].8
Maps only in kallsyms:
ffffffffff600130-ffffffffff6003ff 0 [kernel].4
---- end ----
vmlinux symtab matches kallsyms: Ok
[acme@doppio linux-2.6-tip]$
In the above case we only know the name of the non contiguous kernel ranges in
the address space when reading the symbol information from the ELF symtab in
vmlinux.
The /proc/kallsyms file lack this, we only notice they are separate because
there are modules after the kernel and after that more kernel functions, so we
need to have a module rbtree backed by the module .ko path to get symtabs in
the vmlinux case.
The tool uses it to match by address to emit appropriate warning, but don't
considers this fatal.
The .init.text and .exit.text ines, of course, aren't in kallsyms, so I left
these cases just as extra info in verbose mode.
The end of the sections also aren't in kallsyms, so we the symbols layer does
another pass and sets the end addresses as the next map start minus one, which
sometimes pads, causing harmless mismatches.
But at least the symbols match, tested it by copying /proc/kallsyms to
/tmp/kallsyms and doing changes to see if they were detected.
This first test also should serve as a first stab at documenting the
symbol library by providing a self contained example that exercises it
together with comments about what is being done.
More tests to check if actions done on a monitored app, like doing mmaps, etc,
makes the kernel generate the expected events should be added next.
Cc: Frédéric Weisbecker <fweisbec@gmail.com>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <new-submission>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2010-04-29 21:58:32 +00:00
|
|
|
|
|
|
|
if (pair && pair->start == sym->start) {
|
|
|
|
next_pair:
|
|
|
|
if (strcmp(sym->name, pair->name) == 0) {
|
|
|
|
/*
|
|
|
|
* kallsyms don't have the symbol end, so we
|
|
|
|
* set that by using the next symbol start - 1,
|
|
|
|
* in some cases we get this up to a page
|
|
|
|
* wrong, trace_kmalloc when I was developing
|
|
|
|
* this code was one such example, 2106 bytes
|
|
|
|
* off the real size. More than that and we
|
|
|
|
* _really_ have a problem.
|
|
|
|
*/
|
|
|
|
s64 skew = sym->end - pair->end;
|
|
|
|
if (llabs(skew) < page_size)
|
|
|
|
continue;
|
|
|
|
|
2011-01-22 22:37:02 +00:00
|
|
|
pr_debug("%#" PRIx64 ": diff end addr for %s v: %#" PRIx64 " k: %#" PRIx64 "\n",
|
perf test: Initial regression testing command
First an example with the first internal test:
[acme@doppio linux-2.6-tip]$ perf test
1: vmlinux symtab matches kallsyms: Ok
So it run just one test, that is "vmlinux symtab matches kallsyms", and it was
successful.
If we run it in verbose mode, we'll see details about errors and extra warnings
for non-fatal problems:
[acme@doppio linux-2.6-tip]$ perf test -v
1: vmlinux symtab matches kallsyms:
--- start ---
Looking at the vmlinux_path (5 entries long)
No build_id in vmlinux, ignoring it
No build_id in /boot/vmlinux, ignoring it
No build_id in /boot/vmlinux-2.6.34-rc4-tip+, ignoring it
Using /lib/modules/2.6.34-rc4-tip+/build/vmlinux for symbols
Maps only in vmlinux:
ffffffff81cb81b1-ffffffff81e1149b 0 [kernel].init.text
ffffffff81e1149c-ffffffff9fffffff 0 [kernel].exit.text
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2
Maps in vmlinux with a different name in kallsyms:
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0 in kallsyms as [kernel].0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn in kallsyms as:
*ffffffffff600100-ffffffffff60012f 0 [kernel].2
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1 in kallsyms as [kernel].6
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2 in kallsyms as [kernel].8
Maps only in kallsyms:
ffffffffff600130-ffffffffff6003ff 0 [kernel].4
---- end ----
vmlinux symtab matches kallsyms: Ok
[acme@doppio linux-2.6-tip]$
In the above case we only know the name of the non contiguous kernel ranges in
the address space when reading the symbol information from the ELF symtab in
vmlinux.
The /proc/kallsyms file lack this, we only notice they are separate because
there are modules after the kernel and after that more kernel functions, so we
need to have a module rbtree backed by the module .ko path to get symtabs in
the vmlinux case.
The tool uses it to match by address to emit appropriate warning, but don't
considers this fatal.
The .init.text and .exit.text ines, of course, aren't in kallsyms, so I left
these cases just as extra info in verbose mode.
The end of the sections also aren't in kallsyms, so we the symbols layer does
another pass and sets the end addresses as the next map start minus one, which
sometimes pads, causing harmless mismatches.
But at least the symbols match, tested it by copying /proc/kallsyms to
/tmp/kallsyms and doing changes to see if they were detected.
This first test also should serve as a first stab at documenting the
symbol library by providing a self contained example that exercises it
together with comments about what is being done.
More tests to check if actions done on a monitored app, like doing mmaps, etc,
makes the kernel generate the expected events should be added next.
Cc: Frédéric Weisbecker <fweisbec@gmail.com>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <new-submission>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2010-04-29 21:58:32 +00:00
|
|
|
sym->start, sym->name, sym->end, pair->end);
|
|
|
|
} else {
|
2010-12-22 01:38:37 +00:00
|
|
|
struct rb_node *nnd;
|
|
|
|
detour:
|
|
|
|
nnd = backwards ? rb_prev(&pair->rb_node) :
|
|
|
|
rb_next(&pair->rb_node);
|
perf test: Initial regression testing command
First an example with the first internal test:
[acme@doppio linux-2.6-tip]$ perf test
1: vmlinux symtab matches kallsyms: Ok
So it run just one test, that is "vmlinux symtab matches kallsyms", and it was
successful.
If we run it in verbose mode, we'll see details about errors and extra warnings
for non-fatal problems:
[acme@doppio linux-2.6-tip]$ perf test -v
1: vmlinux symtab matches kallsyms:
--- start ---
Looking at the vmlinux_path (5 entries long)
No build_id in vmlinux, ignoring it
No build_id in /boot/vmlinux, ignoring it
No build_id in /boot/vmlinux-2.6.34-rc4-tip+, ignoring it
Using /lib/modules/2.6.34-rc4-tip+/build/vmlinux for symbols
Maps only in vmlinux:
ffffffff81cb81b1-ffffffff81e1149b 0 [kernel].init.text
ffffffff81e1149c-ffffffff9fffffff 0 [kernel].exit.text
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2
Maps in vmlinux with a different name in kallsyms:
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0 in kallsyms as [kernel].0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn in kallsyms as:
*ffffffffff600100-ffffffffff60012f 0 [kernel].2
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1 in kallsyms as [kernel].6
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2 in kallsyms as [kernel].8
Maps only in kallsyms:
ffffffffff600130-ffffffffff6003ff 0 [kernel].4
---- end ----
vmlinux symtab matches kallsyms: Ok
[acme@doppio linux-2.6-tip]$
In the above case we only know the name of the non contiguous kernel ranges in
the address space when reading the symbol information from the ELF symtab in
vmlinux.
The /proc/kallsyms file lack this, we only notice they are separate because
there are modules after the kernel and after that more kernel functions, so we
need to have a module rbtree backed by the module .ko path to get symtabs in
the vmlinux case.
The tool uses it to match by address to emit appropriate warning, but don't
considers this fatal.
The .init.text and .exit.text ines, of course, aren't in kallsyms, so I left
these cases just as extra info in verbose mode.
The end of the sections also aren't in kallsyms, so we the symbols layer does
another pass and sets the end addresses as the next map start minus one, which
sometimes pads, causing harmless mismatches.
But at least the symbols match, tested it by copying /proc/kallsyms to
/tmp/kallsyms and doing changes to see if they were detected.
This first test also should serve as a first stab at documenting the
symbol library by providing a self contained example that exercises it
together with comments about what is being done.
More tests to check if actions done on a monitored app, like doing mmaps, etc,
makes the kernel generate the expected events should be added next.
Cc: Frédéric Weisbecker <fweisbec@gmail.com>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <new-submission>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2010-04-29 21:58:32 +00:00
|
|
|
if (nnd) {
|
|
|
|
struct symbol *next = rb_entry(nnd, struct symbol, rb_node);
|
|
|
|
|
|
|
|
if (next->start == sym->start) {
|
|
|
|
pair = next;
|
|
|
|
goto next_pair;
|
|
|
|
}
|
|
|
|
}
|
2010-12-22 01:38:37 +00:00
|
|
|
|
|
|
|
if (backwards) {
|
|
|
|
backwards = false;
|
|
|
|
pair = first_pair;
|
|
|
|
goto detour;
|
|
|
|
}
|
|
|
|
|
2011-01-22 22:37:02 +00:00
|
|
|
pr_debug("%#" PRIx64 ": diff name v: %s k: %s\n",
|
perf test: Initial regression testing command
First an example with the first internal test:
[acme@doppio linux-2.6-tip]$ perf test
1: vmlinux symtab matches kallsyms: Ok
So it run just one test, that is "vmlinux symtab matches kallsyms", and it was
successful.
If we run it in verbose mode, we'll see details about errors and extra warnings
for non-fatal problems:
[acme@doppio linux-2.6-tip]$ perf test -v
1: vmlinux symtab matches kallsyms:
--- start ---
Looking at the vmlinux_path (5 entries long)
No build_id in vmlinux, ignoring it
No build_id in /boot/vmlinux, ignoring it
No build_id in /boot/vmlinux-2.6.34-rc4-tip+, ignoring it
Using /lib/modules/2.6.34-rc4-tip+/build/vmlinux for symbols
Maps only in vmlinux:
ffffffff81cb81b1-ffffffff81e1149b 0 [kernel].init.text
ffffffff81e1149c-ffffffff9fffffff 0 [kernel].exit.text
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2
Maps in vmlinux with a different name in kallsyms:
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0 in kallsyms as [kernel].0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn in kallsyms as:
*ffffffffff600100-ffffffffff60012f 0 [kernel].2
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1 in kallsyms as [kernel].6
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2 in kallsyms as [kernel].8
Maps only in kallsyms:
ffffffffff600130-ffffffffff6003ff 0 [kernel].4
---- end ----
vmlinux symtab matches kallsyms: Ok
[acme@doppio linux-2.6-tip]$
In the above case we only know the name of the non contiguous kernel ranges in
the address space when reading the symbol information from the ELF symtab in
vmlinux.
The /proc/kallsyms file lack this, we only notice they are separate because
there are modules after the kernel and after that more kernel functions, so we
need to have a module rbtree backed by the module .ko path to get symtabs in
the vmlinux case.
The tool uses it to match by address to emit appropriate warning, but don't
considers this fatal.
The .init.text and .exit.text ines, of course, aren't in kallsyms, so I left
these cases just as extra info in verbose mode.
The end of the sections also aren't in kallsyms, so we the symbols layer does
another pass and sets the end addresses as the next map start minus one, which
sometimes pads, causing harmless mismatches.
But at least the symbols match, tested it by copying /proc/kallsyms to
/tmp/kallsyms and doing changes to see if they were detected.
This first test also should serve as a first stab at documenting the
symbol library by providing a self contained example that exercises it
together with comments about what is being done.
More tests to check if actions done on a monitored app, like doing mmaps, etc,
makes the kernel generate the expected events should be added next.
Cc: Frédéric Weisbecker <fweisbec@gmail.com>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <new-submission>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2010-04-29 21:58:32 +00:00
|
|
|
sym->start, sym->name, pair->name);
|
|
|
|
}
|
|
|
|
} else
|
2011-01-22 22:37:02 +00:00
|
|
|
pr_debug("%#" PRIx64 ": %s not on kallsyms\n", sym->start, sym->name);
|
perf test: Initial regression testing command
First an example with the first internal test:
[acme@doppio linux-2.6-tip]$ perf test
1: vmlinux symtab matches kallsyms: Ok
So it run just one test, that is "vmlinux symtab matches kallsyms", and it was
successful.
If we run it in verbose mode, we'll see details about errors and extra warnings
for non-fatal problems:
[acme@doppio linux-2.6-tip]$ perf test -v
1: vmlinux symtab matches kallsyms:
--- start ---
Looking at the vmlinux_path (5 entries long)
No build_id in vmlinux, ignoring it
No build_id in /boot/vmlinux, ignoring it
No build_id in /boot/vmlinux-2.6.34-rc4-tip+, ignoring it
Using /lib/modules/2.6.34-rc4-tip+/build/vmlinux for symbols
Maps only in vmlinux:
ffffffff81cb81b1-ffffffff81e1149b 0 [kernel].init.text
ffffffff81e1149c-ffffffff9fffffff 0 [kernel].exit.text
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2
Maps in vmlinux with a different name in kallsyms:
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0 in kallsyms as [kernel].0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn in kallsyms as:
*ffffffffff600100-ffffffffff60012f 0 [kernel].2
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1 in kallsyms as [kernel].6
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2 in kallsyms as [kernel].8
Maps only in kallsyms:
ffffffffff600130-ffffffffff6003ff 0 [kernel].4
---- end ----
vmlinux symtab matches kallsyms: Ok
[acme@doppio linux-2.6-tip]$
In the above case we only know the name of the non contiguous kernel ranges in
the address space when reading the symbol information from the ELF symtab in
vmlinux.
The /proc/kallsyms file lack this, we only notice they are separate because
there are modules after the kernel and after that more kernel functions, so we
need to have a module rbtree backed by the module .ko path to get symtabs in
the vmlinux case.
The tool uses it to match by address to emit appropriate warning, but don't
considers this fatal.
The .init.text and .exit.text ines, of course, aren't in kallsyms, so I left
these cases just as extra info in verbose mode.
The end of the sections also aren't in kallsyms, so we the symbols layer does
another pass and sets the end addresses as the next map start minus one, which
sometimes pads, causing harmless mismatches.
But at least the symbols match, tested it by copying /proc/kallsyms to
/tmp/kallsyms and doing changes to see if they were detected.
This first test also should serve as a first stab at documenting the
symbol library by providing a self contained example that exercises it
together with comments about what is being done.
More tests to check if actions done on a monitored app, like doing mmaps, etc,
makes the kernel generate the expected events should be added next.
Cc: Frédéric Weisbecker <fweisbec@gmail.com>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <new-submission>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2010-04-29 21:58:32 +00:00
|
|
|
|
|
|
|
err = -1;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!verbose)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
pr_info("Maps only in vmlinux:\n");
|
|
|
|
|
|
|
|
for (nd = rb_first(&vmlinux.kmaps.maps[type]); nd; nd = rb_next(nd)) {
|
|
|
|
struct map *pos = rb_entry(nd, struct map, rb_node), *pair;
|
|
|
|
/*
|
|
|
|
* If it is the kernel, kallsyms is always "[kernel.kallsyms]", while
|
|
|
|
* the kernel will have the path for the vmlinux file being used,
|
|
|
|
* so use the short name, less descriptive but the same ("[kernel]" in
|
|
|
|
* both cases.
|
|
|
|
*/
|
|
|
|
pair = map_groups__find_by_name(&kallsyms.kmaps, type,
|
|
|
|
(pos->dso->kernel ?
|
|
|
|
pos->dso->short_name :
|
|
|
|
pos->dso->name));
|
|
|
|
if (pair)
|
|
|
|
pair->priv = 1;
|
|
|
|
else
|
|
|
|
map__fprintf(pos, stderr);
|
|
|
|
}
|
|
|
|
|
|
|
|
pr_info("Maps in vmlinux with a different name in kallsyms:\n");
|
|
|
|
|
|
|
|
for (nd = rb_first(&vmlinux.kmaps.maps[type]); nd; nd = rb_next(nd)) {
|
|
|
|
struct map *pos = rb_entry(nd, struct map, rb_node), *pair;
|
|
|
|
|
|
|
|
pair = map_groups__find(&kallsyms.kmaps, type, pos->start);
|
|
|
|
if (pair == NULL || pair->priv)
|
|
|
|
continue;
|
|
|
|
|
|
|
|
if (pair->start == pos->start) {
|
|
|
|
pair->priv = 1;
|
2011-01-22 22:37:02 +00:00
|
|
|
pr_info(" %" PRIx64 "-%" PRIx64 " %" PRIx64 " %s in kallsyms as",
|
perf test: Initial regression testing command
First an example with the first internal test:
[acme@doppio linux-2.6-tip]$ perf test
1: vmlinux symtab matches kallsyms: Ok
So it run just one test, that is "vmlinux symtab matches kallsyms", and it was
successful.
If we run it in verbose mode, we'll see details about errors and extra warnings
for non-fatal problems:
[acme@doppio linux-2.6-tip]$ perf test -v
1: vmlinux symtab matches kallsyms:
--- start ---
Looking at the vmlinux_path (5 entries long)
No build_id in vmlinux, ignoring it
No build_id in /boot/vmlinux, ignoring it
No build_id in /boot/vmlinux-2.6.34-rc4-tip+, ignoring it
Using /lib/modules/2.6.34-rc4-tip+/build/vmlinux for symbols
Maps only in vmlinux:
ffffffff81cb81b1-ffffffff81e1149b 0 [kernel].init.text
ffffffff81e1149c-ffffffff9fffffff 0 [kernel].exit.text
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2
Maps in vmlinux with a different name in kallsyms:
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0 in kallsyms as [kernel].0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn in kallsyms as:
*ffffffffff600100-ffffffffff60012f 0 [kernel].2
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1 in kallsyms as [kernel].6
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2 in kallsyms as [kernel].8
Maps only in kallsyms:
ffffffffff600130-ffffffffff6003ff 0 [kernel].4
---- end ----
vmlinux symtab matches kallsyms: Ok
[acme@doppio linux-2.6-tip]$
In the above case we only know the name of the non contiguous kernel ranges in
the address space when reading the symbol information from the ELF symtab in
vmlinux.
The /proc/kallsyms file lack this, we only notice they are separate because
there are modules after the kernel and after that more kernel functions, so we
need to have a module rbtree backed by the module .ko path to get symtabs in
the vmlinux case.
The tool uses it to match by address to emit appropriate warning, but don't
considers this fatal.
The .init.text and .exit.text ines, of course, aren't in kallsyms, so I left
these cases just as extra info in verbose mode.
The end of the sections also aren't in kallsyms, so we the symbols layer does
another pass and sets the end addresses as the next map start minus one, which
sometimes pads, causing harmless mismatches.
But at least the symbols match, tested it by copying /proc/kallsyms to
/tmp/kallsyms and doing changes to see if they were detected.
This first test also should serve as a first stab at documenting the
symbol library by providing a self contained example that exercises it
together with comments about what is being done.
More tests to check if actions done on a monitored app, like doing mmaps, etc,
makes the kernel generate the expected events should be added next.
Cc: Frédéric Weisbecker <fweisbec@gmail.com>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <new-submission>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2010-04-29 21:58:32 +00:00
|
|
|
pos->start, pos->end, pos->pgoff, pos->dso->name);
|
|
|
|
if (pos->pgoff != pair->pgoff || pos->end != pair->end)
|
2011-01-22 22:37:02 +00:00
|
|
|
pr_info(": \n*%" PRIx64 "-%" PRIx64 " %" PRIx64 "",
|
perf test: Initial regression testing command
First an example with the first internal test:
[acme@doppio linux-2.6-tip]$ perf test
1: vmlinux symtab matches kallsyms: Ok
So it run just one test, that is "vmlinux symtab matches kallsyms", and it was
successful.
If we run it in verbose mode, we'll see details about errors and extra warnings
for non-fatal problems:
[acme@doppio linux-2.6-tip]$ perf test -v
1: vmlinux symtab matches kallsyms:
--- start ---
Looking at the vmlinux_path (5 entries long)
No build_id in vmlinux, ignoring it
No build_id in /boot/vmlinux, ignoring it
No build_id in /boot/vmlinux-2.6.34-rc4-tip+, ignoring it
Using /lib/modules/2.6.34-rc4-tip+/build/vmlinux for symbols
Maps only in vmlinux:
ffffffff81cb81b1-ffffffff81e1149b 0 [kernel].init.text
ffffffff81e1149c-ffffffff9fffffff 0 [kernel].exit.text
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2
Maps in vmlinux with a different name in kallsyms:
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0 in kallsyms as [kernel].0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn in kallsyms as:
*ffffffffff600100-ffffffffff60012f 0 [kernel].2
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1 in kallsyms as [kernel].6
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2 in kallsyms as [kernel].8
Maps only in kallsyms:
ffffffffff600130-ffffffffff6003ff 0 [kernel].4
---- end ----
vmlinux symtab matches kallsyms: Ok
[acme@doppio linux-2.6-tip]$
In the above case we only know the name of the non contiguous kernel ranges in
the address space when reading the symbol information from the ELF symtab in
vmlinux.
The /proc/kallsyms file lack this, we only notice they are separate because
there are modules after the kernel and after that more kernel functions, so we
need to have a module rbtree backed by the module .ko path to get symtabs in
the vmlinux case.
The tool uses it to match by address to emit appropriate warning, but don't
considers this fatal.
The .init.text and .exit.text ines, of course, aren't in kallsyms, so I left
these cases just as extra info in verbose mode.
The end of the sections also aren't in kallsyms, so we the symbols layer does
another pass and sets the end addresses as the next map start minus one, which
sometimes pads, causing harmless mismatches.
But at least the symbols match, tested it by copying /proc/kallsyms to
/tmp/kallsyms and doing changes to see if they were detected.
This first test also should serve as a first stab at documenting the
symbol library by providing a self contained example that exercises it
together with comments about what is being done.
More tests to check if actions done on a monitored app, like doing mmaps, etc,
makes the kernel generate the expected events should be added next.
Cc: Frédéric Weisbecker <fweisbec@gmail.com>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <new-submission>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2010-04-29 21:58:32 +00:00
|
|
|
pair->start, pair->end, pair->pgoff);
|
|
|
|
pr_info(" %s\n", pair->dso->name);
|
|
|
|
pair->priv = 1;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
pr_info("Maps only in kallsyms:\n");
|
|
|
|
|
|
|
|
for (nd = rb_first(&kallsyms.kmaps.maps[type]);
|
|
|
|
nd; nd = rb_next(nd)) {
|
|
|
|
struct map *pos = rb_entry(nd, struct map, rb_node);
|
|
|
|
|
|
|
|
if (!pos->priv)
|
|
|
|
map__fprintf(pos, stderr);
|
|
|
|
}
|
|
|
|
out:
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
2011-01-04 13:55:27 +00:00
|
|
|
#include "util/cpumap.h"
|
2011-01-04 02:16:20 +00:00
|
|
|
#include "util/evsel.h"
|
|
|
|
#include <sys/types.h>
|
|
|
|
|
2011-01-15 12:42:46 +00:00
|
|
|
static int trace_event__id(const char *evname)
|
2011-01-04 02:16:20 +00:00
|
|
|
{
|
|
|
|
char *filename;
|
|
|
|
int err = -1, fd;
|
|
|
|
|
|
|
|
if (asprintf(&filename,
|
|
|
|
"/sys/kernel/debug/tracing/events/syscalls/%s/id",
|
2011-01-15 12:42:46 +00:00
|
|
|
evname) < 0)
|
2011-01-04 02:16:20 +00:00
|
|
|
return -1;
|
|
|
|
|
|
|
|
fd = open(filename, O_RDONLY);
|
|
|
|
if (fd >= 0) {
|
|
|
|
char id[16];
|
|
|
|
if (read(fd, id, sizeof(id)) > 0)
|
|
|
|
err = atoi(id);
|
|
|
|
close(fd);
|
|
|
|
}
|
|
|
|
|
|
|
|
free(filename);
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int test__open_syscall_event(void)
|
|
|
|
{
|
|
|
|
int err = -1, fd;
|
|
|
|
struct thread_map *threads;
|
|
|
|
struct perf_evsel *evsel;
|
2011-01-07 03:11:09 +00:00
|
|
|
struct perf_event_attr attr;
|
2011-01-04 02:16:20 +00:00
|
|
|
unsigned int nr_open_calls = 111, i;
|
|
|
|
int id = trace_event__id("sys_enter_open");
|
|
|
|
|
|
|
|
if (id < 0) {
|
2011-01-04 12:40:08 +00:00
|
|
|
pr_debug("is debugfs mounted on /sys/kernel/debug?\n");
|
2011-01-04 02:16:20 +00:00
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
|
|
|
threads = thread_map__new(-1, getpid());
|
|
|
|
if (threads == NULL) {
|
2011-01-04 12:40:08 +00:00
|
|
|
pr_debug("thread_map__new\n");
|
2011-01-04 02:16:20 +00:00
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
2011-01-07 03:11:09 +00:00
|
|
|
memset(&attr, 0, sizeof(attr));
|
|
|
|
attr.type = PERF_TYPE_TRACEPOINT;
|
|
|
|
attr.config = id;
|
|
|
|
evsel = perf_evsel__new(&attr, 0);
|
2011-01-04 02:16:20 +00:00
|
|
|
if (evsel == NULL) {
|
2011-01-04 12:40:08 +00:00
|
|
|
pr_debug("perf_evsel__new\n");
|
2011-01-04 02:16:20 +00:00
|
|
|
goto out_thread_map_delete;
|
|
|
|
}
|
|
|
|
|
2011-04-14 14:20:14 +00:00
|
|
|
if (perf_evsel__open_per_thread(evsel, threads, false) < 0) {
|
2011-01-04 12:40:08 +00:00
|
|
|
pr_debug("failed to open counter: %s, "
|
|
|
|
"tweak /proc/sys/kernel/perf_event_paranoid?\n",
|
|
|
|
strerror(errno));
|
2011-01-04 02:16:20 +00:00
|
|
|
goto out_evsel_delete;
|
|
|
|
}
|
|
|
|
|
|
|
|
for (i = 0; i < nr_open_calls; ++i) {
|
|
|
|
fd = open("/etc/passwd", O_RDONLY);
|
|
|
|
close(fd);
|
|
|
|
}
|
|
|
|
|
|
|
|
if (perf_evsel__read_on_cpu(evsel, 0, 0) < 0) {
|
2011-04-14 14:20:14 +00:00
|
|
|
pr_debug("perf_evsel__read_on_cpu\n");
|
2011-01-04 02:16:20 +00:00
|
|
|
goto out_close_fd;
|
|
|
|
}
|
|
|
|
|
2011-01-04 12:40:08 +00:00
|
|
|
if (evsel->counts->cpu[0].val != nr_open_calls) {
|
2011-01-22 22:37:02 +00:00
|
|
|
pr_debug("perf_evsel__read_on_cpu: expected to intercept %d calls, got %" PRIu64 "\n",
|
2011-01-04 02:16:20 +00:00
|
|
|
nr_open_calls, evsel->counts->cpu[0].val);
|
2011-01-04 12:40:08 +00:00
|
|
|
goto out_close_fd;
|
|
|
|
}
|
2011-01-04 02:16:20 +00:00
|
|
|
|
|
|
|
err = 0;
|
|
|
|
out_close_fd:
|
|
|
|
perf_evsel__close_fd(evsel, 1, threads->nr);
|
|
|
|
out_evsel_delete:
|
|
|
|
perf_evsel__delete(evsel);
|
|
|
|
out_thread_map_delete:
|
|
|
|
thread_map__delete(threads);
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
2011-01-04 13:55:27 +00:00
|
|
|
#include <sched.h>
|
|
|
|
|
|
|
|
static int test__open_syscall_event_on_all_cpus(void)
|
|
|
|
{
|
|
|
|
int err = -1, fd, cpu;
|
|
|
|
struct thread_map *threads;
|
|
|
|
struct cpu_map *cpus;
|
|
|
|
struct perf_evsel *evsel;
|
|
|
|
struct perf_event_attr attr;
|
|
|
|
unsigned int nr_open_calls = 111, i;
|
2011-01-23 01:14:20 +00:00
|
|
|
cpu_set_t cpu_set;
|
2011-01-04 13:55:27 +00:00
|
|
|
int id = trace_event__id("sys_enter_open");
|
|
|
|
|
|
|
|
if (id < 0) {
|
|
|
|
pr_debug("is debugfs mounted on /sys/kernel/debug?\n");
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
|
|
|
threads = thread_map__new(-1, getpid());
|
|
|
|
if (threads == NULL) {
|
|
|
|
pr_debug("thread_map__new\n");
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
|
|
|
cpus = cpu_map__new(NULL);
|
2011-01-14 23:00:50 +00:00
|
|
|
if (cpus == NULL) {
|
|
|
|
pr_debug("cpu_map__new\n");
|
|
|
|
goto out_thread_map_delete;
|
2011-01-04 13:55:27 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
2011-01-23 01:14:20 +00:00
|
|
|
CPU_ZERO(&cpu_set);
|
2011-01-04 13:55:27 +00:00
|
|
|
|
|
|
|
memset(&attr, 0, sizeof(attr));
|
|
|
|
attr.type = PERF_TYPE_TRACEPOINT;
|
|
|
|
attr.config = id;
|
|
|
|
evsel = perf_evsel__new(&attr, 0);
|
|
|
|
if (evsel == NULL) {
|
|
|
|
pr_debug("perf_evsel__new\n");
|
2011-01-23 01:14:20 +00:00
|
|
|
goto out_thread_map_delete;
|
2011-01-04 13:55:27 +00:00
|
|
|
}
|
|
|
|
|
2011-04-14 14:20:14 +00:00
|
|
|
if (perf_evsel__open(evsel, cpus, threads, false) < 0) {
|
2011-01-04 13:55:27 +00:00
|
|
|
pr_debug("failed to open counter: %s, "
|
|
|
|
"tweak /proc/sys/kernel/perf_event_paranoid?\n",
|
|
|
|
strerror(errno));
|
|
|
|
goto out_evsel_delete;
|
|
|
|
}
|
|
|
|
|
|
|
|
for (cpu = 0; cpu < cpus->nr; ++cpu) {
|
|
|
|
unsigned int ncalls = nr_open_calls + cpu;
|
2011-01-23 01:14:20 +00:00
|
|
|
/*
|
|
|
|
* XXX eventually lift this restriction in a way that
|
|
|
|
* keeps perf building on older glibc installations
|
|
|
|
* without CPU_ALLOC. 1024 cpus in 2010 still seems
|
|
|
|
* a reasonable upper limit tho :-)
|
|
|
|
*/
|
|
|
|
if (cpus->map[cpu] >= CPU_SETSIZE) {
|
|
|
|
pr_debug("Ignoring CPU %d\n", cpus->map[cpu]);
|
|
|
|
continue;
|
|
|
|
}
|
2011-01-04 13:55:27 +00:00
|
|
|
|
2011-01-23 01:14:20 +00:00
|
|
|
CPU_SET(cpus->map[cpu], &cpu_set);
|
|
|
|
if (sched_setaffinity(0, sizeof(cpu_set), &cpu_set) < 0) {
|
perf test: Use cpu_map->[cpu] when setting affinity
When some of CPUs are offline:
# cat /sys/devices/system/cpu/online
0,6-31
perf test will fail on #3 testcase:
3: detect open syscall event on all cpus:
--- start ---
perf_evsel__read_on_cpu: expected to intercept 111 calls on cpu 0, got 681
perf_evsel__read_on_cpu: expected to intercept 112 calls on cpu 1, got 117
perf_evsel__read_on_cpu: expected to intercept 113 calls on cpu 2, got 118
perf_evsel__read_on_cpu: expected to intercept 114 calls on cpu 3, got 119
perf_evsel__read_on_cpu: expected to intercept 115 calls on cpu 4, got 120
perf_evsel__read_on_cpu: expected to intercept 116 calls on cpu 5, got 121
perf_evsel__read_on_cpu: expected to intercept 117 calls on cpu 6, got 122
perf_evsel__read_on_cpu: expected to intercept 118 calls on cpu 7, got 123
perf_evsel__read_on_cpu: expected to intercept 119 calls on cpu 8, got 124
perf_evsel__read_on_cpu: expected to intercept 120 calls on cpu 9, got 125
perf_evsel__read_on_cpu: expected to intercept 121 calls on cpu 10, got 126
....
This patch try to use 'cpus->map[cpu]' when setting cpu affinity, and
will check the return code of sched_setaffinity()
LKML-Reference: <20110120114707.GA11781@hpt.nay.redhat.com>
Signed-off-by: Han Pingtian <phan@redhat.com>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2011-01-20 11:47:07 +00:00
|
|
|
pr_debug("sched_setaffinity() failed on CPU %d: %s ",
|
|
|
|
cpus->map[cpu],
|
|
|
|
strerror(errno));
|
|
|
|
goto out_close_fd;
|
|
|
|
}
|
2011-01-04 13:55:27 +00:00
|
|
|
for (i = 0; i < ncalls; ++i) {
|
|
|
|
fd = open("/etc/passwd", O_RDONLY);
|
|
|
|
close(fd);
|
|
|
|
}
|
2011-01-23 01:14:20 +00:00
|
|
|
CPU_CLR(cpus->map[cpu], &cpu_set);
|
2011-01-04 13:55:27 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Here we need to explicitely preallocate the counts, as if
|
|
|
|
* we use the auto allocation it will allocate just for 1 cpu,
|
|
|
|
* as we start by cpu 0.
|
|
|
|
*/
|
|
|
|
if (perf_evsel__alloc_counts(evsel, cpus->nr) < 0) {
|
|
|
|
pr_debug("perf_evsel__alloc_counts(ncpus=%d)\n", cpus->nr);
|
|
|
|
goto out_close_fd;
|
|
|
|
}
|
|
|
|
|
2011-01-14 18:24:49 +00:00
|
|
|
err = 0;
|
|
|
|
|
2011-01-04 13:55:27 +00:00
|
|
|
for (cpu = 0; cpu < cpus->nr; ++cpu) {
|
|
|
|
unsigned int expected;
|
|
|
|
|
2011-01-23 01:14:20 +00:00
|
|
|
if (cpus->map[cpu] >= CPU_SETSIZE)
|
|
|
|
continue;
|
|
|
|
|
2011-01-04 13:55:27 +00:00
|
|
|
if (perf_evsel__read_on_cpu(evsel, cpu, 0) < 0) {
|
2011-04-14 14:20:14 +00:00
|
|
|
pr_debug("perf_evsel__read_on_cpu\n");
|
2011-01-14 18:24:49 +00:00
|
|
|
err = -1;
|
|
|
|
break;
|
2011-01-04 13:55:27 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
expected = nr_open_calls + cpu;
|
|
|
|
if (evsel->counts->cpu[cpu].val != expected) {
|
2011-01-22 22:37:02 +00:00
|
|
|
pr_debug("perf_evsel__read_on_cpu: expected to intercept %d calls on cpu %d, got %" PRIu64 "\n",
|
perf test: Use cpu_map->[cpu] when setting affinity
When some of CPUs are offline:
# cat /sys/devices/system/cpu/online
0,6-31
perf test will fail on #3 testcase:
3: detect open syscall event on all cpus:
--- start ---
perf_evsel__read_on_cpu: expected to intercept 111 calls on cpu 0, got 681
perf_evsel__read_on_cpu: expected to intercept 112 calls on cpu 1, got 117
perf_evsel__read_on_cpu: expected to intercept 113 calls on cpu 2, got 118
perf_evsel__read_on_cpu: expected to intercept 114 calls on cpu 3, got 119
perf_evsel__read_on_cpu: expected to intercept 115 calls on cpu 4, got 120
perf_evsel__read_on_cpu: expected to intercept 116 calls on cpu 5, got 121
perf_evsel__read_on_cpu: expected to intercept 117 calls on cpu 6, got 122
perf_evsel__read_on_cpu: expected to intercept 118 calls on cpu 7, got 123
perf_evsel__read_on_cpu: expected to intercept 119 calls on cpu 8, got 124
perf_evsel__read_on_cpu: expected to intercept 120 calls on cpu 9, got 125
perf_evsel__read_on_cpu: expected to intercept 121 calls on cpu 10, got 126
....
This patch try to use 'cpus->map[cpu]' when setting cpu affinity, and
will check the return code of sched_setaffinity()
LKML-Reference: <20110120114707.GA11781@hpt.nay.redhat.com>
Signed-off-by: Han Pingtian <phan@redhat.com>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2011-01-20 11:47:07 +00:00
|
|
|
expected, cpus->map[cpu], evsel->counts->cpu[cpu].val);
|
2011-01-14 18:24:49 +00:00
|
|
|
err = -1;
|
2011-01-04 13:55:27 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
out_close_fd:
|
|
|
|
perf_evsel__close_fd(evsel, 1, threads->nr);
|
|
|
|
out_evsel_delete:
|
|
|
|
perf_evsel__delete(evsel);
|
|
|
|
out_thread_map_delete:
|
|
|
|
thread_map__delete(threads);
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
2011-01-15 12:42:46 +00:00
|
|
|
/*
|
|
|
|
* This test will generate random numbers of calls to some getpid syscalls,
|
|
|
|
* then establish an mmap for a group of events that are created to monitor
|
|
|
|
* the syscalls.
|
|
|
|
*
|
|
|
|
* It will receive the events, using mmap, use its PERF_SAMPLE_ID generated
|
|
|
|
* sample.id field to map back to its respective perf_evsel instance.
|
|
|
|
*
|
|
|
|
* Then it checks if the number of syscalls reported as perf events by
|
|
|
|
* the kernel corresponds to the number of syscalls made.
|
|
|
|
*/
|
|
|
|
static int test__basic_mmap(void)
|
|
|
|
{
|
|
|
|
int err = -1;
|
2011-01-29 16:01:45 +00:00
|
|
|
union perf_event *event;
|
2011-01-15 12:42:46 +00:00
|
|
|
struct thread_map *threads;
|
|
|
|
struct cpu_map *cpus;
|
|
|
|
struct perf_evlist *evlist;
|
|
|
|
struct perf_event_attr attr = {
|
|
|
|
.type = PERF_TYPE_TRACEPOINT,
|
|
|
|
.read_format = PERF_FORMAT_ID,
|
|
|
|
.sample_type = PERF_SAMPLE_ID,
|
|
|
|
.watermark = 0,
|
|
|
|
};
|
|
|
|
cpu_set_t cpu_set;
|
|
|
|
const char *syscall_names[] = { "getsid", "getppid", "getpgrp",
|
|
|
|
"getpgid", };
|
|
|
|
pid_t (*syscalls[])(void) = { (void *)getsid, getppid, getpgrp,
|
|
|
|
(void*)getpgid };
|
|
|
|
#define nsyscalls ARRAY_SIZE(syscall_names)
|
|
|
|
int ids[nsyscalls];
|
|
|
|
unsigned int nr_events[nsyscalls],
|
|
|
|
expected_nr_events[nsyscalls], i, j;
|
|
|
|
struct perf_evsel *evsels[nsyscalls], *evsel;
|
2011-06-02 14:04:54 +00:00
|
|
|
int sample_size = __perf_evsel__sample_size(attr.sample_type);
|
2011-01-15 12:42:46 +00:00
|
|
|
|
|
|
|
for (i = 0; i < nsyscalls; ++i) {
|
|
|
|
char name[64];
|
|
|
|
|
|
|
|
snprintf(name, sizeof(name), "sys_enter_%s", syscall_names[i]);
|
|
|
|
ids[i] = trace_event__id(name);
|
|
|
|
if (ids[i] < 0) {
|
|
|
|
pr_debug("Is debugfs mounted on /sys/kernel/debug?\n");
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
nr_events[i] = 0;
|
|
|
|
expected_nr_events[i] = random() % 257;
|
|
|
|
}
|
|
|
|
|
|
|
|
threads = thread_map__new(-1, getpid());
|
|
|
|
if (threads == NULL) {
|
|
|
|
pr_debug("thread_map__new\n");
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
|
|
|
cpus = cpu_map__new(NULL);
|
2011-01-24 23:39:00 +00:00
|
|
|
if (cpus == NULL) {
|
|
|
|
pr_debug("cpu_map__new\n");
|
2011-01-15 12:42:46 +00:00
|
|
|
goto out_free_threads;
|
|
|
|
}
|
|
|
|
|
|
|
|
CPU_ZERO(&cpu_set);
|
|
|
|
CPU_SET(cpus->map[0], &cpu_set);
|
|
|
|
sched_setaffinity(0, sizeof(cpu_set), &cpu_set);
|
|
|
|
if (sched_setaffinity(0, sizeof(cpu_set), &cpu_set) < 0) {
|
|
|
|
pr_debug("sched_setaffinity() failed on CPU %d: %s ",
|
|
|
|
cpus->map[0], strerror(errno));
|
|
|
|
goto out_free_cpus;
|
|
|
|
}
|
|
|
|
|
2011-01-30 13:59:43 +00:00
|
|
|
evlist = perf_evlist__new(cpus, threads);
|
2011-01-24 23:39:00 +00:00
|
|
|
if (evlist == NULL) {
|
2011-01-15 12:42:46 +00:00
|
|
|
pr_debug("perf_evlist__new\n");
|
|
|
|
goto out_free_cpus;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* anonymous union fields, can't be initialized above */
|
|
|
|
attr.wakeup_events = 1;
|
|
|
|
attr.sample_period = 1;
|
|
|
|
|
|
|
|
for (i = 0; i < nsyscalls; ++i) {
|
|
|
|
attr.config = ids[i];
|
|
|
|
evsels[i] = perf_evsel__new(&attr, i);
|
|
|
|
if (evsels[i] == NULL) {
|
|
|
|
pr_debug("perf_evsel__new\n");
|
|
|
|
goto out_free_evlist;
|
|
|
|
}
|
|
|
|
|
|
|
|
perf_evlist__add(evlist, evsels[i]);
|
|
|
|
|
2011-04-14 14:20:14 +00:00
|
|
|
if (perf_evsel__open(evsels[i], cpus, threads, false) < 0) {
|
2011-01-15 12:42:46 +00:00
|
|
|
pr_debug("failed to open counter: %s, "
|
|
|
|
"tweak /proc/sys/kernel/perf_event_paranoid?\n",
|
|
|
|
strerror(errno));
|
|
|
|
goto out_close_fd;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2011-01-30 13:59:43 +00:00
|
|
|
if (perf_evlist__mmap(evlist, 128, true) < 0) {
|
2011-01-15 12:42:46 +00:00
|
|
|
pr_debug("failed to mmap events: %d (%s)\n", errno,
|
|
|
|
strerror(errno));
|
|
|
|
goto out_close_fd;
|
|
|
|
}
|
|
|
|
|
|
|
|
for (i = 0; i < nsyscalls; ++i)
|
|
|
|
for (j = 0; j < expected_nr_events[i]; ++j) {
|
|
|
|
int foo = syscalls[i]();
|
|
|
|
++foo;
|
|
|
|
}
|
|
|
|
|
perf evlist: Fix per thread mmap setup
The PERF_EVENT_IOC_SET_OUTPUT ioctl was returning -EINVAL when using
--pid when monitoring multithreaded apps, as we can only share a ring
buffer for events on the same thread if not doing per cpu.
Fix it by using per thread ring buffers.
Tested with:
[root@felicio ~]# tuna -t 26131 -CP | nl
1 thread ctxt_switches
2 pid SCHED_ rtpri affinity voluntary nonvoluntary cmd
3 26131 OTHER 0 0,1 10814276 2397830 chromium-browse
4 642 OTHER 0 0,1 14688 0 chromium-browse
5 26148 OTHER 0 0,1 713602 115479 chromium-browse
6 26149 OTHER 0 0,1 801958 2262 chromium-browse
7 26150 OTHER 0 0,1 1271128 248 chromium-browse
8 26151 OTHER 0 0,1 3 0 chromium-browse
9 27049 OTHER 0 0,1 36796 9 chromium-browse
10 618 OTHER 0 0,1 14711 0 chromium-browse
11 661 OTHER 0 0,1 14593 0 chromium-browse
12 29048 OTHER 0 0,1 28125 0 chromium-browse
13 26143 OTHER 0 0,1 2202789 781 chromium-browse
[root@felicio ~]#
So 11 threads under pid 26131, then:
[root@felicio ~]# perf record -F 50000 --pid 26131
[root@felicio ~]# grep perf_event /proc/`pidof perf`/maps | nl
1 7fa4a2538000-7fa4a25b9000 rwxs 00000000 00:09 4064 anon_inode:[perf_event]
2 7fa4a25b9000-7fa4a263a000 rwxs 00000000 00:09 4064 anon_inode:[perf_event]
3 7fa4a263a000-7fa4a26bb000 rwxs 00000000 00:09 4064 anon_inode:[perf_event]
4 7fa4a26bb000-7fa4a273c000 rwxs 00000000 00:09 4064 anon_inode:[perf_event]
5 7fa4a273c000-7fa4a27bd000 rwxs 00000000 00:09 4064 anon_inode:[perf_event]
6 7fa4a27bd000-7fa4a283e000 rwxs 00000000 00:09 4064 anon_inode:[perf_event]
7 7fa4a283e000-7fa4a28bf000 rwxs 00000000 00:09 4064 anon_inode:[perf_event]
8 7fa4a28bf000-7fa4a2940000 rwxs 00000000 00:09 4064 anon_inode:[perf_event]
9 7fa4a2940000-7fa4a29c1000 rwxs 00000000 00:09 4064 anon_inode:[perf_event]
10 7fa4a29c1000-7fa4a2a42000 rwxs 00000000 00:09 4064 anon_inode:[perf_event]
11 7fa4a2a42000-7fa4a2ac3000 rwxs 00000000 00:09 4064 anon_inode:[perf_event]
[root@felicio ~]#
11 mmaps, one per thread since we didn't specify any CPU list, so we need one
mmap per thread and:
[root@felicio ~]# perf record -F 50000 --pid 26131
^M
^C[ perf record: Woken up 79 times to write data ]
[ perf record: Captured and wrote 20.614 MB perf.data (~900639 samples) ]
[root@felicio ~]# perf report -D | grep PERF_RECORD_SAMPLE | cut -d/ -f2 | cut -d: -f1 | sort -n | uniq -c | sort -nr | nl
1 371310 26131
2 96516 26148
3 95694 26149
4 95203 26150
5 7291 26143
6 87 27049
7 76 661
8 60 29048
9 47 618
10 43 642
[root@felicio ~]#
Ok, one of the threads, 26151 was quiescent, so no samples there, but all the
others are there.
Then, if I specify one CPU:
[root@felicio ~]# perf record -F 50000 --pid 26131 --cpu 1
^C[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.680 MB perf.data (~29730 samples) ]
[root@felicio ~]# perf report -D | grep PERF_RECORD_SAMPLE | cut -d/ -f2 | cut -d: -f1 | sort -n | uniq -c | sort -nr | nl
1 8444 26131
2 2584 26149
3 2518 26148
4 2324 26150
5 123 26143
6 9 661
7 9 29048
[root@felicio ~]#
This machine has two cores, so fewer threads appeared on the radar, and:
[root@felicio ~]# grep perf_event /proc/`pidof perf`/maps | nl
1 7f484b922000-7f484b9a3000 rwxs 00000000 00:09 4064 anon_inode:[perf_event]
[root@felicio ~]#
Just one mmap, as now we can use just one per-cpu buffer instead of the
per-thread needed in the previous case.
For global profiling:
[root@felicio ~]# perf record -F 50000 -a
^C[ perf record: Woken up 26 times to write data ]
[ perf record: Captured and wrote 7.128 MB perf.data (~311412 samples) ]
[root@felicio ~]# grep perf_event /proc/`pidof perf`/maps | nl
1 7fb49b435000-7fb49b4b6000 rwxs 00000000 00:09 4064 anon_inode:[perf_event]
2 7fb49b4b6000-7fb49b537000 rwxs 00000000 00:09 4064 anon_inode:[perf_event]
[root@felicio ~]#
It uses per-cpu buffers.
For just one thread:
[root@felicio ~]# perf record -F 50000 --tid 26148
^C[ perf record: Woken up 2 times to write data ]
[ perf record: Captured and wrote 0.330 MB perf.data (~14426 samples) ]
[root@felicio ~]# perf report -D | grep PERF_RECORD_SAMPLE | cut -d/ -f2 | cut -d: -f1 | sort -n | uniq -c | sort -nr | nl
1 9969 26148
[root@felicio ~]#
[root@felicio ~]# grep perf_event /proc/`pidof perf`/maps | nl
1 7f286a51b000-7f286a59c000 rwxs 00000000 00:09 4064 anon_inode:[perf_event]
[root@felicio ~]#
Tested-by: David Ahern <dsahern@gmail.com>
Tested-by: Lin Ming <ming.m.lin@intel.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Stephane Eranian <eranian@google.com>
Cc: Tom Zanussi <tzanussi@gmail.com>
Link: http://lkml.kernel.org/r/20110426204401.GB1746@ghostprotocols.net
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2011-05-15 12:39:00 +00:00
|
|
|
while ((event = perf_evlist__mmap_read(evlist, 0)) != NULL) {
|
2011-01-29 15:02:00 +00:00
|
|
|
struct perf_sample sample;
|
2011-01-15 12:42:46 +00:00
|
|
|
|
|
|
|
if (event->header.type != PERF_RECORD_SAMPLE) {
|
|
|
|
pr_debug("unexpected %s event\n",
|
2011-01-29 16:01:45 +00:00
|
|
|
perf_event__name(event->header.type));
|
2011-01-15 12:42:46 +00:00
|
|
|
goto out_munmap;
|
|
|
|
}
|
|
|
|
|
2011-05-22 00:17:22 +00:00
|
|
|
err = perf_event__parse_sample(event, attr.sample_type, sample_size,
|
|
|
|
false, &sample);
|
|
|
|
if (err) {
|
|
|
|
pr_err("Can't parse sample, err = %d\n", err);
|
|
|
|
goto out_munmap;
|
|
|
|
}
|
|
|
|
|
2011-01-15 12:42:46 +00:00
|
|
|
evsel = perf_evlist__id2evsel(evlist, sample.id);
|
|
|
|
if (evsel == NULL) {
|
|
|
|
pr_debug("event with id %" PRIu64
|
|
|
|
" doesn't map to an evsel\n", sample.id);
|
|
|
|
goto out_munmap;
|
|
|
|
}
|
|
|
|
nr_events[evsel->idx]++;
|
|
|
|
}
|
|
|
|
|
|
|
|
list_for_each_entry(evsel, &evlist->entries, node) {
|
|
|
|
if (nr_events[evsel->idx] != expected_nr_events[evsel->idx]) {
|
|
|
|
pr_debug("expected %d %s events, got %d\n",
|
|
|
|
expected_nr_events[evsel->idx],
|
|
|
|
event_name(evsel), nr_events[evsel->idx]);
|
|
|
|
goto out_munmap;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
err = 0;
|
|
|
|
out_munmap:
|
2011-01-30 13:59:43 +00:00
|
|
|
perf_evlist__munmap(evlist);
|
2011-01-15 12:42:46 +00:00
|
|
|
out_close_fd:
|
|
|
|
for (i = 0; i < nsyscalls; ++i)
|
|
|
|
perf_evsel__close_fd(evsels[i], 1, threads->nr);
|
|
|
|
out_free_evlist:
|
|
|
|
perf_evlist__delete(evlist);
|
|
|
|
out_free_cpus:
|
|
|
|
cpu_map__delete(cpus);
|
|
|
|
out_free_threads:
|
|
|
|
thread_map__delete(threads);
|
|
|
|
return err;
|
|
|
|
#undef nsyscalls
|
|
|
|
}
|
|
|
|
|
perf test: Initial regression testing command
First an example with the first internal test:
[acme@doppio linux-2.6-tip]$ perf test
1: vmlinux symtab matches kallsyms: Ok
So it run just one test, that is "vmlinux symtab matches kallsyms", and it was
successful.
If we run it in verbose mode, we'll see details about errors and extra warnings
for non-fatal problems:
[acme@doppio linux-2.6-tip]$ perf test -v
1: vmlinux symtab matches kallsyms:
--- start ---
Looking at the vmlinux_path (5 entries long)
No build_id in vmlinux, ignoring it
No build_id in /boot/vmlinux, ignoring it
No build_id in /boot/vmlinux-2.6.34-rc4-tip+, ignoring it
Using /lib/modules/2.6.34-rc4-tip+/build/vmlinux for symbols
Maps only in vmlinux:
ffffffff81cb81b1-ffffffff81e1149b 0 [kernel].init.text
ffffffff81e1149c-ffffffff9fffffff 0 [kernel].exit.text
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2
Maps in vmlinux with a different name in kallsyms:
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0 in kallsyms as [kernel].0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn in kallsyms as:
*ffffffffff600100-ffffffffff60012f 0 [kernel].2
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1 in kallsyms as [kernel].6
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2 in kallsyms as [kernel].8
Maps only in kallsyms:
ffffffffff600130-ffffffffff6003ff 0 [kernel].4
---- end ----
vmlinux symtab matches kallsyms: Ok
[acme@doppio linux-2.6-tip]$
In the above case we only know the name of the non contiguous kernel ranges in
the address space when reading the symbol information from the ELF symtab in
vmlinux.
The /proc/kallsyms file lack this, we only notice they are separate because
there are modules after the kernel and after that more kernel functions, so we
need to have a module rbtree backed by the module .ko path to get symtabs in
the vmlinux case.
The tool uses it to match by address to emit appropriate warning, but don't
considers this fatal.
The .init.text and .exit.text ines, of course, aren't in kallsyms, so I left
these cases just as extra info in verbose mode.
The end of the sections also aren't in kallsyms, so we the symbols layer does
another pass and sets the end addresses as the next map start minus one, which
sometimes pads, causing harmless mismatches.
But at least the symbols match, tested it by copying /proc/kallsyms to
/tmp/kallsyms and doing changes to see if they were detected.
This first test also should serve as a first stab at documenting the
symbol library by providing a self contained example that exercises it
together with comments about what is being done.
More tests to check if actions done on a monitored app, like doing mmaps, etc,
makes the kernel generate the expected events should be added next.
Cc: Frédéric Weisbecker <fweisbec@gmail.com>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <new-submission>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2010-04-29 21:58:32 +00:00
|
|
|
static struct test {
|
|
|
|
const char *desc;
|
|
|
|
int (*func)(void);
|
|
|
|
} tests[] = {
|
|
|
|
{
|
|
|
|
.desc = "vmlinux symtab matches kallsyms",
|
|
|
|
.func = test__vmlinux_matches_kallsyms,
|
|
|
|
},
|
2011-01-04 02:16:20 +00:00
|
|
|
{
|
|
|
|
.desc = "detect open syscall event",
|
|
|
|
.func = test__open_syscall_event,
|
|
|
|
},
|
2011-01-04 13:55:27 +00:00
|
|
|
{
|
|
|
|
.desc = "detect open syscall event on all cpus",
|
|
|
|
.func = test__open_syscall_event_on_all_cpus,
|
|
|
|
},
|
2011-01-15 12:42:46 +00:00
|
|
|
{
|
|
|
|
.desc = "read samples using the mmap interface",
|
|
|
|
.func = test__basic_mmap,
|
|
|
|
},
|
perf test: Initial regression testing command
First an example with the first internal test:
[acme@doppio linux-2.6-tip]$ perf test
1: vmlinux symtab matches kallsyms: Ok
So it run just one test, that is "vmlinux symtab matches kallsyms", and it was
successful.
If we run it in verbose mode, we'll see details about errors and extra warnings
for non-fatal problems:
[acme@doppio linux-2.6-tip]$ perf test -v
1: vmlinux symtab matches kallsyms:
--- start ---
Looking at the vmlinux_path (5 entries long)
No build_id in vmlinux, ignoring it
No build_id in /boot/vmlinux, ignoring it
No build_id in /boot/vmlinux-2.6.34-rc4-tip+, ignoring it
Using /lib/modules/2.6.34-rc4-tip+/build/vmlinux for symbols
Maps only in vmlinux:
ffffffff81cb81b1-ffffffff81e1149b 0 [kernel].init.text
ffffffff81e1149c-ffffffff9fffffff 0 [kernel].exit.text
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2
Maps in vmlinux with a different name in kallsyms:
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0 in kallsyms as [kernel].0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn in kallsyms as:
*ffffffffff600100-ffffffffff60012f 0 [kernel].2
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1 in kallsyms as [kernel].6
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2 in kallsyms as [kernel].8
Maps only in kallsyms:
ffffffffff600130-ffffffffff6003ff 0 [kernel].4
---- end ----
vmlinux symtab matches kallsyms: Ok
[acme@doppio linux-2.6-tip]$
In the above case we only know the name of the non contiguous kernel ranges in
the address space when reading the symbol information from the ELF symtab in
vmlinux.
The /proc/kallsyms file lack this, we only notice they are separate because
there are modules after the kernel and after that more kernel functions, so we
need to have a module rbtree backed by the module .ko path to get symtabs in
the vmlinux case.
The tool uses it to match by address to emit appropriate warning, but don't
considers this fatal.
The .init.text and .exit.text ines, of course, aren't in kallsyms, so I left
these cases just as extra info in verbose mode.
The end of the sections also aren't in kallsyms, so we the symbols layer does
another pass and sets the end addresses as the next map start minus one, which
sometimes pads, causing harmless mismatches.
But at least the symbols match, tested it by copying /proc/kallsyms to
/tmp/kallsyms and doing changes to see if they were detected.
This first test also should serve as a first stab at documenting the
symbol library by providing a self contained example that exercises it
together with comments about what is being done.
More tests to check if actions done on a monitored app, like doing mmaps, etc,
makes the kernel generate the expected events should be added next.
Cc: Frédéric Weisbecker <fweisbec@gmail.com>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <new-submission>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2010-04-29 21:58:32 +00:00
|
|
|
{
|
|
|
|
.func = NULL,
|
|
|
|
},
|
|
|
|
};
|
|
|
|
|
|
|
|
static int __cmd_test(void)
|
|
|
|
{
|
|
|
|
int i = 0;
|
|
|
|
|
|
|
|
page_size = sysconf(_SC_PAGE_SIZE);
|
|
|
|
|
|
|
|
while (tests[i].func) {
|
|
|
|
int err;
|
|
|
|
pr_info("%2d: %s:", i + 1, tests[i].desc);
|
|
|
|
pr_debug("\n--- start ---\n");
|
|
|
|
err = tests[i].func();
|
|
|
|
pr_debug("---- end ----\n%s:", tests[i].desc);
|
|
|
|
pr_info(" %s\n", err ? "FAILED!\n" : "Ok");
|
|
|
|
++i;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static const char * const test_usage[] = {
|
|
|
|
"perf test [<options>]",
|
|
|
|
NULL,
|
|
|
|
};
|
|
|
|
|
|
|
|
static const struct option test_options[] = {
|
2010-05-17 18:51:10 +00:00
|
|
|
OPT_INTEGER('v', "verbose", &verbose,
|
perf test: Initial regression testing command
First an example with the first internal test:
[acme@doppio linux-2.6-tip]$ perf test
1: vmlinux symtab matches kallsyms: Ok
So it run just one test, that is "vmlinux symtab matches kallsyms", and it was
successful.
If we run it in verbose mode, we'll see details about errors and extra warnings
for non-fatal problems:
[acme@doppio linux-2.6-tip]$ perf test -v
1: vmlinux symtab matches kallsyms:
--- start ---
Looking at the vmlinux_path (5 entries long)
No build_id in vmlinux, ignoring it
No build_id in /boot/vmlinux, ignoring it
No build_id in /boot/vmlinux-2.6.34-rc4-tip+, ignoring it
Using /lib/modules/2.6.34-rc4-tip+/build/vmlinux for symbols
Maps only in vmlinux:
ffffffff81cb81b1-ffffffff81e1149b 0 [kernel].init.text
ffffffff81e1149c-ffffffff9fffffff 0 [kernel].exit.text
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2
Maps in vmlinux with a different name in kallsyms:
ffffffffff600000-ffffffffff6000ff 0 [kernel].vsyscall_0 in kallsyms as [kernel].0
ffffffffff600100-ffffffffff6003ff 0 [kernel].vsyscall_fn in kallsyms as:
*ffffffffff600100-ffffffffff60012f 0 [kernel].2
ffffffffff600400-ffffffffff6007ff 0 [kernel].vsyscall_1 in kallsyms as [kernel].6
ffffffffff600800-ffffffffffffffff 0 [kernel].vsyscall_2 in kallsyms as [kernel].8
Maps only in kallsyms:
ffffffffff600130-ffffffffff6003ff 0 [kernel].4
---- end ----
vmlinux symtab matches kallsyms: Ok
[acme@doppio linux-2.6-tip]$
In the above case we only know the name of the non contiguous kernel ranges in
the address space when reading the symbol information from the ELF symtab in
vmlinux.
The /proc/kallsyms file lack this, we only notice they are separate because
there are modules after the kernel and after that more kernel functions, so we
need to have a module rbtree backed by the module .ko path to get symtabs in
the vmlinux case.
The tool uses it to match by address to emit appropriate warning, but don't
considers this fatal.
The .init.text and .exit.text ines, of course, aren't in kallsyms, so I left
these cases just as extra info in verbose mode.
The end of the sections also aren't in kallsyms, so we the symbols layer does
another pass and sets the end addresses as the next map start minus one, which
sometimes pads, causing harmless mismatches.
But at least the symbols match, tested it by copying /proc/kallsyms to
/tmp/kallsyms and doing changes to see if they were detected.
This first test also should serve as a first stab at documenting the
symbol library by providing a self contained example that exercises it
together with comments about what is being done.
More tests to check if actions done on a monitored app, like doing mmaps, etc,
makes the kernel generate the expected events should be added next.
Cc: Frédéric Weisbecker <fweisbec@gmail.com>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <new-submission>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2010-04-29 21:58:32 +00:00
|
|
|
"be more verbose (show symbol address, etc)"),
|
|
|
|
OPT_END()
|
|
|
|
};
|
|
|
|
|
|
|
|
int cmd_test(int argc, const char **argv, const char *prefix __used)
|
|
|
|
{
|
|
|
|
argc = parse_options(argc, argv, test_options, test_usage, 0);
|
|
|
|
if (argc)
|
|
|
|
usage_with_options(test_usage, test_options);
|
|
|
|
|
|
|
|
symbol_conf.priv_size = sizeof(int);
|
|
|
|
symbol_conf.sort_by_name = true;
|
|
|
|
symbol_conf.try_vmlinux_path = true;
|
|
|
|
|
|
|
|
if (symbol__init() < 0)
|
|
|
|
return -1;
|
|
|
|
|
|
|
|
setup_pager();
|
|
|
|
|
|
|
|
return __cmd_test();
|
|
|
|
}
|