363b785f38 added synthesized fork events and set a thread's parent id to
itself. Since we are already processing /proc/<pid>/status the ppid can
be determined properly. Make it so.
Signed-off-by: David Ahern <dsahern@gmail.com>
Acked-by: Don Zickus <dzickus@redhat.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Cc: Joe Mario <jmario@redhat.com>
Link: http://lkml.kernel.org/r/1427747758-18510-2-git-send-email-dsahern@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Rather than parsing /proc/pid/status file one line at a time, read it
into a buffer in one shot and search for all strings in one pass.
tgid conversion also simplified -- removing the isspace walk. As noted
by Arnaldo those are not needed for atoi == strtol calls.
Signed-off-by: David Ahern <dsahern@gmail.com>
Acked-by: Don Zickus <dzickus@redhat.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Cc: Joe Mario <jmario@redhat.com>
Link: http://lkml.kernel.org/r/1427747758-18510-1-git-send-email-dsahern@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Commit 2e77784bb7 ("perf callchain: Move cpumode resolve code to
add_callchain_ip") promised "No change in behavior.".
As this commit breaks callchains on s390x (symbols not getting resolved,
observed when profiling the kernel), this statement is wrong. The cpumode
must be kept when iterating over all ips, otherwise the default
(PERF_RECORD_MISC_USER) will be used by error.
Signed-off-by: David Hildenbrand <dahi@linux.vnet.ibm.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Hildenbrand <dahi@linux.vnet.ibm.com>
Cc: Hendrik Brueckner <brueckner@linux.vnet.ibm.com>
Cc: Kan Liang <kan.liang@intel.com>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1427703060-59883-1-git-send-email-dahi@linux.vnet.ibm.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Use of a bad filter currently generates the message:
Error: failed to set filter with 22 (Invalid argument)
Add the event name to make it clear to which event the filter
failed to apply:
Error: Failed to set filter "foo" on event sched:sg_lb_stats: 22: Invalid argument
To test it use something like:
# perf record -e sched:sched_switch -e sched:*fork --filter parent_pid==1 -e sched:*wait* --filter bla usleep 1
Error: failed to set filter "bla" on event sched:sched_stat_iowait with 22 (Invalid argument)
#
Based-on-a-patch-by: David Ahern <dsahern@gmail.com>
Acked-by: David Ahern <dsahern@gmail.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Borislav Petkov <bp@suse.de>
Cc: Don Zickus <dzickus@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/n/tip-d7gq2fjvaecozp9o2i0siifu@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
perf timechart -T on sparc64 is terminating due to SIGBUS. Backtrace:
Program received signal SIGBUS, Bus error.
0x0000000000173d7c in perf_evsel__intval (evsel=<value optimized out>, sample=0x7feffffda28, name=0x289b28 "prev_state")
at util/evsel.c:1918
1918 util/evsel.c: No such file or directory.
in util/evsel.c
Missing separate debuginfos, use: debuginfo-install audit-libs-2.3.7-1.0.1.el6.sparc64 bzip2-libs-1.0.5-7.el6_0.sparc64 elfutils-libelf-0.155-2.0.3.el6.sparc64 elfutils-libs-0.155-2.0.3.el6.sparc64 glibc-2.12-1.132.0.8.el6_5.sparc64 numactl-2.0.7-8.el6.sparc64 python-libs-2.6.6-52.0.2.el6.sparc64 slang-2.2.1-1.el6.sparc64 xz-libs-4.999.9-0.3.beta.20091007git.el6.sparc64 zlib-1.2.3-29.el6.sparc64
(gdb) bt
0 0x0000000000173d7c in perf_evsel__intval (evsel=<value optimized out>, sample=0x7feffffda28,
name=0x289b28 "prev_state") at util/evsel.c:1918
1 0x0000000000123b94 in process_sample_sched_switch (tchart=0x7feffffe040, evsel=0x4ca850, sample=0x7feffffda28,
backtrace=0xc39010 "") at builtin-timechart.c:627
2 0x0000000000122828 in process_sample_event (tool=0x7feffffe040, event=<value optimized out>, sample=0x7feffffda28,
evsel=0x4ca850, machine=0x4c9c88) at builtin-timechart.c:569
Another extended load on unaligned pointer. As before fix by copying to
a temporary variable using memcpy.
Signed-off-by: David Ahern <david.ahern@oracle.com>
Link: http://lkml.kernel.org/r/1427228049-51893-1-git-send-email-david.ahern@oracle.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
The 'record' and 'top' tools already allow a user to specify a CSV of
pids and/or tids of tasks to collect data.
Add those options to the 'report' and 'script' analysis commands to only
consider samples related to the given pids/tids.
This is also inline with the existing comm option.
Signed-off-by: David Ahern <dsahern@gmail.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/1427212361-7066-1-git-send-email-dsahern@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Since 6ea22486ba ("tracing: Add array printing helper") trace can
generate traces with variable element size arrays. Add support to
parse them.
Signed-off-by: Javi Merino <javi.merino@arm.com>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Acked-by: Steven Rostedt <rostedt@goodmis.org>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/1427195239-15730-1-git-send-email-javi.merino@arm.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Before, when some problem happened while trying to load the kernel
symtab, 'perf top' would show:
┌─Warning:───────────────────────────┐
│The vmlinux file can't be used. │
│Kernel samples will not be resolved.│
│ │
│ │
│Press any key... │
└────────────────────────────────────┘
Now, it reports:
# perf top --vmlinux /dev/null
┌─Warning:───────────────────────────────────────────┐
│The /tmp/passwd file can't be used: Invalid ELF file│
│Kernel samples will not be resolved. │
│ │
│ │
│Press any key... │
└────────────────────────────────────────────────────┘
This is possible because we now register the reason for not being able
to load the symtab in the dso->load_errno member, and provide a
dso__strerror_load() routine to format this error into a strerror like
string with a short reason for the error while loading.
That can be just forwarding the dso__strerror_load() call to
strerror_r(), or, for a separate errno range providing a custom message.
Reported-by: Ingo Molnar <mingo@kernel.org>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Borislav Petkov <bp@suse.de>
Cc: David Ahern <dsahern@gmail.com>
Cc: Don Zickus <dzickus@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/n/tip-u5rb5uq63xqhkfb8uv2lxd5u@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
To deal with forwarding the strerror_r (GNU) return we need to check if
the returned value is the buffer we passed or maybe some constant
(unknown error), simplify that action by using scnprintf, that will do
all the buflen size checks, trimming if needed.
Acked-by: Jiri Olsa <jolsa@redhat.com>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Borislav Petkov <bp@suse.de>
Cc: David Ahern <dsahern@gmail.com>
Cc: Don Zickus <dzickus@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/n/tip-d0ik6i5gjew56j0qphql28ou@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
This patch fixes this build error with glibc < 2.6.
CC util/cloexec.o
cc1: warnings being treated as errors
util/cloexec.c: In function ‘perf_flag_probe’:
util/cloexec.c:24: error: implicit declaration of function
‘sched_getcpu’
util/cloexec.c:24: error: nested extern declaration of ‘sched_getcpu’
make: *** [util/cloexec.o] Error 1
Signed-off-by: Vinson Lee <vlee@twitter.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Yann Droneaud <ydroneaud@opteya.com>
Cc: stable@vger.kernel.org # 3.18+
Link: http://lkml.kernel.org/r/1427137761-16119-1-git-send-email-vlee@twopensource.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Fix to get correctly unmapped symbol address on kernel. This allows us
to probe on syscall symbols which are aliases of SyS_ functions with
using debuginfo.
Without this fix:
----
# ./perf probe -a sys_write
Failed to find debug information for address 3b0100
Probe point 'sys_write' not found.
Error: Failed to add events.
----
The address 0x3b0100 is a mapped address, and not usable
in debuginfo.
With this fix:
----
# ./perf probe -a sys_write
Added new event:
probe:sys_write (on sys_write)
You can now use it in all perf tools, such as:
perf record -e probe:sys_write -aR sleep 1
----
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: He Kuang <hekuang@huawei.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/20150322114022.32639.19096.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
When '--sort' is not set, 'perf mem report" will print a null pointer as
the output value of sort order, so fix it.
Example:
Before this patch:
$ perf mem report
# To display the perf.data header info, please use --header/--header-only options.
#
# Samples: 18 of event 'cpu/mem-loads/pp'
# Total weight : 188
# Sort order : (null)
#
...
After this patch:
$ perf mem report
# To display the perf.data header info, please use --header/--header-only options.
#
# Samples: 18 of event 'cpu/mem-loads/pp'
# Total weight : 188
# Sort order : local_weight,mem,sym,dso,symbol_daddr,dso_daddr,snoop,tlb,locked
#
...
Signed-off-by: Yunlong Song <yunlong.song@huawei.com>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/1427082605-12881-1-git-send-email-yunlong.song@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Currently we assume machine__new_module is called only once for each
module so we create its map&dso unconditionally.
However it's possible that it's called multiple times for same module.
Like for perf record:
1) via machine__create_module during machine init
2) via kernel MMAP event processing
Trying to lookup kernel module map before creating one.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/n/tip-kx76xfqpnrpho5hdaapbqm09@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
We no longer need the 'compressed' argument, because all
current users use 'NULL' for it.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/n/tip-d72q2s7ggbmy2yzhumux4zzw@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Replacing the current parsing code with kmod_path__parse function call.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/n/tip-r9mpbbgkp39wp1cdmv13ddq0@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Replacing the file name parsing with kmod_path__parse
and moving the dso update into new separate function.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/n/tip-q0ed76ajcyoaofotntrg5sla@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Using kmod_path__parse to get the module name and update the dso short
name within machine__new_dso function.
This way it's done only first time when dso is created, unlike the
current way when we update it all the time we process memory map of the
kernel module.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/n/tip-8gjmt1ggf5ls1xkk7qi2ko4k@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Separate the dso object addition and update when adding new kernel
module.
Currently we update dso's symtab_type any time we find it in the list,
because we can't distinguish between new and found dso from
__dsos__findnew function.
Adding machine__module_dso that separates finding and adding new dso
objects, so there's no superfluous update of dso.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/n/tip-uvqgs5tyq4wssnq6fm43hgvk@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Separate the creation of new dso object and its addition to the dsos
list. It will be used in following patch.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/n/tip-8j43jod97fdt5dwdsushwwae@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Provides united way of parsing kernel module path
into several components.
The new kmod_path__parse function and few defines:
int __kmod_path__parse(struct kmod_path *m, const char *path,
bool alloc_name, bool alloc_ext);
#define kmod_path__parse(__m, __p) __kmod_path__parse(__m, __p, false, false)
#define kmod_path__parse_name(__m, __p) __kmod_path__parse(__m, __p, true , false)
#define kmod_path__parse_ext(__m, __p) __kmod_path__parse(__m, __p, false, true)
parse kernel module @path and updates @m argument like:
@comp - true if @path contains supported compression suffix,
false otherwise
@kmod - true if @path contains '.ko' suffix in right position,
false otherwise
@name - if (@alloc_name && @kmod) is true, it contains strdup-ed base name
of the kernel module without suffixes, otherwise strudup-ed
base name of @path
@ext - if (@alloc_ext && @comp) is true, it contains strdup-ed string
the compression suffix
It returns 0 if there's no strdup error, -ENOMEM otherwise.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/n/tip-9t6eqg8j610r94l743hkntiv@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
In short, Fedora compresses kernel modules now (since version 21) with
lzma compression.
Adding lzma decompress support into the dso.c:compressions array
introduced by Namhyung earlier.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/n/tip-2glp65kdtbrk0gblmirsjsnt@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Perf tries to find probe function addresses from map when debuginfo
could not be found.
To the first added function, the value of ref_reloc_sym was set in
maps__set_kallsyms_ref_reloc_sym() and can be obtained from
host_machine->kmaps->maps. After that, new maps are added to
host_machine->kmaps->maps in dso__load_kcore(), all these new added maps
do not have a valid ref_reloc_sym.
When adding a second function, get_target_map() may get a map without
valid ref_reloc_sym, and raise the error "Relocated base symbol is not
found".
Fix this by using kernel_get_ref_reloc_sym() to get ref_reloc_sym.
This problem can be reproduced as following:
$ perf probe --add='sys_write' --add='sys_open'
Relocated base symbol is not found!
Error: Failed to add events.
After this patch:
$ perf probe --add='sys_write' --add='sys_open'
Added new event:
probe:sys_write (on sys_write)
You can now use it in all perf tools, such as:
perf record -e probe:sys_write -aR sleep 1
Added new event:
probe:sys_open (on sys_open)
You can now use it in all perf tools, such as:
perf record -e probe:sys_open -aR sleep 1
Signed-off-by: He Kuang <hekuang@huawei.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Tested-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/1426816616-2394-1-git-send-email-hekuang@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
int build_id_cache__add_s(const char *sbuild_id, const char *debugdir,
const char *name, bool is_kallsyms, bool is_vdso)
{
...
if (access(filename, F_OK)) {
^--------------------------------------------------------- [1]
if (is_kallsyms) {
if (copyfile("/proc/kallsyms", filename))
goto out_free;
} else if (link(realname, filename) && copyfile(name, filename))
^-----------------------------^------------- [2]
\------------ [3]
goto out_free;
}
...
When multiple instances of perf record get to [1] at more or less same time and
run access() one or more may get failure because the file does not exist yet
(since the first instance did not have chance to link it yet).
At this point the race moves to link() at [2] where first thread to get
there links file and goes on but second one gets -EEXIST so it runs
copyfile [3] which truncates the file.
reproducer:
rm -rf /root/.debug
for cpu in $(awk '/processor/ {print $3}' /proc/cpuinfo); do
perf record -a -v -T -F 1000 -C $cpu \
-o perf-${cpu}.data sleep 5 2> /dev/null &
done
wait
and simply search for empty files by:
find /lib/modules/`uname -r`/kernel/* -size 0
Signed-off-by: Milos Vyletel <milos@redhat.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/1426847846-11112-1-git-send-email-milos@redhat.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Without this patch, perf report cause segfault if pass "" as '-t':
$ perf report -t ""
# To display the perf.data header info, please use --header/--header-only options.
#
# Samples: 37 of event 'syscalls:sys_enter_write'
# Event count (approx.): 37
#
# Children SelfCommand Shared Object Symbol
Segmentation fault
Since -t is used to add field-separator for generate table, -t "" is
actually meanless. This patch defines a new OPT_STRING_NOEMPTY() option
generator to ensure user never pass empty string to that option.
Signed-off-by: Wang Nan <wangnan0@huawei.com>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: pi3orama@163.com
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Zefan Li <lizefan@huawei.com>
Link: http://lkml.kernel.org/r/1426251114-198991-1-git-send-email-wangnan0@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Commit f1f13af99a ("perf callchain: Cache eh/debug frame offset for
dwarf unwind") introduces a cache for .debug_frame and .eh_frame_hdr.
Unfortunately, it makes them share a same cache (dso->frame_offset).
Which causes unwind failure on ARM:
$ perf test unwind
Test dwarf unwind: FAILED!
The reason is that, if a dso has '.debug_frame' but doesn't have
'.eh_frame_hdr' (like ARM), dso->frame_offset will be filled by offset
of '.debug_frame' during the first time calling of find_proc_info() ->
read_unwind_spec_debug_frame(), and be regarded to '.eh_frame_hdr' when
the second time calling of find_proc_info() ->
read_unwind_spec_eh_frame(), since '.eh_frame_hdr' is checked prior to
'.debug_frame'.
This patch solves the problem by creating two cache fields for
'.eh_frame_hdr' and '.debug_frame'.
Signed-off-by: Wang Nan <wangnan0@huawei.com>
Acked-by: Jiri Olsa <jolsa@redhat.com>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: Li Zefan <lizefan@huawei.com>
Link: http://lkml.kernel.org/r/55028BA0.1030701@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Instead of annotating just the top level hist_entry, allow instead
annotating a map_symbol, i.e. the top level hist_entry or one of the
callchains for which there were samples.
Suggested-by: Stephane Eranian <eranian@google.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Borislav Petkov <bp@suse.de>
Cc: David Ahern <dsahern@gmail.com>
Cc: Don Zickus <dzickus@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Link: http://lkml.kernel.org/n/tip-k1zxj5564je9jei4yd15ouwn@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Since hist_entry__delete() nowadays doesn't actually frees anything that
may be in use by the annotation code.
Eventually we will solve this for good by reference counting struct
symbol.
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Borislav Petkov <bp@suse.de>
Cc: David Ahern <dsahern@gmail.com>
Cc: Don Zickus <dzickus@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/n/tip-uldtgljymtrkns0knpiso5op@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
perf fails to build with gcc "(GCC) 4.4.7 20120313 (Red Hat
4.4.7-4.0.9)" (a.k.a., RHEL6 / CentOS 6 / OL 6):
cc1: warnings being treated as errors
util/probe-event.c: In function ‘get_alternative_line_range’:
util/probe-event.c:359: error: missing initializer
util/probe-event.c:359: error: (near initialization for ‘pp.file’)
util/probe-event.c:359: error: missing initializer
util/probe-event.c:359: error: (near initialization for ‘result.function’)
Fix by bringing in initializers to declaration.
Signed-off-by: David Ahern <david.ahern@oracle.com>
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Link: http://lkml.kernel.org/r/1426084580-60780-1-git-send-email-david.ahern@oracle.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
When zoom into thread/dso/symbol, the fold/unfold stat is cleared in
hists__filter_by_thread/dso/symbol(), but h->nr_rows is not cleared. So
if we toggle fold stat on the unfold entires, nr_entries got a wrong
value.
This bug can be reproduced as follows:
$ perf record -g -e syscalls:sys_enter_open ls
$ perf report
Children Self Command Shared Object Symbol
================================================================
+ 50.00% 0.00% ls ld64.so [.] _dl_get_ready_to_run
- 50.00% 0.00% ls ld64.so [.] _dl_load_shared_library
_dl_load_shared_library <= [Zoom into thread/dso]
_dl_get_ready_to_run
_start
...
In the new thread hists, all entries reset to fold, if we unfold the
same entry as we previously unfolded, nr_entries got wrong value, and we
can't move down cursor to bottom row.
Thread: ls
Children Self Command Shared Object Symbol
================================================================
+ 50.00% 0.00% ls ld64.so [.] _dl_get_ready_to_run
- 50.00% 0.00% ls ld64.so [.] _dl_load_shared_library
_dl_load_shared_library
_dl_get_ready_to_run <= [cursor may stop here, can't move down]
_start
...
This patch clear h->nr_rows to fix this bug.
Signed-off-by: He Kuang <hekuang@huawei.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/1426077363-855-2-git-send-email-hekuang@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
A double free occurred when get source file path failed. If lr->path
failed to assign a new value, it will be freed as the old path and then
be freed again during line_range__clear(), and causes this:
$ perf probe -L do_execve -k vmlinux
*** Error in `/usr/bin/perf': double free or corruption (fasttop):
0x0000000000a9ac50 ***
======= Backtrace: =========
../lib64/libc.so.6(+0x6eeef)[0x7ffff5e44eef]
../lib64/libc.so.6(+0x78cae)[0x7ffff5e4ecae]
../lib64/libc.so.6(+0x79987)[0x7ffff5e4f987]
../bin/perf[0x4ab41f]
...
This patch fix this problem.
Signed-off-by: He Kuang <hekuang@huawei.com>
Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/1425463302-1687-1-git-send-email-hekuang@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
It currently prevents adding probes in weak symbols. But there're cases
that given name is an only weak symbol so that we cannot add probe.
$ perf probe -x /usr/lib/libc.so.6 -a calloc
Failed to find symbol calloc in /usr/lib/libc-2.21.so
Error: Failed to add events.
$ nm /usr/lib/libc.so.6 | grep calloc
000000000007b1f0 t __calloc
000000000007b1f0 T __libc_calloc
000000000007b1f0 W calloc
This change will result in duplicate probes when strong and weak symbols
co-exist in a binary. But I think it's not a big problem since probes
at the weak symbol will never be hit anyway.
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Naohiro Aota <naota@elisp.net>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20150306073129.6904.41078.stgit@localhost.localdomain
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
When perf probe tries to add a probe in a binary using symbol name, it
sometimes failed since some symbols were discard during loading dso.
When it resolves an address to symbol, it'd be better to have just one
symbol at given address. But for finding address from symbol, it'd be
better to keep all names (including aliases).
So allow tools to state that they want to allow aliases via
symbol_conf.allow_aliases.
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Naohiro Aota <naota@elisp.net>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20150306073127.6904.3232.stgit@localhost.localdomain
[ Original patch passwd allow_alias to many functions, use symbol_conf.allow_aliases instead ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
This reverts commit 906451b98b ("perf probe: Fix to fall back to find probe point in symbols").
Since 'perf probe' now retries with the address of given symbol searched from
map before this path, this fall back routine isn't needed anymore.
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Naohiro Aota <naota@elisp.net>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20150306073124.6904.1751.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Fix perf probe --line to handle aliased symbols correctly in glibc.
This makes line_range search failing back to address-based alternative
search as same as --add and --vars.
Without this patch;
-----
# ./perf probe -x /usr/lib64/libc-2.17.so -L malloc
Specified source line is not found.
Error: Failed to show lines.
-----
With this patch;
-----
# ./perf probe -x /usr/lib64/libc-2.17.so -L malloc
<__libc_malloc@/usr/src/debug/glibc-2.17-c758a686/malloc/malloc.c:0>
0 __libc_malloc(size_t bytes)
1 {
mstate ar_ptr;
void *victim;
__malloc_ptr_t (*hook) (size_t, const __malloc_ptr_t)
6 = force_reg (__malloc_hook);
7 if (__builtin_expect (hook != NULL, 0))
8 return (*hook)(bytes, RETURN_ADDRESS (0));
10 arena_lookup(ar_ptr);
12 arena_lock(ar_ptr, bytes);
-----
Note that this actually shows __libc_malloc, since it is the real
instance of malloc. User can use both __libc_malloc and malloc for
--line.
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Naohiro Aota <naota@elisp.net>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20150306073122.6904.18540.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Fix perf probe to handle aliased symbols correctly in glibc. In the
glibc, several symbols are defined as an alias of __libc_XXX, e.g.
malloc is an alias of __libc_malloc.
In such cases, dwarf has no subroutine instances of the alias functions
(e.g. no "malloc" instance), but the map has that symbol and its
address.
Thus, if we search the alieased symbol in debuginfo, we always fail to
find it, but it is in the map.
To solve this problem, this fails back to address-based alternative
search, which searches the symbol in the map, translates its address to
alternative (correct) function name by using debuginfo, and retry to
find the alternative function point from debuginfo.
This adds fail-back process to --vars, --lines and --add options. So,
now you can use those on malloc@libc :)
Without this patch;
-----
# ./perf probe -x /usr/lib64/libc-2.17.so -V malloc
Failed to find the address of malloc
Error: Failed to show vars.
# ./perf probe -x /usr/lib64/libc-2.17.so -a "malloc bytes"
Probe point 'malloc' not found in debuginfo.
Error: Failed to add events.
-----
With this patch;
-----
# ./perf probe -x /usr/lib64/libc-2.17.so -V malloc
Available variables at malloc
@<__libc_malloc+0>
size_t bytes
# ./perf probe -x /usr/lib64/libc-2.17.so -a "malloc bytes"
Added new event:
probe_libc:malloc (on malloc in /usr/lib64/libc-2.17.so with bytes)
You can now use it in all perf tools, such as:
perf record -e probe_libc:malloc -aR sleep 1
-----
Reported-by: Arnaldo Carvalho de Melo <acme@kernel.org>
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Naohiro Aota <naota@elisp.net>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20150306073120.6904.13779.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
From perf_session, will be used in 'trace'.
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Borislav Petkov <bp@suse.de>
Cc: David Ahern <dsahern@gmail.com>
Cc: Don Zickus <dzickus@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/n/tip-mfihndzaumx44h6y37ng2irb@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
It is set by calling thread__set_comm right before the removed line.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/1425396581-17716-2-git-send-email-jolsa@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
It is all about flushing the ordered queue or piping it thru, no need
for a perf_session pointer.
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Borislav Petkov <bp@suse.de>
Cc: David Ahern <dsahern@gmail.com>
Cc: Don Zickus <dzickus@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/n/tip-g47fx3ys0t9271cp0dcabjc7@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
So that we can simplify the deliver method to pass just:
(ordered_events, ordered_event, sample);
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Borislav Petkov <bp@suse.de>
Cc: David Ahern <dsahern@gmail.com>
Cc: Don Zickus <dzickus@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/n/tip-j0s4bpxs5qza5tnkvjwom9rw@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
By keeping pointers to machines, evlist and tool in ordered_events.
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Borislav Petkov <bp@suse.de>
Cc: David Ahern <dsahern@gmail.com>
Cc: Don Zickus <dzickus@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/n/tip-0c6huyaf59mqtm2ek9pmposl@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
For use by tools that are not perf.data based, as maybe 'perf trace' in
live mode.
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Borislav Petkov <bp@suse.de>
Cc: David Ahern <dsahern@gmail.com>
Cc: Don Zickus <dzickus@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/n/tip-nedqe7cmii5w82etfi36urfz@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
When annotating source/disasm lines the perf tools parse the output of
objdump, trying to provide augmented output that allows navigating
jumps, calls, etc.
But when a line output by objdump can't be parsed the annotation code
falls back to just presenting the unparsed line.
When fixing a leak in the 0fb9f2aab7 commit ("perf annotate: Fix
memory leaks in LOCK handling") we failed to take that into account and
instead tried to free one of the data structures that should be freed
only when successfully allocated, oops, segfault.
There was a change in the way the objdump output for lock prefixed
instructions is formatted that lead the relevant parser to fail to grok
it.
At least RHEL7 works ok, but Fedora 20 segfaults.
Fix it by making the ins__delete() destructor work like the most basic
destructor: free().
Namely make it accept a NULL pointer and when handling it just do
nothing.
Further investigation is needed to figure out the nature of the objdump
output change so as to make the parser grok it.
Reported-by: Linus Torvalds <torvalds@linux-foundation.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Rabin Vincent <rabin@rab.in>
Link: http://lkml.kernel.org/n/tip-7wsy0zo292pif0yjoqpfryrz@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
We need to do that to stop accumulating entries in the dead_threads
linked list, i.e. we were keeping references to threads in struct hists
that continue to exist even after a thread exited and was removed from
the machine threads rbtree.
We still keep the dead_threads list, but just for debugging, allowing us
to iterate at any given point over the threads that still are referenced
by things like struct hist_entry.
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Borislav Petkov <bp@suse.de>
Cc: David Ahern <dsahern@gmail.com>
Cc: Don Zickus <dzickus@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/n/tip-3ejvfyed0r7ue61dkurzjux4@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Remove bias offset to find probe point by address.
Without this patch, probe points on kernel and executables are shown
correctly, but do not work with libraries:
# ./perf probe -l
probe:do_fork (on do_fork@kernel/fork.c)
probe_libc:malloc (on malloc in /usr/lib64/libc-2.17.so)
probe_perf:strlist__new (on strlist__new@util/strlist.c in /home/mhiramat/ksrc/linux-3/tools/perf/perf)
Removing bias allows it to show it as real place:
# ./perf probe -l
probe:do_fork (on do_fork@kernel/fork.c)
probe_libc:malloc (on __libc_malloc@malloc/malloc.c in /usr/lib64/libc-2.17.so)
probe_perf:strlist__new (on strlist__new@util/strlist.c in /home/mhiramat/ksrc/linux-3/tools/perf/perf)
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Naohiro Aota <naota@elisp.net>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20150302124946.9191.64085.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Warn if given uprobe event accesses memory on older kernel.
Until 3.14, uprobe event only supports accessing registers so this warns
to upgrade kernel if uprobe-event returns -EINVAL and an argument of the
event accesses memory ($stack, @+offset, and +|-offs() symtax).
With this patch (on 3.10.0-123.13.2.el7.x86_64);
-----
# ./perf probe -x ./perf warn_uprobe_event_compat stack=-0\(%sp\)
Added new event:
Failed to write event: Invalid argument
Please upgrade your kernel to at least 3.14 to have access to feature -0(%sp)
Error: Failed to add events.
-----
Suggested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/20150228025329.32106.70581.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
