When using 'perf script' to look at PT traces it is often useful to
ignore the initialization code at the beginning.
On larger traces which may have many millions of instructions in
initialization code doing that in a pipeline can be very slow, with perf
script spending a lot of CPU time calling printf and writing data.
This patch adds an extension to the --itrace argument that skips 'n'
events (instructions, branches or transactions) at the beginning. This
is much more efficient.
v2:
Add support for BTS (Adrian Hunter)
Document in itrace.txt
Fix branch check
Check transactions and instructions too
Committer note:
To test intel_pt one needs to make sure VT-x isn't active, i.e.
stopping KVM guests on the test machine, as described by Andi Kleen
at http://lkml.kernel.org/r/20160301234953.GD23621@tassilo.jf.intel.com
Signed-off-by: Andi Kleen <ak@linux.intel.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/1459187142-20035-1-git-send-email-andi@firstfloor.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
In 473398a21d ("perf tools: Add cpumode to struct perf_sample"), I
missed some places where perf_sample fields are directly initialized in
addition to what is done in perf_evsel__parse_sample(), namely when
synthesizing PERF_RECORD_{MMAP*,COMM,FORK,EXIT} for pre-existing threads
and also in intel_pt and intel_bts when synthesizing events from
processor trace, the jitdump code also was affected, fix it.
The problem was noticed with running:
# perf record -e intel_pt//u true
# perf script
Where the samples wouldn't get resolved because perf_sample.cpumode
would be left as zero, i.e. PERF_RECORD_MISC_CPUMODE_UNKNOWN, not
resolving as kernel, hypervisor or user cpu modes.
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Stephane Eranian <eranian@google.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Fixes: 473398a21d ("perf tools: Add cpumode to struct perf_sample")
Link: http://lkml.kernel.org/n/tip-n5sdauxgk24d5nun8kuuu2mh@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
intel_pt_process_auxtrace_info() creates a pt->unknown_thread thread
that eventually needs to be freed by the last thread__put() on it, when
its refcount hits zero, which may happen in
intel_pt_process_auxtrace_info() error handling path and triggers the
following segfault, which would happen as well at intel_pt_free, when
tools using this intel_pt codebase frees up resources:
# perf record -I -e intel_pt/tsc=1,noretcomp=1/u /bin/ls
0 a anaconda-ks.cfg bin perf.data perf.data.old perf-f23-bringup.todo
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.217 MB perf.data ]
#
# perf script -F event,comm,pid,tid,time,addr,ip,sym,dso,iregs
Samples for 'instructions:u' event do not have IREGS attribute set. Cannot print 'iregs' field.
intel_pt_synth_events: failed to synthesize 'instructions' event type
Segmentation fault (core dumped)
#
The problem is: there's a union in 'struct thread' combines a list_head
and a rb_node. The standard life cycle of a thread is: init rb_node in
the constructor, insert it into machine->threads rbtree using rb_node,
move it to machine->dead_threads using list_head, clean in the last
thread__put: list_del_init(&thread->node).
In the above command, it clean a thread before adding it into list,
causes the above segfault.
Since pt->unknown_thread will never live in an rbtree, initialize its
list node so that when list_del_init() is done on it we don't segfault.
After this patch:
# perf script -F event,comm,pid,tid,time,addr,ip,sym,dso,iregs
Samples for 'instructions:u' event do not have IREGS attribute set. Cannot print 'iregs' field.
intel_pt_synth_events: failed to synthesize 'instructions' event type
0x248 [0x88]: failed to process type: 70
#
Reported-by: Tong Zhang <ztong@vt.edu>
Reported-by: Wang Nan <wangnan0@huawei.com>
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Link: http://lkml.kernel.org/r/1454296865-19749-1-git-send-email-wangnan0@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
We should always return from thread__new(), the constructor, with the
object with a reference count of one, so that:
struct thread *thread = thread__new();
thread__put(thread);
Will call thread__delete().
If any reference is made to that 'thread' variable, it better use
thread__get(thread) to hold a reference.
We were returning with thread->refcnt set to zero, fix it and some cases
where thread__delete() was being called, which were not a problem
because just one reference was being used, now that we set it to 1, use
thread__put() instead.
Reported-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/n/tip-4b9mkuk66to4ecckpmpvqx6s@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
And it is also a step in the direction of killing the separation of data
and text maps in map_groups.
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/n/tip-rrds86kb3wx5wk8v38v56gw8@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
autofdo incorrectly expects branch flags to include either mispred or
predicted. In fact mispred = predicted = 0 is valid and means the flags
are not supported, which they aren't by Intel PT.
To make autofdo work, add a config option which will cause Intel PT
decoder to set the mispred flag on all branches.
Below is an example of using Intel PT with autofdo. The example is
also added to the Intel PT documentation. It requires autofdo
(https://github.com/google/autofdo) and gcc version 5. The bubble
sort example is from the AutoFDO tutorial (https://gcc.gnu.org/wiki/AutoFDO/Tutorial)
amended to take the number of elements as a parameter.
$ gcc-5 -O3 sort.c -o sort_optimized
$ ./sort_optimized 30000
Bubble sorting array of 30000 elements
2254 ms
$ cat ~/.perfconfig
[intel-pt]
mispred-all
$ perf record -e intel_pt//u ./sort 3000
Bubble sorting array of 3000 elements
58 ms
[ perf record: Woken up 2 times to write data ]
[ perf record: Captured and wrote 3.939 MB perf.data ]
$ perf inject -i perf.data -o inj --itrace=i100usle --strip
$ ./create_gcov --binary=./sort --profile=inj --gcov=sort.gcov -gcov_version=1
$ gcc-5 -O3 -fauto-profile=sort.gcov sort.c -o sort_autofdo
$ ./sort_autofdo 30000
Bubble sorting array of 30000 elements
2155 ms
Note there is currently no advantage to using Intel PT instead of LBR,
but that may change in the future if greater use is made of the data.
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/1443186956-18718-26-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Add support for generating branch stack context for PT samples. The
decoder reports a configurable number of branches as branch context for
each sample. Internally it keeps track of them by using a simple sliding
window. We also flush the last branch buffer on each sample to avoid
overlapping intervals.
This is useful for:
- Reporting accurate basic block edge frequencies through the perf
report branch view
- Using with --branch-history to get the wider context of samples
- Other users of LBRs
Also the Documentation is updated.
Examples:
Record with Intel PT:
perf record -e intel_pt//u ls
Branch stacks are used by default if synthesized so:
perf report --itrace=ile
is the same as:
perf report --itrace=ile -b
Branch history can be requested also:
perf report --itrace=igle --branch-history
Based-on-patch-by: Andi Kleen <ak@linux.intel.com>
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/1443186956-18718-15-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
intel_pt_synth_branch_sample() skips synthesizing if the branch does not
match the branch filter. That logic was sitting in the middle of the
function but is more efficiently placed at the start of the function, so
move it.
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/1443186956-18718-14-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Instruction tracing options (i.e. --itrace) include an option for
sampling instructions at an arbitrary period. e.g.
--itrace=i10us
means make an 'instructions' sample for every 10us of trace.
Currently the logic does not distinguish between a period of
zero and no period being specified at all, so it gets treated
as the default period which is 100000. That doesn't really
make sense.
Fix it so that zero period is accepted and treated as meaning
"as often as possible".
In the case of Intel PT that is the same as a period of 1 and
a unit of 'instructions' (i.e. --itrace=i1i).
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/1443186956-18718-2-git-send-email-adrian.hunter@intel.com
[ Add a few lines describing this in the Documentation/intel-pt.txt file ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
User visible:
- Use PERF_RECORD_SWITCH when available in intel-pt, instead of
"sched:sched_switch" events, enabling an unprivileged user to trace
multi-threaded or multi-process workloads (Adrian Hunter)
- Always use non inlined file name for 'srcfile' sort key (Andi Kleen)
- Quieten failed to read counter message, helps in systems without
backend-stalled-cycles (Andi Kleen)
Infrastructure:
- Add a 'perf test' entry for decoding of new x86 instructions (Adrian Hunter)
- Add new instructions (sha, clflushopt, clwb, pcommit, rdpkru, wrpkru, xsavec,
xsaves, xrstors) to the x86 instruction decoder (Adrian Hunter)
- Add a build test to warn when source code drifts happen for the
instruction decoder files in the kernel and in tools/perf (Adrian Hunter)
- Copy linux/filter.h to tools/include (He Kuang)
- Support function __get_dynamic_array_len in libtraceevent (He Kuanguuu)
- Tracing path finding/mounting/error reporting refactorings (Jiri Olsa)
- Store CPU socket and core IDs in perf.data (Kan Liang)
- Reorganize add/del probe insertion routines in 'perf probe' (Namhyung Kim, Wang Nan)
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQIcBAABAgAGBQJV6ciXAAoJENZQFvNTUqpAyQwP/0X+yVHyXFbwsLQSWtESeu/V
HCgep/4CA6dTG/Si6bIVqQpVAQ3446tTP4L1OlKB4qvkO2iQdXTm/oGfzkLAuvsf
sbl6//fyDQrMEDWhJYs/0C2aNVc6viSd3nYZTwtkM4Khs9L3QMDBCHfFxgETOq66
KKtddMc74qQTkSAC+xNCGoIL9bOrud414kM+XZ5XtrKwHHj4DUqIl8v39giGi/TI
Gu4vr+NuVtY8PHi1ApOT1GY4laAgbOmf4ITa+e2QwxtSybQvBwhLZwvLs1gTSjih
u45Qk3yJNYbExwVILV8gN/J8Uy2RqBZz6dlR4ejTeMF/YHAJZAct5ZCT0ebZI4+2
P4eqg4C0rE9z9EZWpa/hM21QTMOHd9zXn3NLPZXd209VrFRvxWqcXhLg7qBtQjMS
JQBZbszv/tD3wmCZWG1uyzFWsTimiutm8XUfXV5e1UPGpk5G8ZthTtouq+J2L3Yq
U4Spaox76eVEGFFBAEt7eBQVjZMDa68/telhQb9bnmGffpth5rnEf75L7bhuLznj
dtudYN+FaXT7BkljvyxlUdM3RLR3zN4FmrXCKRjKa8hgFMM8qOh/uL6C05CBx5hT
4flaE02K2xc3TaY/FshB7EZRpBEWuYz0ITum6IKB+SxvdT3yVKHrQWbXdTXqSFMa
7BuYILR9CcKGQmXrWliy
=uRtf
-----END PGP SIGNATURE-----
Merge tag 'perf-core-for-mingo' of git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux into perf/core
Pull perf/core improvements and fixes from Arnaldo Carvalho de Melo:
User visible changes:
- Use PERF_RECORD_SWITCH when available in intel-pt, instead of
"sched:sched_switch" events, enabling an unprivileged user to trace
multi-threaded or multi-process workloads. (Adrian Hunter)
- Always use non inlined file name for 'srcfile' sort key. (Andi Kleen)
- Quieten failed to read counter message, helps in systems without
backend-stalled-cycles. (Andi Kleen)
Infrastructure changes:
- Add a 'perf test' entry for decoding of new x86 instructions. (Adrian Hunter)
- Add new instructions (sha, clflushopt, clwb, pcommit, rdpkru, wrpkru, xsavec,
xsaves, xrstors) to the x86 instruction decoder. (Adrian Hunter)
- Add a build test to warn when source code drifts happen for the
instruction decoder files in the kernel and in tools/perf. (Adrian Hunter)
- Copy linux/filter.h to tools/include. (He Kuang)
- Support function __get_dynamic_array_len in libtraceevent. (He Kuanguuu)
- Tracing path finding/mounting/error reporting refactorings. (Jiri Olsa)
- Store CPU socket and core IDs in perf.data. (Kan Liang)
- Reorganize add/del probe insertion routines in 'perf probe'. (Namhyung Kim, Wang Nan)
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Add support for selecting and processing PERF_RECORD_SWITCH events for
use by Intel PT. If they are available, they will be used in preference
to sched_switch events.
This enables an unprivileged user to trace multi-threaded or
multi-process workloads with any level of perf_event_paranoid. However
it depends on kernel support for PERF_RECORD_SWITCH.
Without this patch, tracing a multi-threaded workload will decode
without error but all the data will be attributed to the main thread.
Without this patch, tracing a multi-process workload will result in
decoder errors because the decoder will not know which executable is
executing.
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/1439458857-30636-3-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
In a couple of cases the 'comm' member of 'union event' has been used
instead of the correct member ('fork') when processing exit events.
In the cases where it has been used incorrectly, only the 'pid' and
'tid' are affected. The 'pid' value would be correct anyway because it
is in the same position in 'comm' and 'fork' events, but the 'tid' would
have been incorrectly assigned from 'ppid'.
However, for exit events, the kernel puts the current task in the 'ppid'
and 'ttid' which is the same as the exiting task. That is 'ppid' ==
'pid' and if the task is not multi-threaded, 'pid' == 'tid' i.e. the
data goes wrong only when tracing multi-threaded programs.
It is hard to find an example of how this would produce an error in
practice. There are 3 occurences of the fix:
1. perf script is only affected if !sample_id_all which only happens on
old kernels.
2. intel_pt is only affected when decoding without timestamps
and would probably still decode correctly - the exit event is
only used to flush out data which anyway gets flushed at the
end of the session
3. intel_bts also uses the exit event to flush data which
would probably not cause errors as it would get flushed at
the end of the session instead
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/1439888825-27708-1-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Record additional information in the AUXTRACE_INFO event in preparation
for decoding MTC and CYC packets. Pass the information to the decoder.
The AUXTRACE_INFO record can be extended by using the size to indicate
the presence of new members.
The additional information includes PMU config bit positions and the TSC
to CTC (hardware crystal clock) ratio needed to decode MTC packets.
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/1437150840-31811-20-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
The period on synthesized 'instructions' samples was being set to a
fixed value, whereas the correct value is the number of instructions
since the last sample, which is a value that the decoder can provide.
So do it that way.
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/1437150840-31811-14-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Events that don't sample the timestamp have a timestamp value of -1.
Intel PT processing wasn't taking that into account.
This is particularly noticeable with Intel BTS because timestamps are
not requested by default.
Then, if the conversion of -1 to TSC results in a small number, the
processing is unaffected.
However if the conversion results in a big number, then the data is
processed prematurely before relevant sideband data like mmap events,
which in turn results in samples with unknown dsos.
Commiter note:
Since BTS wasn't upstream, I split the patch to fold the BTS part with
the patch introducing it, to avoid having this bug in the commit
history. PT was already upstream, so this patch contains that part.
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/1440060692-5585-1-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Add support for Intel Processor Trace.
Intel PT support fits within the new auxtrace infrastructure. Recording
is supporting by identifying the Intel PT PMU, parsing options and
setting up events.
Decoding is supported by queuing up trace data by cpu or thread and then
decoding synchronously delivering synthesized event samples into the
session processing for tools to consume.
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/1437150840-31811-7-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
