linux/kernel/trace
Steven Rostedt (VMware) 097350d1c6 ring-buffer: Zero out time extend if it is nested and not absolute
Currently the ring buffer makes events that happen in interrupts that preempt
another event have a delta of zero. (Hopefully we can change this soon). But
this is to deal with the races of updating a global counter with lockless
and nesting functions updating deltas.

With the addition of absolute time stamps, the time extend didn't follow
this rule. A time extend can happen if two events happen longer than 2^27
nanoseconds appart, as the delta time field in each event is only 27 bits.
If that happens, then a time extend is injected with 2^59 bits of
nanoseconds to use (18 years). But if the 2^27 nanoseconds happen between
two events, and as it is writing the event, an interrupt triggers, it will
see the 2^27 difference as well and inject a time extend of its own. But a
recent change made the time extend logic not take into account the nesting,
and this can cause two time extend deltas to happen moving the time stamp
much further ahead than the current time. This gets all reset when the ring
buffer moves to the next page, but that can cause time to appear to go
backwards.

This was observed in a trace-cmd recording, and since the data is saved in a
file, with trace-cmd report --debug, it was possible to see that this indeed
did happen!

  bash-52501   110d... 81778.908247: sched_switch:         bash:52501 [120] S ==> swapper/110:0 [120] [12770284:0x2e8:64]
  <idle>-0     110d... 81778.908757: sched_switch:         swapper/110:0 [120] R ==> bash:52501 [120] [509947:0x32c:64]
 TIME EXTEND: delta:306454770 length:0
  bash-52501   110.... 81779.215212: sched_swap_numa:      src_pid=52501 src_tgid=52388 src_ngid=52501 src_cpu=110 src_nid=2 dst_pid=52509 dst_tgid=52388 dst_ngid=52501 dst_cpu=49 dst_nid=1 [0:0x378:48]
 TIME EXTEND: delta:306458165 length:0
  bash-52501   110dNh. 81779.521670: sched_wakeup:         migration/110:565 [0] success=1 CPU:110 [0:0x3b4:40]

and at the next page, caused the time to go backwards:

  bash-52504   110d... 81779.685411: sched_switch:         bash:52504 [120] S ==> swapper/110:0 [120] [8347057:0xfb4:64]
CPU:110 [SUBBUFFER START] [81779379165886:0x1320000]
  <idle>-0     110dN.. 81779.379166: sched_wakeup:         bash:52504 [120] success=1 CPU:110 [0:0x10:40]
  <idle>-0     110d... 81779.379167: sched_switch:         swapper/110:0 [120] R ==> bash:52504 [120] [1168:0x3c:64]

Link: https://lkml.kernel.org/r/20200622151815.345d1bf5@oasis.local.home

Cc: Ingo Molnar <mingo@kernel.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Tom Zanussi <zanussi@kernel.org>
Cc: stable@vger.kernel.org
Fixes: dc4e2801d4 ("ring-buffer: Redefine the unimplemented RINGBUF_TYPE_TIME_STAMP")
Reported-by: Julia Lawall <julia.lawall@inria.fr>
Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
2020-06-23 11:18:42 -04:00
..
blktrace.c blktrace: Avoid sparse warnings when assigning q->blk_trace 2020-06-17 09:07:11 -06:00
bpf_trace.c maccess: rename probe_user_{read,write} to copy_{from,to}_user_nofault 2020-06-17 10:57:41 -07:00
fgraph.c tracing: Define MCOUNT_INSN_SIZE when not defined without direct calls 2020-01-02 21:56:44 -05:00
ftrace_internal.h x86/ftrace: Have ftrace trampolines turn read-only at the end of system boot up 2020-05-12 18:24:34 -04:00
ftrace.c ftrace: Fix maybe-uninitialized compiler warning 2020-06-17 17:13:18 -04:00
Kconfig Tracing updates for 5.8: 2020-06-09 10:06:18 -07:00
kprobe_event_gen_test.c tracing: Add kprobe event command generation test module 2020-01-30 09:46:28 -05:00
Makefile Rebase locking/kcsan to locking/urgent 2020-06-11 20:02:46 +02:00
power-traces.c
preemptirq_delay_test.c tracing: Wait for preempt irq delay thread to execute 2020-05-11 17:00:34 -04:00
ring_buffer_benchmark.c tracing: Make struct ring_buffer less ambiguous 2020-01-13 13:19:38 -05:00
ring_buffer.c ring-buffer: Zero out time extend if it is nested and not absolute 2020-06-23 11:18:42 -04:00
rpm-traces.c
synth_event_gen_test.c tracing: Have synthetic event test use raw_smp_processor_id() 2020-02-20 17:43:41 -05:00
trace_benchmark.c trace: Use pr_warn instead of pr_warning 2019-10-18 15:01:57 +02:00
trace_benchmark.h
trace_boot.c tracing/boottime: Fix kprobe event API usage 2020-05-06 09:04:11 -04:00
trace_branch.c tracing: Make struct ring_buffer less ambiguous 2020-01-13 13:19:38 -05:00
trace_clock.c
trace_dynevent.c tracing: Use seq_buf for building dynevent_cmd string 2020-02-01 13:10:15 -05:00
trace_dynevent.h tracing: Remove check_arg() callbacks from dynevent args 2020-02-01 13:09:23 -05:00
trace_entries.h tracing: Make ftrace packed events have align of 1 2020-06-16 21:21:02 -04:00
trace_event_perf.c Merge branch 'perf/urgent' into perf/core, to pick up fixes 2019-10-28 12:38:26 +01:00
trace_events_filter_test.h
trace_events_filter.c tracing: Avoid memory leak in process_system_preds() 2019-12-19 18:24:17 -05:00
trace_events_hist.c tracing: Move synthetic events to a separate file 2020-06-01 08:23:22 -04:00
trace_events_inject.c tracing: Initialize val to zero in parse_entry of inject code 2020-01-02 19:04:57 -05:00
trace_events_synth.c tracing: Move synthetic events to a separate file 2020-06-01 08:23:22 -04:00
trace_events_trigger.c tracing: Fix the race between registering 'snapshot' event trigger and triggering 'snapshot' operation 2020-04-14 22:02:10 -04:00
trace_events.c tracing: Add hist_debug trace event files for histogram debugging 2020-06-01 08:22:30 -04:00
trace_export.c tracing: Make ftrace packed events have align of 1 2020-06-16 21:21:02 -04:00
trace_functions_graph.c ring-buffer: Rename ring_buffer_read() to read_buffer_iter_advance() 2020-03-19 19:11:19 -04:00
trace_functions.c trace: Fix typo in allocate_ftrace_ops()'s comment 2020-06-16 21:21:02 -04:00
trace_hwlat.c tracing: Have hwlat ts be first instance and record count of instances 2020-03-03 17:33:43 -05:00
trace_irqsoff.c tracing: Rename trace_buffer to array_buffer 2020-01-13 13:19:38 -05:00
trace_kdb.c tracing: Rename trace_buffer to array_buffer 2020-01-13 13:19:38 -05:00
trace_kprobe_selftest.c
trace_kprobe_selftest.h
trace_kprobe.c maccess: rename probe_user_{read,write} to copy_{from,to}_user_nofault 2020-06-17 10:57:41 -07:00
trace_mmiotrace.c tracing: Make struct ring_buffer less ambiguous 2020-01-13 13:19:38 -05:00
trace_nop.c
trace_output.c mmap locking API: use coccinelle to convert mmap_sem rwsem call sites 2020-06-09 09:39:14 -07:00
trace_output.h
trace_preemptirq.c x86/entry: Rename trace_hardirqs_off_prepare() 2020-06-11 15:15:24 +02:00
trace_printk.c tracing: Add locked_down checks to the open calls of files created for tracefs 2019-10-12 20:48:06 -04:00
trace_probe_tmpl.h tracing/probe: Support user-space dereference 2019-05-25 23:04:42 -04:00
trace_probe.c tracing/probe: Fix memleak in fetch_op_data operations 2020-06-16 21:21:02 -04:00
trace_probe.h tracing/probe: Replace zero-length array with flexible-array 2020-06-15 23:08:32 -05:00
trace_sched_switch.c tracing: Fix sched switch start/stop refcount racy updates 2020-01-30 09:46:10 -05:00
trace_sched_wakeup.c tracing: Make struct ring_buffer less ambiguous 2020-01-13 13:19:38 -05:00
trace_selftest_dynamic.c
trace_selftest.c tracing: Rename trace_buffer to array_buffer 2020-01-13 13:19:38 -05:00
trace_seq.c tracing: Remove unused TRACE_SEQ_BUF_USED 2020-01-21 18:39:54 -05:00
trace_stack.c trace: fix an incorrect __user annotation on stack_trace_sysctl 2020-06-08 10:13:56 -04:00
trace_stat.c tracing: Fix tracing_stat return values in error handling paths 2020-01-24 18:06:48 -05:00
trace_stat.h tracing: Use generic type for comparator function 2019-11-14 13:15:11 -05:00
trace_synth.h tracing: Move synthetic events to a separate file 2020-06-01 08:23:22 -04:00
trace_syscalls.c Tracing updates: 2020-02-06 07:12:11 +00:00
trace_uprobe.c tracing/probe: Fix bpf_task_fd_query() for kprobes and uprobes 2020-06-09 11:10:12 -07:00
trace.c tracing: Remove unused event variable in tracing_iter_reset 2020-06-16 21:21:03 -04:00
trace.h tracing: Make ftrace packed events have align of 1 2020-06-16 21:21:02 -04:00
tracing_map.c tracing: Convert local functions in tracing_map.c to static 2020-04-22 22:07:26 -04:00
tracing_map.h