KFT
&
Tracing Collaboration
Tim Bird
Sony Electronics
July 28, 2017
1
KFT Quickie Overview
• KFT = Kernel Function Trace
– Uses a compiler switch (-finstrument_functions) to
instrument EVERY function in kernel
– Low-intrusion patch to kernel source
– Optimized for finding latency problems
• Filters for long-duration function calls
• Maintains a function stack from which short duration functions
are constantly popped
• Historically used to find places to improve bootup
time
July 28, 2017
2
KFT Example
• kftstatic.conf
trigger start entry start_kernel
trigger end entry to_userspace
filter mintime 500
• Output: (cat /proc/kft_data | addr2sym -m System.map)
Entry Delta PID Function
Called At
----- ------ --- ------------------------- -----------------1
0
0 start_kernel
L6+0x0
14
8687
0 setup_arch
start_kernel+0x35
39
891
0 setup_memory
setup_arch+0x2a8
53
872
0 register_bootmem_low_pages setup_memory+0x8f
...
July 28, 2017
3
More Information
•
•
•
•
Has triggers, filters
Can trace very early - from first kernel instruction
Uses procfs for control and data (everything ASCII)
Post-processing tool shows:
– functions sorted by duration, call count, local time, etc.
– call tree, including interrupts
• Google: “KernelFunctionTrace” (all one word)
July 28, 2017
4
Tracing Collaboration
July 28, 2017
5
Why Collaborate
• Want to avoid duplicating effort
• Want to avoid conflicts and re-inforce efforts
– E.g. Kernel will support AT MOST one set of static
tracepoints
• Want to produce awesome tools with great
capabilities and usefulness
• It’s open source, dang-it!
• How to proceed?
– Groups have talked before - was there any outcome?
July 28, 2017
6
Tracing Terminology
• Phases:
– trace definition/configuration
– trace generation
• activation, control, capture, termination
– post-processing
• formatting the data and visualization
• Kinds of data gathering
– event-based
– sample-based
July 28, 2017
7
Tracing Terminology
•
•
•
•
•
•
•
•
•
tracepoint/probe point - a location in the traced software, which, when
encountered, “emits” an event
trace/probe handler - the code to execute when an event occurs or a sample is
taken
trace definition - the set of tracepoints, handlers and constraints which
determine what events are collected and how they are processed
post-processing - manipulation of the trace data after the trace is collected
aggregation - updating statistics or other analytical information, based on trace
events or samples
filters - criteria used to limit the events that are captured or processed
triggers - criteria used to start and stop tracing automatically
visualization tool - tool to present trace data graphically to make it easier to
analyze
Tracepoints and trace handlers can be defined either in-source (as compiled-in,
or static tracepoints) or externally (inserted as dynamic tracepoints)
July 28, 2017
8
Tracer Comparison Chart
SystemTap
KFT
Specialized
tracers
July 28, 2017
LTTng
Flexibility
9
Tracer Comparison Table
Target user
KFT
LTTng
embedded
developer
system admin
Instrumentation every function, static definitions
SystemTAP
system admin
by compiler
in source
programmable, using
external definition
Control
interface
Overhead
echo and cat,
using /proc
lttctl and lttd,
using netlink
stap, using insmod
and kprobe
medium
low
high
Trace format
ascii, fixed
binary, with xml
schema
programmable, printf
and ascii-art
Post-processing
kd, textoriented
lttv, graphical
systemtapgui??
July 28, 2017
10
Tracer Commonalities
• Ummm… <crickets chirping>
July 28, 2017
11
Ideas
• Integrate KFT into LTTng
• Have LTTng support externally-defined, but
compiled-in, tracepoints
• Stabilize trace buffer mechanism in kernel
• Build best-of-systemtap functions module,
which can be compiled-in
• Standardize trace file format
• What else???
July 28, 2017
12