DTrace
A cheat-sheet on DTrace
In macOS, run csrutil enable --without dtrace
in
recovery mode (Cmd+R
during boot).
Command Line
- List all available probes in the system:
dtrace -l
- Execute in quiet mode:
dtrace -q
You can also add the following pragma:
#pragma D option quiet
- Enable the C pre-processor:
dtrace -C
- Inspect the arguments of a specific probe:
dtrace -lvn <probe>
One-Liners
- All read files in the system:
dtrace -n 'syscall::read:entry /execname != "dtrace"/ { @reads[execname, fds[arg0].fi_pathname] = count(); }'
- A distribution graph of the most often I/O requested sizes:
dtrace -n 'io:::start { @bytes = quantize(args[0]->b_bcount); }'
- The system calls (and the amount) used by each process:
dtrace -n 'syscall:::entry { @sc[execname, probefunc] = count(); }'
- What processes are opening what files:
dtrace -n 'syscall::open:entry { printf("%s %s", execname, copyinstr(arg0)); }'
- Kernel functions running on-CPU:
dtrace -n 'profile-997hz /arg0/ { @[func(arg0)] = count(); }
- User functions running on-CPU:
dtrace -n 'profile-997hz /arg1/ { @[execname, ufunc(arg1)] = count(); }'
- Processes running the most system calls:
dtrace -n 'syscall:::entry { @[pid, execname] = count(); }'
- Memory page-faults by process name:
dtrace -n 'vminfo:::as_fault { @mem[execname] = sum(arg0); }'
- Count requested
malloc
sizes in a process:
dtrace -n 'pid$target::malloc:entry { @[arg0] = count(); }' -p <PID>
- Distribution of requested
malloc
sizes in a process:
dtrace -n 'pid$target::malloc:entry { @ = quantize(arg0); }' -p <PID>
- User stack traces that resulted in the most heap memory requests:
dtrace -n 'pid$target::malloc:entry { @[ustack()] = sum(arg0); }' -p <PID>
- Disk I/O per second:
dtrace -n 'io:::start { @io = count(); } tick-1sec { printa("Disk I/Os per second: %@d \n", @io); trunc(@io); }'
Scripts
- The top 10 processes causing the most I/O, along with the corresponding files
#!/usr/sbin/dtrace -qs
fsinfo::: /execname != "dtrace"/ {
@[execname, probename, args[0]->fi_fs, args[0]->fi_pathname] = count();
}
dtrace:::END {
trunc(@, 10);
printf("%-16s %-8s %-8s %-42s %-8s\n", "EXEC", "FUNCTION", "FS TYPE", "PATH", "COUNT");
printa("%-16s %-8s %-8s %-42s %-@8d\n", @);
}
The D Language
From an inline script:
dtrace -n 'probe /predicate/ { actions }'
From the command line:
dtrace -s script.d
From a script file:
#!/usr/sbin/dtrace -s
probe
/predicate/
{
actions
}
More than one probe can be defined, separated by comma, if the predicate and actions remain the same.
You can enable all probes of a certain namespace. For
example, instead of saying syscall::exit:entry
, you
can enable probes for the entry points of all system calls as
syscall:::entry
. DTrace will interpret blank spaces
between colons as wildcards. You can also use wildcards. Like
syscall::read*:entry
.
A string can be checked for emptiness by comparing it with
NULL
.
Global variables might be accessed by probes firing from different CPUs, so they can become corrupted.
Thread local variables are stored inside the
self
namespace. For example
self->a
.
Clause local variables can only be used from a single action
group. These are stored inside the this
namespace.
These variables don’t need to be freed as they are automatically
destroyed once the probe is executed.
File Descriptors
Given a file descriptor (i.e. as an argument to the
write
system call), we can fetch a
fileinfo_t
structure with information about the
file descriptor using the fds
object.
The fileinfo_t
structure looks like this:
typedef struct fileinfo {
; // Basename of `fi_pathname`
string fi_name; // Dirname of `fi_pathname`
string fi_dirname; // Full path name
string fi_pathname; // Offset within the file
offset_t fi_offset; // Filesystem
string fi_fs; // Mount point of file system
string fi_mount}
For example, we can find what files we’re writing to with
write
as:
syscall::write:entry {
@[fds[arg0].fi_pathname] = count();
}
Probes
Probe names are specified like this:
provider:module:function:name
provider
: The library providing such probemodule
: The kernel module or shared object library containing the probefunction
: The software function that contains this probename
: The descriptive name of the probe
dtrace:::BEGIN
Triggers at the start of the program. Useful to print output headers.
dtrace:::END
Triggers at the end of the program. Useful to print final reports.
profile:::profile-<number><unit>
Fires on every CPU at a rate of
<number><unit>
. For example
profile:::profile-199hz
.
This probe has two arguments: arg0
is the
program counter if running on the kernel, and arg1
is the program counter when running in user mode. Therefore if
arg0
is true, then we’re running in the kernel, and
vice-versa.
profile:::tick-<number><unit>
Fires on one CPU only, every
<number><unit>
. For example:
profile:::tick-1s
.
Globals
The D language provides the following variables in predicates:
arg0...arg9
: Probe arguments as unsigned 64-bit integers. Specific to each probeargs[]
: Typed probe argumentscpu
: Current CPU idcurpsinfo
: Process state info for the current threadcurthread
: OS structure for the current threaderrno
: Error value from the last system callexecname
: Program namepid
: Process IDppid
: Parent process IDprobeprov
: Provider name of the current probeprobemod
: Module name of the current probeprobefunc
: Function name of the current probeprobename
: Name of the current probestackdepth
: Current thread’s stack frame depthtid
: Current thread IDtimestamp
: Time since bootuid
: Real user IDuregs[]
: Current thread register valuesvtimestamp
: Current thread’s time in CPUwalltimestamp
: Epoch time
Macros
$target
: The process passed as-p <pid>
or-c <command>
$1..$N
: Command line arguments to the D script$$1..$$N
: Command line arguments to the D script, coerced to string
Aggregates
Variables prefixed by @
. For example:
@a = count();
They can be indexed, and the indexes appear as different columns:
@a[uid, probefunc] = count();
If the script only uses one aggregation, then you can just
use @
directly (without a name).
The result is automatically sorted in ascending order.
Aggregation Functions
count
: The number of times a probe was executedsum
: The total valueavg
: The average valuemin
: The smallest valuemax
: The largest valuestddev
: The standard deviationlquantize
: A linear frequency distributionquantize
: A power-of-two frequency distribution
Functions
trace()
Print a variable.
printf()
Print with formatting.
tracemem()
Print a region of memory.
Providers
These are libraries of probes. The most common ones are:
dtrace
: Housekeeping probessyscall
: Probes at the entry and return points of all system callsproc
: Probes for process and threads eventsprofile
: Probes for time-based data collectionfbt
: Boundary tracing, and probes at the entry and exit point of most kernel functionslockstat
: Probes for kernel synchronization primitivesio
: Probes for I/O tracing
You can see a list of probes from a particular provider as:
dtrace -l -P <provider>