strace
is awesome [1] [2] [3]. It lets you see exactly what system calls are being made by your running application. Wondering what configuration files the framework looks for? Want to know why the remote connection is hung up? strace
can help... if you're running Linux.
If you're running OS X on your development machine, you might assume you're out of luck: there's no strace
on OS X. But I've got good news for you: DTrace can do magic like strace
, and even more! And it comes with OS X (/usr/bin/dtrace
), so you can get started right away (though you'll have a few steps to go through first if you've upgraded to El Capitan [4]). Sorry Linux and Windows users—this post probably isn't the one you're looking for [5].
Below are a few nice tricks you can do with DTrace to impress your coworkers, and also to get better insights into what your system is really doing.
From strace
to dtrace
I've found I can emulate most strace
behaviors I want with DTrace, if I'm willing to put in the time. But there's a fancy built-in DTrace script (wrapped in a shell script) called dtruss
that lets you do the most common things without even learning the D programing language (no, not that one either—that's unrelated).
One of my most common strace
workflows is to take a running process (like a web server), and trace the open
system calls it sends to figure out which files it's opening:
$ sudo strace -e open -p $SERVER_PID
Process 10738 attached
open("/home/vagrant/practice/strace.txt", O_RDONLY) = 5
You can do the same thing with dtruss
:
$ sudo dtruss -t open_nocancel -p $SERVER_PID
SYSCALL(args) = return
open_nocancel(".\0", 0x0, 0x1) = 8 0
open_nocancel("/Users/colin/Practice/dtrace/runocc.d\0", 0x0, 0x1B6) = 8 0
open_nocancel("/usr/share/zoneinfo/UTC\0", 0x0, 0x0) = 9 0
open_nocancel(".\0", 0x0, 0x1) = 8 0
open_nocancel("/Users/colin/Practice/dtrace/foo\0", 0x0, 0x1B6) = -1 Err#2
So it's pretty similar to use, just a difference in flags (strace -e
vs. dtruss -t
) and system call names we're tracing (open
vs. open_nocancel
).
In the dtruss
version, we see the name of the system call, the arguments to that call, and the return value, along with an error code where applicable.
On Linux, you can run man syscalls
to get a full list of available system calls to trace. For OS X you can either find the list from the kernel source or just use DTrace for that:
$ sudo dtrace -ln 'syscall:::entry'
ID PROVIDER MODULE FUNCTION NAME
141 syscall syscall entry
143 syscall exit entry
145 syscall fork entry
147 syscall read entry
149 syscall write entry
151 syscall open entry
153 syscall close entry
155 syscall wait4 entry
### etc.
If you need to trace multiple system calls instead of just one, you can drop to using dtrace
instead of dtruss
, but it gets a little complicated if you want to retain information about arguments & return values like strace
gives you. We'll skip all that noise, but it's a one-liner if you can settle for the basics:
$ sudo dtrace -qn 'syscall::write:entry, syscall::sendto:entry /pid == $target/ { printf("(%d) %s %s", pid, probefunc, copyinstr(arg1)); }' -p $SERVER_PID
(41941) sendto HTTP/1.0 200 OK
(41941) sendto Server: SimpleHTTP/0.6 Python/2.7.10
(41941) sendto Date: Thu, 05 Nov 2015 04:44:00 GMT
(41941) sendto Content-type: text/html; charset=utf-8
(41941) sendto Content-Length: 350
(41941) sendto
I'll leave the full D language tutorial to others, but the gist of it is that you specify:
-
one or more probes (required), which are essentially the kinds of events you want to trace (the
write
andsendto
system calls, in our case) - a predicate (optional), which lets you filter events in the kernel and avoid passing them back up to user space (matching PID, in our case)
- an action (required), which lets you write the code that'll be executed when the events that you're interested in fire
What you see above is using the syscall
DTrace provider (the first part of the probe: syscall::write:entry
)—but you can do way more than just system calls with DTrace. And that's why people say it's more powerful (for some hazy definition of "better") than strace
—it's able to do a lot more things. Also, since DTrace executes in kernel context, it doesn't have to keep switching between user mode & kernel mode the way strace
does, so there's less context-switching overhead. This means you can even use DTrace on production systems (though I'm sure there are caveats, like only enabling the probes you need).
You don't have unlimited power in D (for example, no loops!), but there's a lot you can do very concisely, and keep in mind you're writing code to be executed at the kernel level—this is pretty amazing stuff!
Built-in OS X DTrace scripts
You can learn a ton about using DTrace by going through examples on Brendan Gregg's page, but you also already have a bunch of his examples on your computer (they shipped with your OS). Check them out:
$ ls /usr/bin/*.d
/usr/bin/bitesize.d /usr/bin/kill.d /usr/bin/seeksize.d
/usr/bin/cpu_profiler.d /usr/bin/loads.d /usr/bin/setuids.d
/usr/bin/cpuwalk.d /usr/bin/newproc.d /usr/bin/sigdist.d
/usr/bin/creatbyproc.d /usr/bin/pathopens.d /usr/bin/syscallbypid.d
/usr/bin/dispqlen.d /usr/bin/pidpersec.d /usr/bin/syscallbyproc.d
/usr/bin/filebyproc.d /usr/bin/priclass.d /usr/bin/syscallbysysc.d
/usr/bin/hotspot.d /usr/bin/pridist.d /usr/bin/timer_analyser.d
/usr/bin/httpdstat.d /usr/bin/runocc.d /usr/bin/weblatency.d
/usr/bin/iofile.d /usr/bin/rwbypid.d
/usr/bin/iofileb.d /usr/bin/rwbytype.d
Try digging into a few of these and seeing how they work. You'll probably be surprised at how compact they are—many are essentially one-liners.
For example, filebyproc.d
traces for files being opened, by process:
$ sudo filebyproc.d
dtrace: script '/usr/bin/filebyproc.d' matched 7 probes
CPU ID FUNCTION:NAME
2 937 open_nocancel:entry kextd //private/var/run/installd.commit.pid
1 937 open_nocancel:entry Google Chrome /var/folders/5s/bfpr9qhx1nb2lk1spyp78gm40000gn/T/.com.google.Chrome.K6nrPC
0 151 open:entry mds_stores .
0 151 open:entry mds_stores .
0 151 open:entry CrashPlanServic /Users/colin/javadoc/1.4.2/docs/api/java/awt/color
2 151 open:entry mds /Users/colin/javadoc/1.4.2/docs/api/java/awt/color
0 937 open_nocancel:entry CrashPlanServic /Users/colin/javadoc/1.4.2/docs/api/java/awt/color
0 151 open:entry CrashPlanServic /Users/colin/javadoc/1.4.2/docs/api/java/awt/color/class-use
You can see in the output above that I found CrashPlan (a backup service) backing up some JavaDocs from way back in version 1.4.2—why did I even have those?! Deleted.
This is a pretty cool feature, and it's literally only one line—we could do the exact same thing via dtrace
directly, and then customize it however we want to get further details:
$ sudo dtrace -n 'syscall::open*:entry { printf("%s %s", execname, copyinstr(arg0)); }'
Writing your own
It's pretty straightforward to write your own scripts from scratch, using other D scripts for inspiration. To start, try just tweaking existing scripts to slightly change the output or calculations. It's been instructive for me to do things like pick a feature I'd like to have (like Linux vmstat
's display of the current run queue), then pull ideas from a few scripts together to get even more insight than I was looking for.
Incidentally, I found on my machine, and those of a few coworkers, that the built-in D scripts related to the CPU run queues, in particular, mistakenly thought the run queues were always stuck at 0 regardless of actual load. But after a couple hours of spelunking through some OS X kernel headers, I was able to make 1-line changes to these files and get working versions of both runocc.d
and dispqlen.d
. I don't know much about the OS X kernel and how processors are grouped in sets, so there may be problems with my version, but what I see now reflects the system load a lot more than a bunch of blank lines or zeroes!
To me, this all speaks pretty highly about the ease of writing D scripts, and the ability to iterate quickly on new ideas (no long compile/install cycles)—I've only been looking into DTrace for a week at this point, and hadn't dug into kernel programming before either.
Going further
These are the main resources I'm looking at to continue learning more about DTrace. I'd be excited to have others join me on the ride—this stuff is super-fun!
References
[1] Julia Evans has a terrific set of blog posts on strace
, and even an strace
zine you can print out and carry around with you.
[2] Brendan Gregg also wrote up a nice intro to strace
.
[3] Chad Fowler did an strace
intro too.
[4] OS X 10.11, El Capitan, has some new security restrictions whose default configurations break DTrace (though some scripts may will still work). This article explains System Integrity Protection (SIP) and shows you how to get DTrace back in working order, among other things. Caveat: I haven't upgraded to El Capitan, but I'm assuming this approach works, based on seeing the same advice in other places as well.
[5] If you're on Linux, there is a DTrace for Linux project, but it seems like there's more documentation and community around tools like perf_events (an official, built-into-the-kernel Linux tool) and SystemTap. You should take a look at Brendan Gregg's Linux performance page to see what tools make the most sense for you. Windows users—sorry, I'm sure there are great tools there too, but I have no idea!