If you're running OS X on your development machine, you might assume you're out of luck: there's no strace on OS X. But I've got good news for you: DTrace can do magic like strace, and even more! And it comes with OS X (/usr/bin/dtrace), so you can get started right away (though you'll have a few steps to go through first if you've upgraded to El Capitan ^[4]). Sorry Linux and Windows users—this post probably isn't the one you're looking for ^[5].

Below are a few nice tricks you can do with DTrace to impress your coworkers, and also to get better insights into what your system is really doing.

From strace to dtrace

I've found I can emulate most strace behaviors I want with DTrace, if I'm willing to put in the time. But there's a fancy built-in DTrace script (wrapped in a shell script) called dtruss that lets you do the most common things without even learning the D programing language (no, not that one either—that's unrelated).

One of my most common strace workflows is to take a running process (like a web server), and trace the open system calls it sends to figure out which files it's opening:

$ sudo strace -e open -p $SERVER_PID
Process 10738 attached
open("/home/vagrant/practice/strace.txt", O_RDONLY) = 5

You can do the same thing with dtruss:

$ sudo dtruss -t open_nocancel -p $SERVER_PID
SYSCALL(args) = return
open_nocancel(".\0", 0x0, 0x1) = 8 0
open_nocancel("/Users/colin/Practice/dtrace/runocc.d\0", 0x0, 0x1B6) = 8 0
open_nocancel("/usr/share/zoneinfo/UTC\0", 0x0, 0x0) = 9 0
open_nocancel(".\0", 0x0, 0x1) = 8 0
open_nocancel("/Users/colin/Practice/dtrace/foo\0", 0x0, 0x1B6) = -1 Err#2

So it's pretty similar to use, just a difference in flags (strace -e vs. dtruss -t) and system call names we're tracing (open vs. open_nocancel).

In the dtruss version, we see the name of the system call, the arguments to that call, and the return value, along with an error code where applicable.

On Linux, you can run man syscalls to get a full list of available system calls to trace. For OS X you can either find the list from the kernel source or just use DTrace for that:

$ sudo dtrace -ln 'syscall:::entry'
 ID PROVIDER MODULE FUNCTION NAME
 141 syscall syscall entry
 143 syscall exit entry
 145 syscall fork entry
 147 syscall read entry
 149 syscall write entry
 151 syscall open entry
 153 syscall close entry
 155 syscall wait4 entry
### etc.

If you need to trace multiple system calls instead of just one, you can drop to using dtrace instead of dtruss, but it gets a little complicated if you want to retain information about arguments & return values like strace gives you. We'll skip all that noise, but it's a one-liner if you can settle for the basics:

$ sudo dtrace -qn 'syscall::write:entry, syscall::sendto:entry /pid == $target/ { printf("(%d) %s %s", pid, probefunc, copyinstr(arg1)); }' -p $SERVER_PID
(41941) sendto HTTP/1.0 200 OK
(41941) sendto Server: SimpleHTTP/0.6 Python/2.7.10
(41941) sendto Date: Thu, 05 Nov 2015 04:44:00 GMT
(41941) sendto Content-type: text/html; charset=utf-8
(41941) sendto Content-Length: 350
(41941) sendto

I'll leave the full D language tutorial to others, but the gist of it is that you specify:

• one or more probes (required), which are essentially the kinds of events you want to trace (the write and sendto system calls, in our case)
• a predicate (optional), which lets you filter events in the kernel and avoid passing them back up to user space (matching PID, in our case)
• an action (required), which lets you write the code that'll be executed when the events that you're interested in fire

What you see above is using the syscall DTrace provider (the first part of the probe: syscall::write:entry)—but you can do way more than just system calls with DTrace. And that's why people say it's more powerful (for some hazy definition of "better") than strace—it's able to do a lot more things. Also, since DTrace executes in kernel context, it doesn't have to keep switching between user mode & kernel mode the way strace does, so there's less context-switching overhead. This means you can even use DTrace on production systems (though I'm sure there are caveats, like only enabling the probes you need).

You don't have unlimited power in D (for example, no loops!), but there's a lot you can do very concisely, and keep in mind you're writing code to be executed at the kernel level—this is pretty amazing stuff!

Built-in OS X DTrace scripts

You can learn a ton about using DTrace by going through examples on Brendan Gregg's page, but you also already have a bunch of his examples on your computer (they shipped with your OS). Check them out:

$ ls /usr/bin/*.d
/usr/bin/bitesize.d /usr/bin/kill.d /usr/bin/seeksize.d
/usr/bin/cpu_profiler.d /usr/bin/loads.d /usr/bin/setuids.d
/usr/bin/cpuwalk.d /usr/bin/newproc.d /usr/bin/sigdist.d
/usr/bin/creatbyproc.d /usr/bin/pathopens.d /usr/bin/syscallbypid.d
/usr/bin/dispqlen.d /usr/bin/pidpersec.d /usr/bin/syscallbyproc.d
/usr/bin/filebyproc.d /usr/bin/priclass.d /usr/bin/syscallbysysc.d
/usr/bin/hotspot.d /usr/bin/pridist.d /usr/bin/timer_analyser.d
/usr/bin/httpdstat.d /usr/bin/runocc.d /usr/bin/weblatency.d
/usr/bin/iofile.d /usr/bin/rwbypid.d
/usr/bin/iofileb.d /usr/bin/rwbytype.d

Try digging into a few of these and seeing how they work. You'll probably be surprised at how compact they are—many are essentially one-liners.

For example, filebyproc.d traces for files being opened, by process:

$ sudo filebyproc.d
dtrace: script '/usr/bin/filebyproc.d' matched 7 probes
CPU ID FUNCTION:NAME
 2 937 open_nocancel:entry kextd //private/var/run/installd.commit.pid
 1 937 open_nocancel:entry Google Chrome /var/folders/5s/bfpr9qhx1nb2lk1spyp78gm40000gn/T/.com.google.Chrome.K6nrPC
 0 151 open:entry mds_stores .
 0 151 open:entry mds_stores .
 0 151 open:entry CrashPlanServic /Users/colin/javadoc/1.4.2/docs/api/java/awt/color
 2 151 open:entry mds /Users/colin/javadoc/1.4.2/docs/api/java/awt/color
 0 937 open_nocancel:entry CrashPlanServic /Users/colin/javadoc/1.4.2/docs/api/java/awt/color
 0 151 open:entry CrashPlanServic /Users/colin/javadoc/1.4.2/docs/api/java/awt/color/class-use

You can see in the output above that I found CrashPlan (a backup service) backing up some JavaDocs from way back in version 1.4.2—why did I even have those?! Deleted.

This is a pretty cool feature, and it's literally only one line—we could do the exact same thing via dtrace directly, and then customize it however we want to get further details:

$ sudo dtrace -n 'syscall::open*:entry { printf("%s %s", execname, copyinstr(arg0)); }'

Writing your own

It's pretty straightforward to write your own scripts from scratch, using other D scripts for inspiration. To start, try just tweaking existing scripts to slightly change the output or calculations. It's been instructive for me to do things like pick a feature I'd like to have (like Linux vmstat's display of the current run queue), then pull ideas from a few scripts together to get even more insight than I was looking for.

Incidentally, I found on my machine, and those of a few coworkers, that the built-in D scripts related to the CPU run queues, in particular, mistakenly thought the run queues were always stuck at 0 regardless of actual load. But after a couple hours of spelunking through some OS X kernel headers, I was able to make 1-line changes to these files and get working versions of both runocc.d and dispqlen.d. I don't know much about the OS X kernel and how processors are grouped in sets, so there may be problems with my version, but what I see now reflects the system load a lot more than a bunch of blank lines or zeroes!

To me, this all speaks pretty highly about the ease of writing D scripts, and the ability to iterate quickly on new ideas (no long compile/install cycles)—I've only been looking into DTrace for a week at this point, and hadn't dug into kernel programming before either.

Going further

These are the main resources I'm looking at to continue learning more about DTrace. I'd be excited to have others join me on the ride—this stuff is super-fun!

• Brendan Gregg's site
• The DTrace Guide
• The DTrace Book

References

[1] Julia Evans has a terrific set of blog posts on strace, and even an strace zine you can print out and carry around with you.

[2] Brendan Gregg also wrote up a nice intro to strace.

[3] Chad Fowler did an strace intro too.

[4] OS X 10.11, El Capitan, has some new security restrictions whose default configurations break DTrace (though some scripts may will still work). This article explains System Integrity Protection (SIP) and shows you how to get DTrace back in working order, among other things. Caveat: I haven't upgraded to El Capitan, but I'm assuming this approach works, based on seeing the same advice in other places as well.

[5] If you're on Linux, there is a DTrace for Linux project, but it seems like there's more documentation and community around tools like perf_events (an official, built-into-the-kernel Linux tool) and SystemTap. You should take a look at Brendan Gregg's Linux performance page to see what tools make the most sense for you. Windows users—sorry, I'm sure there are great tools there too, but I have no idea!