You've probably heard about what a unit test is. In fact, you may have even written one or two in your lifetime. Great!
If you've ever been on a project that has a suite of several hundred, thousand, or ten-thousand of these, then you've probably felt the pain of waiting for the tests to finish.
Now, let me start off by saying that I'm not here to talk about what a "real" unit test is, how to write one, how not to write one, or anything like that. I'm just here to introduce one tool that you can use to speed up your unit, integration, acceptance, end-to-end, black-box, white-box, automated, semi-automated, high-level, low-level, whatchamacallit test suite.
This can be applied to any sort of tests that hit the database, or even any situation where you "need" a database for something to work, but you don't really need that database to stick around for too long.
Introducing: tmpfs
You can Google tmpfs for more information, but essentially it is an in-memory file system. Creating it is easy: you create an empty directory and use it as the mount point with the mount command:
$ mkdir my-folder
$ mount -t tmpfs -o <span class="nv">size</span><span class="o">=</span>200m tmpfs my-folder
The above two commands will create the temporary folder, and reserve 200 megabytes of memory in RAM for anything that gets thrown in there. From here on in, anything that you put in my-folder will never actually touch the disk. It will get stored in RAM and the processes that write to it will never know the difference. Whether it's your Java program, database, or shell script, anything that goes in that folder gets stored in RAM.
Now don't forget, when your machine gets rebooted everything in RAM gets wiped, and so will your folder.
So now what?
What does this have to do with our tests? Well, if your tests use a database, then your database probably writes stuff to a physical disk somewhere—and we all know how fast those things are. With your my-folder in place, just configure your database server to write its data files to that folder, and there you have it. Anything that ever gets sent for persistence in your database will always be going to RAM—never touching disk.
For MySql, it's a matter of configuring the datadir variable[4] and restarting the database server. When it comes back up, perform an ls of your my-folder and you'll see a bunch of MySql-specific files located in that folder.
That's it! Configuration aside, this is all you need to get your tests using a database that writes completely to memory. Your mileage may vary, but my team was able to shave around 30%-40% off of every CI build—that's pretty good if you ask me.
Where else can you use this?
We used this to speed up a long-running test suite that was pounding the database on each run, but here are some other potentially useful use cases for tmpfs:
- temporary files of a development/staging server (think logs and cache)
- intermediate files of a long-running, data-crunching process
- inter-process communication[5]
Have fun!
[1] The tmpfs filesystem is only available on some Unix operating systems—sorry Mac and Windows users—but with proliferation of virtualization and container technology, getting your hands on a Unix box is easier than ever.
[2] Don't forget to umount if your machine is long-lived and won't get restarted soon. This mount goes away on a restart. There are ways to always have this mounted when the server starts (just search for "/etc/vfstab" on the web), but that still doesn't mean that files written there will ever get persisted between reboots.
[3] An alternative to tmpfs is ramfs, which is very similar with the caveat that it "never" runs out of space (because it is swapped onto disk when it does). TMPFS will actually give you "out of space" errors once you hit the limit you specify when creating it. Depending on your situation this may or may not be a better option. For something like a test suite, I would recommend tmpfs so when you do hit the limit, something breaks and you know right away.
[4] There are plenty of resources on the web on how to do this.
[5] Some systems automatically mount the /tmp folder and a few other folders as a tmpfs.
