Late last year, Reid Draper came to 8th Light University to give a keynote presentation on property-based testing in Clojure. Reid is a software developer at Helium and the author of Clojure's test.check. He was kind enough to respond to some questions about this topic for our blog, and the unedited Q&A is reproduced below. For more from Reid, you can follow him on Twitter.
What is test.check, and how is it different than other automated testing tools?
test.check is a property-based testing tool, inspired by Haskell and Erlang's QuickCheck libraries. Instead of writing individual test-cases, you declare what input is valid for your system, and then assert that certain properties hold true, for all valid randomly generated input. With constant effort, you're able to get as many test-cases as you have CPU cycles. A classic example of this is testing that a list-reversing function returns the original list, when applied twice.
Do you see property-based testing as a complement to example-based tests, or a replacement of them? Why?
I see them as complementary. That being said, I personally tend to reach for property-based testing more often. Example-based tests are great for documentation and preventing specific regressions. In some cases too, writing an example-based test takes less time than writing a property-based test.
What motivated you to write test.check? What has the process of writing test.check looked like? Is it a straight port, or more of a rewrite?
A few years ago I was working on a Clojure project that would have benefitted from property-based testing. No mature tool existed, and having been exposed to both the Haskell and Erlang QuickCheck, I knew what I wanted. test.check is somewhere between a port and a rewrite of Haskell QuickCheck. It follows the Haskell version in many places, but also improves on it, with things like automatic shrinking. Automatic shrinking is a feature inspired by the closed-source Erlang implementation.
What (if any) are the tradeoffs to consider when deciding to use test.check? What kinds of problems is property-based testing best suited for?
Property-based testing takes some experience to get used to, so one tradeoff is trading time to learn and improve your property-based testing skills, in exchange for a future of finding much more interesting bugs. Property-based testing is a general purpose testing strategy, it just may take some time and creativity to figure out how to apply it to your domain. Like with any testing, the more formal the specification for the software, the easier it will be to test.
What are some good resources for people looking to get started using and/or contributing to test.check?
The test.check generator writing guide and my Clojure/West 2014 talk are both good resources. test.check is also actively discussed on the #clojure IRC channel on Freenode.