If there is one thing I’ve learned about Acceptance Testing (AT) it’s that there are a lot of ways to mess it up. I’ve made my fair share of mistakes and thought it’d be nice to know I’m not alone.
I did an informal and unscientific survey on some acceptance testing mistakes to find out how many of us do it wrong. Here are the results:
-
Does your team document the system in word docs?
A great benefit of writing acceptance tests is that they become a living documentation for your software. They live because they are executable. They have bits that turn green when they work and red when they fail.
Any static document that is written in the process of developing a piece of software is destined to become a lie. These documents get lost and abandoned and provide very little value. The truth is in the code. Acceptance tests that execute against the code must stay up to date.
-
Are acceptance tests handed off to you?
Acceptance tests are a communication medium. They are not a communication mode. They aren’t a message written over on one side of a wall and passed to the other.
They are a common medium that enables the conversation between the People Who Know What The Software Should Do (PWKWTSSD) and the People That Know How To Make The Software (PTKHTMTS).
Out of this interaction comes the development of a common language that everyone can use to communicate about the system. That language will never develop unless the authorship of the tests is shared between the two groups of people.
-
Do you begin each new test with copy-paste?
Duplication is just as abhorrent and problematic in tests as it is in code. It is the easiest way to begin a new test, but will lead to massive maintenance burdens down the road.
Duplication is often the result of lots and lots of workflow type tests (do this, click that, check that, do this other thing…). These test are a great place to start, but after you have a few of them, it’s time to extract out higher level scenarios that get rid of the duplication.
In FitNesse, you can use scenario tables to transform a script table into a decision table. In Cucumber, you can use a scenario outline to transform a scenario into an example table. Both techniques result in cleaner tables, little duplication, and more expressive tests.
-
Are your tests kept in source control?
The acceptance tests should live right alongside the code. When the source code is branched or tagged, then so should the tests be. This means that everyone needs access to the repository and everyone needs to know how to use it. PWKWTSSD and PTKHTMTS alike.
-
Are your tests generic or specific?
This is how most software specification begins. We don’t want to limit or constrain what our new app will do, so we make the specifications as general as possible. The problem with these documents is that they are entirely works of fiction. They can never be built.
Software is best specified by example. Tell me about one person and how they will use the software to accomplish one thing. Give me a specific example.
-
Do you write acceptance tests alone?
The number one purpose of Acceptance Tests is to be a communication medium between PWKWTSSD and PTKHTMTS. It’s perfectly valid for a PTKHTMTS to sit alone and write some tests.
There is just no point in using an acceptance testing tool like FitNesse or Cucumber to write the tests. These tools are excellent as a bridge between the natural language world and the world of code, but if you are already writing code, then just write the test in a programming language.
There a much fewer hoops to jump through and a lot more powerful tools at your disposal.
-
Are you are using a “QA Test Tool?”
Making acceptance tests tool-centric is a big mistake. FitNesse and Cucumber are no silver bullet tools that will suddenly fix all the quality problems in your project. It’s not about the tool, it’s about the communication. The tool is just, well, a tool.
If your goal is simply automation, then FitNesse and Cucumber both make poor scripting languages. Use a scripting language if you want to automate a repetitive task.
-
Are your tests running?
If your tests aren’t running, why did you bother to write them? They ought to run on PWKWTSSD and PTKHTMTS’s machines and on a common integration server. Running acceptance tests overnight is not running them. That feedback is too long in coming.
By the time you come in the next morning, you’ve lost your context and it’s expensive to go back and figure it out. This either slows development down, or test failures start to be ignored.
-
If you test the entire system, you might be doing AT wrong.
It’s a pretty common desire. We want to know that the entire system works all together, so we write the tests exercising the entire stack.
The problem here again is the time that it takes to bring the entire system up and back down for every test. Even a couple of seconds will exponentially turn into minutes and hours as a test suite grows.
Integration tests are still an important thing, but Integration tests should test the integration of systems, not the integrated system. What we need are thin tests on the boundaries of systems.
These tests are run (and subsequently break) for reasons very different than acceptance tests. For example, an integration test may fail when some third party server goes down, or a server has been mis–configured.
-
Do you test through the GUI?
It’s so seductive. More and more tools continue to pop up allowing you to automate the interface of your application. Tests through the GUI all share two problems: they are fragile and they are slow.
They are fragile because they change for lots of reason unrelated to the code and because a small change to one element on the screen can mean dozens of changes to tests that reference that element.
You can mitigate this cost though by avoiding duplication. But even then, the tests are destined to be too slow and put you in the situation where you stop running the tests because they take so long.
And here is the thing about the graphical user interface. It’s graphical. The only way to really test it is to look at it. GUIs must look right. There is no way to automate that.
It’s okay to have a few tests that hit the GUI, but you need to be able to get to the core business rules (read models) without exercising the GUI.