A simple example of this is a set of end-to-end tests for a web application. When the test suite begins to run, it is reasonable for it to assume that the web application that it is testing is up and running. Here is a docker-compose.yml file that models this.
version: '2'
services:
e2e_tests:
image: my_e2e_tests
depends_on:
- web
web:
image: my_web_image
When you docker-compose run my_e2e_tests, Docker Compose will start your e2e_tests and your web service. The problem is that "start," in this context, does not mean what we typically want it to mean.
So what does "start" actually mean?
What Docker Compose guarantees is that dependency services are started in the same sense that your desktop computer is started when you press the power button–it still takes a while for the computer to get through its startup screens before you actually get a login screen and the computer is "ready for use."
In our case here, Docker Compose only guarantees that the web service had its "power button pressed," but not that the web server is actually ready to start accepting connections; it still has to go through its regular startup procedure and do a few things before it's actually "ready for use."
Problem?
Yes. We have a race condition. Because both services start at the same time, it is possible that the e2e_tests attempt to initiate a connection to the web service before the web service is ready to accept connections. Kablamo.
A Solution
One workaround is to modify our e2e_tests startup script so that it has knowledge of this fact and does something about it. It's pretty trivial to add some "preload" block of code to e2e_tests startup code so that it tries to connect to the web service, then retries, and sleeps, and tries again and again until it succeeds.
Although this will work, it's not ideal. Reason being is that it gives your tests knowledge of the fact that they live in some sort of containerized world where the services it depends on was literally just started and needs some time to warm up. You're adding more and more "stuff" to the container that it otherwise wouldn't have. When running something such as end-to-end tests, it would be nice to just assume that thing they're testing is running and accepting connections.
A better solution
An alternative is to lean on the tools that you're already using to containerize your apps: Docker Compose. Although you don't get anything out-of-the-box that allows you to say "this service depends on this open port of that service," the tools for introducing such behavior are right in front of you: simply introduce a "port-checking" service.
To demonstrate this, let us first modify our existing docker-compose.yml file a bit so that we introduce the race condition: [1].
version: '2'
services:
e2e_tests:
image: ubuntu:14.04
depends_on:
- web
command: 'nc -vz web 8080'
web:
image: ubuntu:14.04
command: >
/bin/bash -c "
sleep 5;
nc -lk 0.0.0.0 8080;
"
What we have here is a service called web that simulates a delayed startup time by sleeping for 5 seconds (sleep 5). After the sleep, it is ready to accept incoming TCP connections on port 8080 (nc -lk 0.0.0.0 8080). This mimics the delay that a typical service may incur when starting up cold.
Next, we have e2e_tests which, upon startup, immediately try to connect to web (nc -z web 8080). If you try running e2e_tests then you will see a message about web container being created, but the e2e tests will fail to run because they will not be able to make a connection to web.
Let's run this and confirm our expectations:
$ docker-compose run e2e_tests
Creating network "your_project_default" with the default driver
Creating your_project_web
nc: connect to web port 8080 (tcp) failed: Connection refused
Now that we have reliably simulated the race condition, we can add in our "port-checking" service. This service will wait for the relevant ports to open up before continuing, at which point we can be certain that our services that are expected to be running on those ports have been fully started and are ready to accept connections.
Go ahead and add the following snippet to the bottom of your docker-compose.yml [2] (make sure you include the depends_on configuration option).
start_dependencies:
image: ubuntu:14.04
depends_on:
- web
command: >
/bin/bash -c "
while ! nc -z web 8080;
do
echo sleeping;
sleep 1;
done;
echo Connected!
"
What happens here is pretty simple. This service attempts to make a TCP connection to port 8080 of the web container and loops until it is successful, sleeping 1 second on each loop. Once successful, the loop terminates, the message "Connected!" is printed to the terminal, and the service terminates.
Docker Compose will ensure that the dependent service (web) remains running even after this container terminates. What this means is that, immediately after running the start_dependencies service, e2e_tests can be started with certainty that web is ready to accept connections right away.
So how do you actually run these now?
Just like this [3]:
$ docker-compose run start_dependencies
Creating network "your_project_default" with the default driver
Creating your_project_web
sleeping
sleeping
sleeping
sleeping
Connected!
$ docker-compose run e2e_tests
Connection to web 8080 port [tcp/http-alt] succeeded!
Voilà! Not only does this address our race condition, but it's also very intuitive and easy to follow. First we start dependencies and then run the e2e_tests.
I don't want to add all that stuff into my compose file.
That's perfectly understandable. I won't put ketchup in my burger, and I certainly won't put multi-line shell code in my yaml.
To accommodate your preferences, I've put all of this into a small image that you can use directly, without all that bash. You can find the image on Docker Hub or the source on GitHub. All you need to do is reference that image instead and modify the command configuration options to list the services/ports you would like to wait for in [host]:[port] format, like so:
start_dependencies:
image: dadarek/wait-for-dependencies
depends_on:
- web
command: web:8080
You can list multiple dependencies by separating them with a space:
command: web:8080 database:5432
By default, the container will sleep 2 seconds in between each connection attempt. You can control that by passing in a SLEEP_LENGTH environment variable:
environment:
- SLEEP_LENGTH: 0.5
What's next?
That's all I have for now. I hope this helps you Docker cleanly!
[1] My apologies for using the 100+ MB Ubuntu image instead of the ~5 MB Alpine image. I could not get this example to work correctly with the version of netcat that comes with Alpine.
[2] I put everything into a single docker-compose.yml file for simplicity. It may make more sense for you to separate things into multiple files and merge them together.
