If you have ever used Docker Compose to run multi-container applications, there is a good chance that you have run into the following situation. Service A depends on service B, but service B takes a a while to start up and be ready. Because of this, you must add some extra "wait for service B" logic into service A's startup procedure.
A simple example of this is a set of end-to-end tests for a web application.
When the test suite begins to run, it is reasonable for it to assume that
the web application that it is testing is up and running. Here is a
docker-compose.yml file that models this.
version: '2' services: e2e_tests: image: my_e2e_tests depends_on: - web web: image: my_web_image
docker-compose run my_e2e_tests, Docker Compose will start your
e2e_tests and your
web service. The problem is that "start," in this
context, does not mean what we typically want it to mean.
So what does "start" actually mean?
What Docker Compose guarantees is that dependency services are started in the same sense that your desktop computer is started when you press the power button–it still takes a while for the computer to get through its startup screens before you actually get a login screen and the computer is "ready for use."
In our case here, Docker Compose only guarantees that the web service had its "power button pressed," but not that the web server is actually ready to start accepting connections; it still has to go through its regular startup procedure and do a few things before it's actually "ready for use."
Yes. We have a race condition. Because both services start at the same time, it
is possible that the
e2e_tests attempt to initiate a connection to the web
service before the web service is ready to accept connections. Kablamo.
One workaround is to modify our
e2e_tests startup script so that it has
knowledge of this fact and does something about it. It's pretty trivial to add
some "preload" block of code to
e2e_tests startup code so that it tries to
connect to the web service, then retries, and sleeps, and tries again and again
until it succeeds.
Although this will work, it's not ideal. Reason being is that it gives your tests knowledge of the fact that they live in some sort of containerized world where the services it depends on was literally just started and needs some time to warm up. You're adding more and more "stuff" to the container that it otherwise wouldn't have. When running something such as end-to-end tests, it would be nice to just assume that thing they're testing is running and accepting connections.
A better solution
An alternative is to lean on the tools that you're already using to containerize your apps: Docker Compose. Although you don't get anything out-of-the-box that allows you to say "this service depends on this open port of that service," the tools for introducing such behavior are right in front of you: simply introduce a "port-checking" service.
To demonstrate this, let us first modify our existing
a bit so that we introduce the race condition: .
version: '2' services: e2e_tests: image: ubuntu:14.04 depends_on: - web command: 'nc -vz web 8080' web: image: ubuntu:14.04 command: > /bin/bash -c " sleep 5; nc -lk 0.0.0.0 8080; "
What we have here is a service called
web that simulates a delayed startup
time by sleeping for 5 seconds (
sleep 5). After the sleep, it is ready to
accept incoming TCP connections on port 8080 (
nc -lk 0.0.0.0 8080). This
mimics the delay that a typical service may incur when starting up cold.
Next, we have
e2e_tests which, upon startup, immediately try to connect to
nc -z web 8080). If you try running
e2e_tests then you will see a
web container being created, but the e2e tests will fail
to run because they will not be able to make a connection to
Let's run this and confirm our expectations:
$ docker-compose run e2e_tests Creating network "your_project_default" with the default driver Creating your_project_web nc: connect to web port 8080 (tcp) failed: Connection refused
Now that we have reliably simulated the race condition, we can add in our "port-checking" service. This service will wait for the relevant ports to open up before continuing, at which point we can be certain that our services that are expected to be running on those ports have been fully started and are ready to accept connections.
Go ahead and add the following snippet to the bottom of your
(make sure you include the
depends_on configuration option).
start_dependencies: image: ubuntu:14.04 depends_on: - web command: > /bin/bash -c " while ! nc -z web 8080; do echo sleeping; sleep 1; done; echo Connected!; "
What happens here is pretty simple. This service attempts to make a TCP
connection to port 8080 of the
web container and loops until it is
successful, sleeping 1 second on each loop. Once successful, the loop
terminates, the message "Connected!" is printed to the terminal, and the
Docker Compose will ensure that the dependent service (
web) remains running
even after this container terminates. What this means is that, immediately
after running the
e2e_tests can be started with
web is ready to accept connections right away.
So how do you actually run these now?
Just like this :
$ docker-compose run start_dependencies Creating network "your_project_default" with the default driver Creating your_project_web sleeping sleeping sleeping sleeping Connected! $ docker-compose run e2e_tests Connection to web 8080 port [tcp/http-alt] succeeded!
Voilà! Not only does this address our race condition, but it's also very
intuitive and easy to follow. First we start dependencies and then run the
I don't want to add all that stuff into my compose file.
That's perfectly understandable. I won't put ketchup in my burger, and I certainly won't put multi-line shell code in my yaml.
To accommodate your preferences, I've put all of this into a small image that
you can use directly, without all that bash. You can find the image on Docker
Hub or the source on
GitHub. All you need
to do is reference that image instead and modify the
options to list the services/ports you would like to wait for in [host]:[port]
format, like so:
start_dependencies: image: dadarek/wait-for-dependencies depends_on: - web command: web:8080
You can list multiple dependencies by separating them with a space:
command: web:8080 database:5432
By default, the container will sleep 2 seconds in between each connection attempt. You can control that
by passing in a
SLEEP_LENGTH environment variable:
environment: - SLEEP_LENGTH: 0.5
That's all I have for now. I hope this helps you Docker cleanly!
 My apologies for using the 100+ MB Ubuntu image instead of the ~5 MB Alpine image. I could not get this example to work correctly with the version of netcat that comes with Alpine.