Starting services with inter-dependencies and prepare steps

Here’s my desired workflow job:

  • Build app from Dockerfile
  • Start PostgreSQL.
  • Create a PostgreSQL database and user
  • Start multiple instances of the app
  • Run integration tests against the app

This workflow is challenging because GitHub Action steps are sequential, but the only supported way of starting unordered background tasks is with “container services”, which start before steps and have no way of expressing dependency relationships. (For example, job.services.postgresql.ports[5432] does not work inside a service declaration!)

So the actual execution order becomes:

  1. Start PostgreSQL
  2. Start multiple instances of the app
  3. Build app from Dockerfile
  4. Create a PostgreSQL database and user
  5. Run integration tests against the app

Unfortunately, that means the app attempts to start before everything is ready, and fails. The app starts before it’s built, and before PostgreSQL starts, and before there is even a database.

Now, I can of course build the app in a separate step and then push it to my Docker registry. That’s unfortunate and deeply inefficient (why upload to some remote location only to fetch it again?), but I can accept this, since I expect jobs to be parallelizable and may not run on the same machine. If performance really matters, I can set up a local Docker registry; dumb, but it will work.

However, I am unable to find a way to only start my app after everything is ready. The app can, of course, retry database connections until they’re successful, but expecting my app to retry until there is a database and user is beyond reasonable.

I could put some wait stuff into the image startup, but I really don’t want to. The image doesn’t have the right tools to do so — it would mean bundling the Postgres client tools (psql, createuser, createdb) inside the Docker image. CI stuff doesn’t belong in that Dockerfile. I could build a special Dockerfile that “inherits” from the first one and adds CI tools, but that seems entirely wrong to me. These are CI steps — they belong in the workflow file!

Ideally, I’d like to see services as integrating with the steps:

jobs:
  test:
    steps:
    - image: postgres:12
      service: true  # Causes it to detach and not block next steps
      hostname: postgres  # Expose service as this hostname
      healthcheck: tcp:localhost:5432  # Wait for healthcheck
    - run: ./setup_database_and_user.sh
    - run: docker build --tag myservice
    - image: myservice  # Image I just built
      args: --dbhost postgres
      service: true
      hostname: myservice1
      healthcheck: http://localhost:80
    - image: myservice
      args: --dbhost postgres
      service: true
      hostname: myservice1
      healthcheck: http://localhost:80
    - run: make integration_tests

This would solve the inter-dependency and ordering problems. (This is pretty much exactly how CircleCI does things.)

What is the “official” solution to this?

1 Like

Why not just start PostgrSQL as a background service, and then the app containers with docker run after the build? The one catch is that you need to put the app containers in the same Docker network as the service, you can use something like --network=${{ job.container.network }} for that (see job context).

Thanks. That sounds feasible, though needlessly complicated. Not just the network stuff, but I’d have to store the container IDs as output data so that I could ensure that they are stopped at the end. Then I’d have to add my own health check to wait for these containers to start up and fail if they don’t. I’ll give it a shot.