Mysql service never comes up healthy in action

At some point in the last couple weeks, the mysql:5.7 service config I was using to expose mysql to my action stopped working. Here’s the config I’m using:

name: Run Tests
on: pull_request
jobs:
  test:
    runs-on: ubuntu-latest
    services:
      mysql:
        image: mysql:5.7
        ports:
        - 3306
        options: --health-cmd "mysqladmin ping" --health-interval 10s --health-timeout 5s --health-retries 10

    steps:
      - uses: actions/checkout@v1.0.0
        with:
          fetch-depth: 1
      - uses: actions/setup-elixir@v1.0.0
        with:
          otp-version: 22.x
          elixir-version: 1.9.x
      - run: MIX_ENV=test mix deps.get
      - name: setup db and test
        run: |
          MIX_ENV=test mix ecto.setup
          mix test
        env:
          DATABASE_URL: "mysql2://root:root@127.0.0.1:${{ job.services.mysql.ports['3306'] }}/myapp"

That config was working perfectly fine for a week or two and then, seemingly out of nowhere, it started failing consistently with the service reporting (almost immediately) that mysql is unhealthy:

Status: Downloaded newer image for mysql:5.7
docker.io/library/mysql:5.7
/usr/bin/docker create --name 926a41768ca34ea8b1364385026a6af1_mysql57_8b5743 --label 2e4e3a --network github_network_169d64eaca1a440991d3382af497b76f --network-alias mysql -p 3306 --health-cmd "mysqladmin ping" --health-interval 10s --health-timeout 5s --health-retries 10 -e GITHUB_ACTIONS=true mysql:5.7
03afaad28cc629cb34a9507a2d19435c2c32ddc1f7fd4f94da2033c01f26c938
/usr/bin/docker start 03afaad28cc629cb34a9507a2d19435c2c32ddc1f7fd4f94da2033c01f26c938
03afaad28cc629cb34a9507a2d19435c2c32ddc1f7fd4f94da2033c01f26c938
/usr/bin/docker ps --all --filter id=03afaad28cc629cb34a9507a2d19435c2c32ddc1f7fd4f94da2033c01f26c938 --filter status=running --no-trunc --format "{{.ID}} {{.Status}}"
03afaad28cc629cb34a9507a2d19435c2c32ddc1f7fd4f94da2033c01f26c938 Up Less than a second (health: starting)
/usr/bin/docker port 03afaad28cc629cb34a9507a2d19435c2c32ddc1f7fd4f94da2033c01f26c938
/usr/bin/docker inspect --format="{{if .Config.Healthcheck}}{{print .State.Health.Status}}{{end}}" 03afaad28cc629cb34a9507a2d19435c2c32ddc1f7fd4f94da2033c01f26c938
unhealthy
##[error]Failed to initialize, mysql service is unhealthy.

As I mentioned, that final error (Failed to initialize, mysql service is unhealthy.) appears almost immediately, and it seems that my health cmd and interval options aren’t being respected.

Anyone know what I might be missing to get this to work?

2 Likes

Hi peburrows,

Glad to hear you in Github Community Forum!

I copied your code and reproduced the issue. After i add ‘jobs.<job_id>.services.env’ to the service, the error is gone, code as below, hope it helps, thanks.

services:
      mysql:
        image: mysql:5.7
        env:
          MYSQL_ROOT_PASSWORD: password
        ports:
        - 3306
        options: --health-cmd "mysqladmin ping" --health-interval 10s --health-timeout 5s --health-retries 10

3 Likes

Awesome, thank you! That seems to have fixed it.

1 Like

Just for future reference for others that might run into this issue. I had an action that setup a postgres and a redis service. This was what the service code looked like in my main.yml:

services:
db:
image: postgres:11
ports: ['5432:5432']
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
redis:
image: redis
ports: ['6379:6379']
options: --entrypoint redis-server

It had been running completely fine for a few weeks and then out of the blue the setup of the redis server start failing with the roughly the same issue.

##[error]Failed to initialize, db service is unhealthy.

I came across this post and thought it wasn’t relevant at first. After much testing I tried setting up my redis service through another Action from the marketplace. At that point my **postgres** service started failing with the same message above!

##[error]Failed to initialize, db service is unhealthy.

Roughly following the advice from this thread I added an .env section with passwords, username, and db env variables to my postgres service. It worked! This is what the code looks like now: 

services:
db:
image: postgres:11
env:
POSTGRES\_USER: postgres
POSTGRES\_PASSWORD: postgres
POSTGRES\_DB: postgres
ports: ['5432:5432']
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
redis:
image: redis
ports: ['6379:6379']
options: --entrypoint redis-server

Hope this is helpful for future devs. Also no clue why this suddenly started happening. Nothing changed in my code that would have affected this.

26 Likes

I just had same issue with postgres 11, running fine for weeks then today:  “Failed to initialize, db service is unhealthy”.  Just started about an hour ago.     This is only thing I found on web that discusses this.   Adding the env part worked first time!  Thank you very much!  

1 Like

Same here, my postgres container stopped working, this fixed it. Thanks!

1 Like

Hey can you show the changes you have done on script? thanks

1 Like

That fixed the container step, but in my rails db:prepare step, I was suddenly getting an issue about a missing postgres password.

Make sure to include the env’s:

DATABASE_URL: postgres://postgres:@localhost:5432/test
          PGUSER: postgres
          PGPASSWORD: postgres

for any other step that needs postgres, such as:

bundle exec rails db:prepare
4 Likes

Thank you! This resolved my issue.

1 Like

It seems that github actions no longer passes the job:env to services, and that you have to replicate the service env vars.

ie, this used to work:

build:
    runs-on: ubuntu-latest
    env:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: postgres

    services:
      postgres:
        image: postgres:11-alpine
        ports:
          - 5432:5432
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

but now you have to repeat the env for the service:

build:
    runs-on: ubuntu-latest
    env:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: postgres

    services:
      postgres:
        image: postgres:11-alpine
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: postgres
        ports:
          - 5432:5432
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

Github: stop making it hard to DRY!!!

2 Likes

Another happy postgres user whose CI action mysteriously stopped working and found this thread. Thanks for getting me running again!

I’d tried all of the suggestions here, but still received the same error.

Maybe it had something to do with me using mysql:5.6 instead of a higher version like I see being used in every snippet.

Anyway, after moving the environment variables to the service, using the MySQL server image did the trick for me:

services:
      mysql:
        image: mysql/mysql-server:5.6
        env:
          MYSQL_USER: testuser
          MYSQL_PASSWORD: testpassword
        options: >-
          --health-cmd "mysql health ping"
          --health-interval 10s
          --health-timeout 10s
          --health-retries 10