GitHub Actions Failing from Last Night: MariaDB via Docker has "Gone Away"

Hello all!

We’re absolutely stumped trying to figure out why our project’s GitHub Actions builds have stopped working as of a scheduled action run that we have set up to run at 2AM UTC every Sunday.

Nothing at all code-wise changed between our last successful build and the first failing build, but every build subsequent to that one throughout the day today has failed for the same reasons.

To summarize our setup, we run functional tests by briefly spinning up a set of dependencies via Docker Compose. One of these dependencies is MariaDB, which has suddenly stopped accepting connections as of this morning.

On the connecting side, the following error occurs:

SQLSTATE[HY000] [2006] MySQL server has gone away

On the database, the following error appears with each connection attempt:

[Warning] Aborted connection 154 to db: 'unconnected' user: 'unauthenticated' host: '172.18.0.5' (This connection closed normally without authentication)

We’ve checked every other portion of our setup and can confirm that there are NO changes that were made to:

  • Our own Docker images or their upstream dependencies for either the web application or the MariaDB container
  • Any of our PHP dependencies or the PHP version we’re using in this case

However, the exact version of the GitHub Actions virtual environment (ubuntu-20.04 version 20210504.1) is also unchanged between the succeeding and the failing runs.

Basically, our entire workflow just stopped working out of the blue and we have absolutely no idea what changed to trigger it, so we’re hoping perhaps we’re not the only ones having issues right now and this is somehow a problem with GitHub Actions having pushed some minor changes up this weekend that broke things.

If anyone has any feedback they can provide on this, it would be much appreciated.

Update: It turns out this wasn’t caused by GitHub Actions in particular, but the actual story is kind of nuts, so I figured I’d post it here for posterity’s sake.

  • On May 7, MariaDB released a new minor version of 10.5, 10.5.10. The Docker images hadn’t been updated to reference this new version yet as of May 8.
  • On May 8, our own DB image rebuilt. We build in an “apt-get upgrade” (basically) into our own image building process, which doesn’t cause problems 99% of the time, but if you’re in that little 1% window where apt has a newer version than the upstream Docker image builds, apt-get upgrading it wipes out the Docker-specific configuration.
  • This new DB image, having been rebuilt 2 hours before our main repo’s scheduled CI build at 2am, was used as the source for that CI, which promptly failed because the networking configuration wasn’t appropriate for Docker.
  • Every single CI build afterward, and every user who updated to that new DB container, was met with the same issue.

Simply removing the “apt-get upgrade” fixed it. Yeesh.

1 Like