Worker running out of disk space

GitHub Actions
#1

Hello Github Actions,

We are using Github Action to build our TensorFlow docker images. We’ve seen that some actions are failing on “out of disk space” error.

#12 58.88 Downloading tensorflow-1.15.2-cp37-cp37m-manylinux2010_x86_64.whl (110.5 MB)
**#12 70.86 ERROR: Could not install packages due to an EnvironmentError: [Errno 28] No space left on device**#12 70.86
#12 ERROR: executor failed running [/bin/bash -c python3.7 -m pip install --upgrade pip && python3.7 -m pip install -r requirements-gpu.txt]: runc did not terminate sucessfully
------
 > [6/6] RUN --mount=type=ssh python3.7 -m pip install --upgrade pip && python3.7 -m pip install -r requirements-gpu.txt:
------
failed to solve with frontend dockerfile.v0: failed to solve with frontend gateway.v0: rpc error: code = Unknown desc = failed to build LLB: executor failed running [/bin/bash -c python3.7 -m pip install --upgrade pip && python3.7 -m pip install -r requirements-gpu.txt]: runc did not terminate sucessfully
##[error]Process completed with exit code 1.

While debugging this we’ve discovred that the 20GB space provided by the build worker is mounted under /mnt while the docker daemon is configued to use /var/lib/docker on the root fs which has ~7GB of free space.

To workaround this issue we’ve added a docker configuartion step to have it use the /mnt folder instead of the the /var/lib disk location.

Pleae find here for the benefit of the community:

steps:
    - name: Patch Docker Daemon data-root
      run: |
        DOCKER_DATA_ROOT='/mnt/var/lib/docker'
        DOCKER_DAEMON_JSON='/etc/docker/daemon.json'
        sudo mkdir -p "${DOCKER_DATA_ROOT}"
        jq --arg dataroot "${DOCKER_DATA_ROOT}" '. + {"data-root": $dataroot}' "${DOCKER_DAEMON_JSON}" > "/tmp/docker.json.tmp"
        sudo mv "/tmp/docker.json.tmp" "${DOCKER_DAEMON_JSON}"
        sudo systemctl restart docker

I would be happy to have the worker by deafult set to use the /mnt path, or perhpas even better have the deafult “/” mount disk extended to include +20GB of storage data which would eliviate the need to configure this custom path (as it’s likely that other tools will use /var path as well reaching the same issue as described above.

1 Like
#2

Hi @maximveksler ,

Many thanks for your sharing! It will be quite helpful for other guys who encounter same diskspace problem.

Regarding the default worker path, You can raise a feedbac.k ticket here where github product manager will take a review. Happy coding!

#3

@weide-zhou wrote:

You can raise a feedbac.k ticket here where github product manager will take a review. Happy coding!

Yes thank you, that was done combined with posting the thread.

#4

Hi, I ran into the same kind of issue ie. “No Space Left On Device” using github actions for building several python based docker images (with spark and lot of libs…).

We need at least 60Gb of free space, may be more to ensure we can build all the docker images in a row. So above customisation isn’t enough for us.

So my question is simple, is there a way to get more space for our builds ?

I know I could find ways to optimize our builds by purging temporary contents, pruning useless docker layers etc. but isn’t the goal of external CI/CD mechanism like actions to concentrate on our code not on the CI/CD ? If so we shouldn’t have to care about resources limits or allocation.

#5

Reply to myself in case others have the same issue : use external runners !