Worker running out of disk space #26357
-
Hello Github Actions, We are using Github Action to build our TensorFlow docker images. We’ve seen that some actions are failing on “out of disk space” error.
While debugging this we’ve discovred that the 20GB space provided by the build worker is mounted under /mnt while the docker daemon is configued to use /var/lib/docker on the root fs which has ~7GB of free space. To workaround this issue we’ve added a docker configuartion step to have it use the /mnt folder instead of the the /var/lib disk location. Pleae find here for the benefit of the community:
I would be happy to have the worker by deafult set to use the /mnt path, or perhpas even better have the deafult “/” mount disk extended to include +20GB of storage data which would eliviate the need to configure this custom path (as it’s likely that other tools will use /var path as well reaching the same issue as described above. |
Beta Was this translation helpful? Give feedback.
Replies: 5 comments
-
Hi @maximveksler , Many thanks for your sharing! It will be quite helpful for other guys who encounter same diskspace problem. Regarding the default worker path, You can raise a feedbac.k ticket here where github product manager will take a review. Happy coding! |
Beta Was this translation helpful? Give feedback.
-
Yes thank you, that was done combined with posting the thread. |
Beta Was this translation helpful? Give feedback.
-
Hi, I ran into the same kind of issue ie. “No Space Left On Device” using github actions for building several python based docker images (with spark and lot of libs…). We need at least 60Gb of free space, may be more to ensure we can build all the docker images in a row. So above customisation isn’t enough for us. So my question is simple, is there a way to get more space for our builds ? I know I could find ways to optimize our builds by purging temporary contents, pruning useless docker layers etc. but isn’t the goal of external CI/CD mechanism like actions to concentrate on our code not on the CI/CD ? If so we shouldn’t have to care about resources limits or allocation. |
Beta Was this translation helpful? Give feedback.
-
Reply to myself in case others have the same issue : use external runners !
|
Beta Was this translation helpful? Give feedback.
-
I would not consider this a solution, as that means you are again maintaining infrastructure. In which case, you might as well consider using Prow. |
Beta Was this translation helpful? Give feedback.
Hi @maximveksler ,
Many thanks for your sharing! It will be quite helpful for other guys who encounter same diskspace problem.
Regarding the default worker path, You can raise a feedbac.k ticket here where github product manager will take a review. Happy coding!