Seeing this too https://github.com/renovatebot/renovate/pull/5996/checks?check_run_id=597383462

ref: https://github.community/t5/GitHub-Actions/Diminishing-disk-space-on-Ubuntu-images/m-p/54332#M9137

We’ve also experienced problems since yesterday that seem related to disk space on Actions.

Here’s an example where individually installing all the dependencies, that would be installed in the stage that is failing, succeeds (test (ubuntu-latest, 3.7)) but when they are all installed normally the error occurs (notebooks (3.7)). We can see (with df -h) from the stage that succeeds that there is still plenty of space available, so this is all very strange.

Before installs:

Filesystem Size Used Avail Use% Mounted on
udev 3.4G 0 3.4G 0% /dev
tmpfs 695M 960K 694M 1% /run
/dev/sda1 84G 75G 8.3G 91% /
tmpfs 3.4G 8.0K 3.4G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.4G 0 3.4G 0% /sys/fs/cgroup
/dev/loop0 40M 40M 0 100% /snap/hub/43
/dev/loop1 94M 94M 0 100% /snap/core/8935
/dev/sda15 105M 3.6M 101M 4% /boot/efi
/dev/sdb1 14G 41M 13G 1% /mnt

After individual installs:

Filesystem Size Used Avail Use% Mounted on
udev 3.4G 0 3.4G 0% /dev
tmpfs 695M 960K 694M 1% /run
/dev/sda1 84G 79G 5.0G 95% /
tmpfs 3.4G 8.0K 3.4G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.4G 0 3.4G 0% /sys/fs/cgroup
/dev/loop0 40M 40M 0 100% /snap/hub/43
/dev/loop1 94M 94M 0 100% /snap/core/8935
/dev/sda15 105M 3.6M 101M 4% /boot/efi
/dev/sdb1 14G 41M 13G 1% /mnt

My earlier message got deleted by someone, so reposting this:

We can see an instance where installing all the pacakges individually works (test (ubunut-latest, 3.7)), but then installing all the packages normally fails (notebooks (3.7)). We can see (with df -h) in the case that suceeds that there is plenty of space remaining, which makes this all very strange.

Before installs:

Filesystem Size Used Avail Use% Mounted on
udev 3.4G 0 3.4G 0% /dev
tmpfs 695M 960K 694M 1% /run
/dev/sda1 84G 75G 8.3G 91% /
tmpfs 3.4G 8.0K 3.4G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.4G 0 3.4G 0% /sys/fs/cgroup
/dev/loop0 40M 40M 0 100% /snap/hub/43
/dev/loop1 94M 94M 0 100% /snap/core/8935
/dev/sda15 105M 3.6M 101M 4% /boot/efi
/dev/sdb1 14G 41M 13G 1% /mnt

After individually installing packages:

Filesystem Size Used Avail Use% Mounted on
udev 3.4G 0 3.4G 0% /dev
tmpfs 695M 960K 694M 1% /run
/dev/sda1 84G 79G 5.0G 95% /
tmpfs 3.4G 8.0K 3.4G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.4G 0 3.4G 0% /sys/fs/cgroup
/dev/loop0 40M 40M 0 100% /snap/hub/43
/dev/loop1 94M 94M 0 100% /snap/core/8935
/dev/sda15 105M 3.6M 101M 4% /boot/efi
/dev/sdb1 14G 41M 13G 1% /mnt

@rarkinscan you also link to the specific run in GitHub Actions that you’re seeing this error? I’ve submitted a help ticket to GitHub Actions and in the event that it gets reviewed by the engineering team it would be nice to have as many examples of this as possible to review.

I noticed something even more strange: this error is only occuring on Linux runners

runs-on: ubuntu-latestjens-maus

while the macOS builds succeed. I’ve applied @jens-maus’s hack that fixes things (detailed here). I’ve tried this with both removal of /swapfile like they do and found df -h gives

Filesystem Size Used Avail Use% Mounted on
udev 3.4G 0 3.4G 0% /dev
tmpfs 695M 960K 694M 1% /run
/dev/sda1 84G 64G 20G 77% /
tmpfs 3.4G 8.0K 3.4G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.4G 0 3.4G 0% /sys/fs/cgroup
/dev/loop1 40M 40M 0 100% /snap/hub/43
/dev/loop0 94M 94M 0 100% /snap/core/8935
/dev/sda15 105M 3.6M 101M 4% /boot/efi
/dev/sdb1 14G 41M 13G 1% /mnt

and with just running apt-get clean

Filesystem Size Used Avail Use% Mounted on
udev 3.4G 0 3.4G 0% /dev
tmpfs 695M 960K 694M 1% /run
/dev/sda1 84G 72G 12G 87% /
tmpfs 3.4G 8.0K 3.4G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.4G 0 3.4G 0% /sys/fs/cgroup
/dev/loop0 40M 40M 0 100% /snap/hub/43
/dev/loop1 94M 94M 0 100% /snap/core/8935
/dev/sda15 105M 3.6M 101M 4% /boot/efi
/dev/sdb1 14G 41M 13G 1% /mnt

For our project it seems that just running with apt-get clean is enough space restored that our Linux runner jobs can proceed again.

However, I have to agree with @saudet on the point they raised on “Diminishing disk space on Ubuntu images” that this really seems to be a problem. This is targeting only the Linux runners and this started happening silently after months of no issues. Having users go in and manually clean up runners to ensure that there is sufficient space doesn’t really seem like a good solution.

Hi @matthewfeickert ,

Thanks for your sharing! I forked your repo, the disk issue cannot be reproduced on my side now, looks engineering team has fixed it, could you please help to confirm? 

Thanks for your reply, @weide-zhou. The problem is not fixed though.

You’re not seeing the issue as the fork of pyhf you made includes the hack fix that I mentioned in my previous reply (c.f. pyhf PR 819). If you remove the sudo apt-get clean lines the exact same error occurs (example of doing exactly that here). For pyhf this can be easily replicated by committing and pushing the results of the following

sed -i '/sudo apt-get clean/d' .github/workflows/ci.yml

For anyone running into issues with no space left, there is an issue open in actions/virtual-environments that has some discussions going on: actions/runner-images#709

Yes, the VM rollback resolved the issue. I’ve marked this last post as the “Solution” as you’ve linked to the relevant actions/virtual-environments Issue in it. Thanks!