Bigger GitHub-hosted runners disk space

Right now, the GitHub-hosted runners only have 14GB of disk space available , making it insufficient to build larger projects without resorting to using a self-hosted runner (which is pretty silly I guess). I think this 14GB is the upper limit so even if you package it in a Docker image it wouldn’t workaround either.

12 Likes

I’m also very interested in this feature.

In particular, I’m looking to host faily large Windows Docker images on Azure and have them pulled into the Windows runners to run with a whole static build environment. With a lot of the Visual Studio SDK and other SDKs installed, they can get really big. Even without layer caching, having a static environment is a big win. 

Since these runners are using standard Azure instances sizes, I guess this would also be a request to allow choosing larger instance sizes for runners. I think this is already being done for the Rust Programming language project since I remember proposing Azure DevOps when they were looking for a suitable CI system from Travis CI and realizing the default runner specs weren’t up to the specifications they requested. Some Microsoftie chimed in and I believe they were given larger instances to run on. This probably carried over to the move to GitHub Actions. 

1 Like

GitHub hosts Linux and Windows runners on Standard_DS2_v2 virtual machines in Microsoft Azure with the GitHub Actions runner application installed. It only provides 14GB storage(SSD).

https://help.github.com/en/actions/automating-your-workflow-with-github-actions/virtual-environments-for-github-hosted-runners#cloud-hosts-for-github-hosted-runners

Even if packages are in Docker image, when you use it , the image will be pulled into the hosted runner. It still occupy disk space.

To build larger projects, we would recommend you to use self-hosted runner.

This was the same issue with Azure Devops, a good solution would be to be able to request the kind of azure machine one needs for his builds for example as a criteria for host selection.

1 Like

I had builds that were passing successfully in GitHub Actions for the longest time but then somewhere between Jan 7 and Jan 27 some change happened that caused all my GitHub Actions to start failing.

The tools I was using are not great at reporting errors but I eventually found out that an unchanged build system was failing due to lack of disk space on the drive with Program Files.

Running the following powershell command at the start of builds:

Get-WmiObject -Class Win32_logicaldisk

Shows the following amount of free disk space:

DeviceID : A:
DriveType : 2
ProviderName :
FreeSpace :
Size :
VolumeName :

DeviceID : C:
DriveType : 3
ProviderName :
FreeSpace : 8229650432
Size : 136912564224
VolumeName : Windows

DeviceID : D:
DriveType : 3
ProviderName :
FreeSpace : 12924661760
Size : 15031267328
VolumeName : Temporary Storage

So you can see the C: drive with Program Files, etc only has ~7.6 GB of free space on it where before it had enough to prevent my build failures (not sure how much, never had an issue before so wasn’t aware of it).

I used the following powershell script to see what is using up space in Program Files:

$path = "$Env:Programfiles"
    $colItems = Get-ChildItem $path | Where-Object {$_.PSIsContainer -eq $true} | Sort-Object
    foreach ($i in $colItems)
    {
        $subFolderItems = Get-ChildItem $i.FullName -recurse -force | Where-Object {$_.PSIsContainer -eq $false} | Measure-Object -property Length -sum | Select-Object Sum
        $i.FullName + " -- " + "{0:N2}" -f ($subFolderItems.sum / 1MB) + " MB"
    }

Resulting in:

C:\Program Files\7-Zip -- 4.96 MB
C:\Program Files\Android -- 145.74 MB
C:\Program Files\Application Verifier -- 0.34 MB
C:\Program Files\Azure Cosmos DB Emulator -- 782.92 MB
C:\Program Files\Boost -- 4,389.07 MB
C:\Program Files\CMake -- 77.07 MB
C:\Program Files\Common Files -- 68.70 MB
C:\Program Files\Docker -- 362.61 MB
C:\Program Files\dotnet -- 15,257.87 MB
C:\Program Files\Git -- 641.46 MB
C:\Program Files\IIS -- 6.45 MB
C:\Program Files\IIS Express -- 26.83 MB
C:\Program Files\internet explorer -- 2.52 MB
C:\Program Files\Java -- 626.49 MB
C:\Program Files\Mercurial -- 37.56 MB
C:\Program Files\Microsoft SDKs -- 144.34 MB
C:\Program Files\Microsoft Service Fabric -- 603.29 MB
C:\Program Files\Microsoft SQL Server -- 265.85 MB
C:\Program Files\Mozilla Firefox -- 187.14 MB
C:\Program Files\MSBuild -- 0.02 MB
C:\Program Files\nodejs -- 49.08 MB
C:\Program Files\OpenSSL -- 9.85 MB
C:\Program Files\PackageManagement -- 0.17 MB
C:\Program Files\PowerShell -- 140.16 MB
C:\Program Files\Reference Assemblies -- 34.16 MB
C:\Program Files\Unity -- 2,930.66 MB
C:\Program Files\VS2010Schemas -- 0.02 MB
C:\Program Files\VS2012Schemas -- 0.02 MB
C:\Program Files\Windows Defender -- 26.25 MB
C:\Program Files\Windows Defender Advanced Threat Protection -- 12.72 MB
C:\Program Files\Windows Mail -- 0.61 MB
C:\Program Files\Windows Media Player -- 4.48 MB
C:\Program Files\Windows Multimedia Platform -- 0.05 MB
C:\Program Files\windows nt -- 7.42 MB
C:\Program Files\Windows Photo Viewer -- 5.84 MB
C:\Program Files\Windows Portable Devices -- 0.05 MB
C:\Program Files\Windows Security -- 0.11 MB
C:\Program Files\WindowsPowerShell -- 106.93 MB

And deleted some stuff in Program Files that I did not need for my projects using a powershell script before my Build so the tooling that gets installed has enough space:

Remove-Item "$Env:Programfiles\Unity" -Recurse -Force -ErrorAction SilentlyContinue -ErrorVariable err
Write-Host $err

Remove-Item "$Env:Programfiles\Boost" -Recurse -Force -ErrorAction SilentlyContinue -ErrorVariable err
Write-Host $err

Note: The above scripts were cobbled together from various StackOverflow posts.

1 Like

Using self-hosted runners comes with a lot of overhead, as you now also need to manage to deploy and orchestrate those. From an end-user perspective, it would be a much less cumbersome and complex solution if he could just specify the bigger instance, e.g. Standard_DS3_v2 instead of Standard_DS2_v2.

I think a lot of us would not mind paying more for this (or reducing the “free” limit twice as fast), as it would still be much cheaper than managing the self-hosted runners.

4 Likes

LLVM builds on Github just broke, due to out of disk space errors as well.

As far as I can see, this is a deal breaker for Github continuous integration for LLVM.

Please don’t respond “use your own runner,” because Github explicitly recommends NOT using your own runner on public-facing Github projects, due to security concerns.

I’d pay for extra disk space during a build, if it could be purchased; but it doesn’t seem that Github has thought this use case through.

Going to have to find a CI alternative to Github until this gets sorted, unfortunately.

2 Likes

We also run into this issue that we cannot build one of our .net solution as is with Github Actions since it exceedes the available disk space.

Not having control over available disk space on runners is a non starter for the majority of projects who would want to use Github Workflows. Upon further investigation, it seems Github does not even set a static limit set, it’s 10 GB “Guaranteed” and you might get more. For example, at this moment my workflow allocates around 20GB before it fails, but maybe in 5 minutes it will fail at 10GB. Imagine trying to debug a pipeline before a release with this type of variable in your builds.

Not certain if the GHW architect thought this one through because there are certain constraints within a build environment that need to remain constant for a pipeline to fulfill its purpose.

1 Like

I’d definitely like the ability to pay for more disk space. I just run into this issue myself that is preventing me from relying on GitHub actions for our installer builds.