Random unknown blob error when pushing to Docker Github Package repository

I want to preface this issue by saying we as an organization have over 5 Billion pulls from Dockerhub. I am not down right refusing to try things but I am near 100% confident this is a specific infrastructure issue with Github. 

If anyone else is having this issue please bump this thread, even if it only happened once and a subsequent retry worked.

We recently implemted pushes to Gitlab, Quay.io, and Github along with Dockerhub. 

When we push our images we do so in a loop with near identical logic for all of the endpoints. 

Github has a realitively high failure rate since implementation a few days ago ( 5 out of our last 50 builds ), here are some example failures: 

https://ci.linuxserver.io/job/Docker-Pipeline-Builders/job/docker-tvheadend/job/master/141/display/redirect

https://ci.linuxserver.io/job/Docker-Pipeline-Builders/job/docker-guacd/job/master/56/display/redirect

https://ci.linuxserver.io/job/Docker-Pipeline-Builders/job/docker-bazarr/job/development/500/display/redirect

The problem is random and the error is always the same unknown blob

This exact random problem plagued Dockerhub for a brief time during an upgrade: 

https://github.com/docker/hub-feedback/issues/1693

Officially they never wrote a post mortem, but unoficially it was related to load balancers being a little to aggressive and the eventual consistent model not working as frontend machines were not aware of the object that was just uploaded. 

Please let me know if you need anymore information. 

12 Likes

We have also been experiencing this issue with the GitHub package repository for Docker, and arrived at the exact same conclusion. The more steps a Dockerfile contains, the more layers it has, and the more likely it is to run into a problematic outcome of this race condition.

Today, a 15-layer image is failing about about 25% of the time.

Same here, with the same ratio of about 25%.

I’m seeing it more like 75% of the time…just failed on two new workflow runs in a row…