Random unknown blob error when pushing to Docker Github Package repository

Code to Cloud GitHub Actions
#1

I want to preface this issue by saying we as an organization have over 5 Billion pulls from Dockerhub. I am not down right refusing to try things but I am near 100% confident this is a specific infrastructure issue with Github. 

If anyone else is having this issue please bump this thread, even if it only happened once and a subsequent retry worked.

We recently implemted pushes to Gitlab, Quay.io, and Github along with Dockerhub. 

When we push our images we do so in a loop with near identical logic for all of the endpoints. 

Github has a realitively high failure rate since implementation a few days ago ( 5 out of our last 50 builds ), here are some example failures: 

https://ci.linuxserver.io/job/Docker-Pipeline-Builders/job/docker-tvheadend/job/master/141/display/redirect

https://ci.linuxserver.io/job/Docker-Pipeline-Builders/job/docker-guacd/job/master/56/display/redirect

https://ci.linuxserver.io/job/Docker-Pipeline-Builders/job/docker-bazarr/job/development/500/display/redirect

The problem is random and the error is always the same unknown blob. 

This exact random problem plagued Dockerhub for a brief time during an upgrade: 

https://github.com/docker/hub-feedback/issues/1693

Officially they never wrote a post mortem, but unoficially it was related to load balancers being a little to aggressive and the eventual consistent model not working as frontend machines were not aware of the object that was just uploaded. 

Please let me know if you need anymore information. 

21 Likes
#2

We have also been experiencing this issue with the GitHub package repository for Docker, and arrived at the exact same conclusion. The more steps a Dockerfile contains, the more layers it has, and the more likely it is to run into a problematic outcome of this race condition.

Today, a 15-layer image is failing about about 25% of the time.

1 Like
#3

Same here, with the same ratio of about 25%.

1 Like
#4

I’m seeing it more like 75% of the time…just failed on two new workflow runs in a row…

1 Like
#5

Quick touch on this cropping up a lot in the last few days. Tonight I’ve had several workflow run failures due to the dreaded unknown blob

3 Likes
#6

So many unknown blob on docker push I’m not sure I can release docker images anymore.

Another one getting common for me is getaddrinfo EAI_AGAIN registry.npmjs.org error on build stage, so maybe it has something to network?

1 Like
#7

And again, we have unknown blob for like 80-85% of docker pushes from GitHub Actions

3 Likes
#8

And again, been getting unknown blob on all builds this morning…

1 Like
#9

Happens a lot as well. Is there anything that can be done to mitigate this?

1 Like
#10

Problem began to appears 2 weeks ago (very few builds would fails), a bit more since (3 of ten maybe), but today, almost every build ends up failing

The issue has raised too big and fast today, problems comes from GH ?

Could be noted, but manual push seems less prone to fail

1 Like
#11

We have been seeing the same issue for the last few days.

Is github addressing this community wide issue?

2 Likes
#12

I have begun seeing this issue in the last few days as well. However, the failure is happing on 80% of the builds.

1 Like
#13

We are having the same issue since yesterday. Completely random, happens 30-50% of the time. This is very frustrating.

1 Like
#14

Yep! Same here. It’s happened like 10 builds in a row.

1 Like
#15

It can’t get through a single push right now.

This has basically incapacitated development on a project I am working on, as we were relying on Github Packages as part of our CI.

My work around right now is to go to self-hosted runner that is doing the push and to spam docker push until it gets all of the layers.

This does work if you are in under a deadline and can’t switch out package repos right now. If you’re running your actions on a GA VM you may be out of luck.

I had been putting off setting up a local package repo but now it is unavoidable. Absolutely must have stability in package storage in the CI.

FWIW, Docker Hub itself apparently experienced issues with unknown blob in October.

They have yet to offer a postmortem.

2 Likes
#16

docker push to github docker package registries has become 100% unusable over the last hour.

2 Likes
#17

Do we have any word on this? I’ve got customers that are relying on a push I’ve made this morning that’s yet to see production.

#18

It would be nice to get some acknowledgment of the issue. It looks like githubstatus.com does not test push operations against the service.

It looks like there was a scheduled maintenance four days ago, but otherwise no incident report despite this ongoing issue.

#19

Still having the problem today.

This is crippling all of our deployment processes - still haven’t found any tracker or issues replied by GH staff yet.

#20

I’m having the same issue here :frowning:
Did anyone find a workaround?