Docker image upload fails to Github Packages

Hi,

I have a question for those who use Github Actions for uploading Docker images to Github Packages registry (docker.pkg.github.com): Are you able to push images without problems?

I am currently seeing the unknown blob error whenever I am pushing new image (my images have about 800 MB). (This error was discussed here already.) This error was always there, however in the past it reared its ugly head only once per X uploads, now it fails on every one.

2 Likes

Hi @cerna , 

Do you use a public github repo or a private one? And could you please share your workflow yml file and workflow run logs here? 

Could  you please go to organization/personal billing page to see the “Storage for Actions and Packages” section? Does it reach the spend limit? 

billing.png

Hi @yanjingzhu,

thank you for the reply. It is an Open-Source, project. This is the problematic workflow. (It also is failing in my work fork: for example here.)

I think it is connected to Github API errors. Because after the last incident reported on GithubStatus it started to work OK. But then it deteriorated again.

I also tried to limit concurrent upload of Docker deamon to 1, but that doesn’t seem to have any effect.

The Latest testing shows that it is failing pretty much on every run: https://github.com/machinekit/machinekit-hal/runs/633116119?check_suite_focus=true#step:4:99.

However, yesterday in my work fork it was uploading pretty much OK(ish): https://github.com/cerna/machinekit-hal/actions/runs/90643378.

So now, 19.47 GMT+0, the push was successful on all images. To me, it looks awfully close to a load problem on the API.

Also, before I requested rerun of the workflow, there was hanging one from this afternoon:

I can guarantee that the job did not run for 72 hours. Even the original push is not 72 hours old.

Sorry for the bad experience. 

The unknown blob issue is a temporary issue . I am glad to hear that all your images has been pushed successfully.  

For "This run was marked as timed out after more than 72 hours. " issue, would you mind opening a new ticket for it?   

Hi @yanjingzhu,

thanks for the information that effort is exerted to solve this issue. It started to get worse again (not on every push, but on one or two last ones).

In the end I solved it by brute force:

set +e
printf "$GITHUB_TOKEN" | docker login docker.pkg.github.com -u ${GITHUB_OWNER} --password-stdin
TRY=0
while [$TRY -lt ${MAX_TRIES}]
do
  (docker push ${IMAGE_NAME_BASE}${TAG}:latest)
  RETVAL=$?
  if [$RETVAL -eq 0]; then
    break
  fi
  printf "Docker push exited with %d\nError occured during try %d, trying again\n" "$RETVAL" "$TRY"
  TRY=$(( $TRY + 1 ))
done
exit ${RETVAL}

On second or third try in every runner the push is successful.

I will send the new ticket by Github feedback.

Also seeing this issue intermittently. Running from CircleCI workflow which hasnt been changed and has been working fine for months.