Pushing docker image to GPR: manifest invalid: Only schema version 2 is supported

Sorry, I just got mine building with another version and had the same issue again… Are people using the github private repo ?

I also don’t know if it’s worth noting that the docker manual says :

> When pushing images, clients which support the new manifest format should first construct a manifest in

> the new format. If uploading this manifest fails, presumably because the registry only supports the old

> format, the client may fall back to uploading a manifest in the old format.

This makes me think that it may be failing to upload using v2 then reverting to v1 where it then gets this error back from the server.

1 Like

Yes I’m trying to push from a private repo to it’s private repo. This fallback to v1 could be it, there is a noticeable delay after the last layer is pushed before the error shows up.

Nevermind that, for both success and failure the delay is 6 seconds. Success gives you the latest digest SHA And failure that error.

FYI, I sent a message detailing the issue github developer support, who have escalated the issue to the product team with the link to this community post.

I’ll update if I hear back from them.

Update, I figured out my issue:

So nested in the logs ( journalctl ) is the actual issue:

Nov 20 09:42:08 blue-pont dockerd[427]: time="2019-11-20T09:42:08.376878027+11:00" level=warning msg="failed to upload schema2 manifest: unauthorized: This operation would exceed the storage allotment for this account. - falling back to schema1"
Nov 20 09:42:08 blue-pont dockerd[427]: time="2019-11-20T09:42:08.376989057+11:00" level=warning msg="[DEPRECATION NOTICE] registry v2 schema1 support will be removed in an upcoming release. Please contact admins of the docker.pkg.github.com registry NOW to avoid future disruption."
Nov 20 09:42:10 blue-pont dockerd[427]: time="2019-11-20T09:42:10.260794961+11:00" level=error msg="Not continuing with push after error: manifest invalid: Only schema version 2 is supported"

Seems our account is over the storage allocation ( I’m not the certain what this limit is ) and therefore we are no longer able to upload our images.

It is attempting to upload using schema 2 then failing and falling back, docker does not know how to handle the error from the github private repo and is therefor falling back in a reall confusing manner.

So i am not certain if we can delete older packages to free up space or what the limit is.

I used journalctl | grep dockerd (since I aways forget the flags) to get the full logs on Arch Linux, this will be different on OSX (https://docs.docker.com/docker-for-mac/troubleshoot/) or windows (https://docs.docker.com/v17.12/docker-for-windows/troubleshoot/).

But Docker will log the reason it falled back to the version 1 schema in its logs.

2 Likes

So nested in the logs ( journalctl ) is the actual issue:

Nov 20 09:42:08 blue-pont dockerd[427]: time="2019-11-20T09:42:08.376878027+11:00" level=warning msg="failed to upload schema2 manifest: unauthorized: This operation would exceed the storage allotment for this account. - falling back to schema1"
Nov 20 09:42:08 blue-pont dockerd[427]: time="2019-11-20T09:42:08.376989057+11:00" level=warning msg="[DEPRECATION NOTICE] registry v2 schema1 support will be removed in an upcoming release. Please contact admins of the docker.pkg.github.com registry NOW to avoid future disruption."
Nov 20 09:42:10 blue-pont dockerd[427]: time="2019-11-20T09:42:10.260794961+11:00" level=error msg="Not continuing with push after error: manifest invalid: Only schema version 2 is supported"

Seems our account is over the storage allocation ( I’m not the certain what this limit is ) and therefore we are no longer able to upload our images.

It is attempting to upload using schema 2 then failing and falling back, docker does not know how to handle the error from the github private repo and is therefor falling back in a reall confusing manner.

So i am not certain if we can delete older packages to free up space or what the limit is.

I used journalctl | grep dockerd (since I aways forget the flags) to get the full logs on Arch Linux, this will be different on OSX (https://docs.docker.com/docker-for-mac/troubleshoot/) or windows (https://docs.docker.com/v17.12/docker-for-windows/troubleshoot/).

But Docker will log the reason it falled back to the version 1 schema in its logs.

We are seeing the same error and we have fixed tag length. Started after a few days, it was working before. Hopefully some staff member can write here to give more information.

A related question: if the space is limited, can we automatically remove old images via some command/API?

Yes we can, I already have a work in progress action to remove package versions: https://github.com/WyriHaximus/github-action-delete-package

Which is based on this bit of documentation: https://help.github.com/en/github/managing-packages-with-github-packages/deleting-a-package#deleting-a-version-of-a-private-package

The thing is I’m trying to get the list of this packages with this GraphQL query but it’s giving me an unexpected not found error:

query {
  repository(owner:"OWNER", name:"REPOSITORY") {
    registryPackages(last:20) {
      edges {
        node {
          id,
          name
        }
      }
    }
  }
}

Glueing these two together and get them to work should make you able to delete old docker image tags.

1 Like

UPDATE: When we increased our total quota for the registry the action started working.

Also looking into that delete API, but we have a different approach because we don’t want to delete last n, we want to delete all images related to a tag too. Pretty simple to call the API once you get your head around it.

I find compounding actions with uses increases the total run time (have to download the image and run it), so unless the task is complex, it’s better to add a script/step to do it IMO. Also the least dependencies the better.

1 Like

I might be stating things that people already know, But talking to github support they advised that the default storage for Github team is: 

GitHub Team

  • Storage: 2GB
  • Data Transfer: 10GB

Also that the organisation storage can be checked by viewing: https://github.com/organizations/ORG/settings/billing

The following works for me, to list versions of packages on a project.

curl -X POST \
    -H "Accept: application/vnd.github.package-deletes-preview+json" \
    -H "Authorization: bearer TOKEN" \
    -d '{"query":"query {repository(owner:\"ORG\", name:\"REPOSITORY\") {registryPackages(last:20) {edges{node{id, name, versions(last:20){edges {node {id}}}}}}}}"}' \
    https://api.github.com/graphql
7 Likes

One thing you have to be careful with, we just discovered, is not deleting all the versions for a package; Github at that point will prevent you from publishing any version under that name and will return this error:

blob upload invalid: Docker image with name "<image name>" is already associated with another repository under owner "<account>"

Still waiting for an answer from support to understand how to fix it.

5 Likes

@thepont Yeah that fixes it. Would be nice to get a more descriptive error message in the long run tbh. Thank you for fixing the GraphQL query as well, they will get me to something working this weekend 🎉.

@aledalgrande That’s good to know! Wanted to do a latest tag for master builds and SHA or tag title for tagged releases, but sounds like the latest tag for master builds isn’t an option then :(. The reason I’m also doing an Action is to both give back and make it easier to compose actions between my projects. I’m intending to build an auto clean up of old image tags that aren’t relevant anymore but are still taking up space. (Also where I can I’m building Actions directly on alpine with a quick package install keeping that build time as low as possible.)

@thepont @aledalgrande Thank you both for all the help on this. I’ve accepted @thepont’s post because it’s a nice concise summary, but want to let you know you both been very helpful 👍.

2 Likes

Any update from support on this error?

Did you ever get an answer? I am having the exact same problem now.

I’m experiencing the same issue now, were you able to find a solution?

How did any of you delete a docker package?
when i try to do so, it sez. “Version deletion is currently unsupported for docker”

3 Likes

Adding to this thread.

Apparently

  • The limits are hard-coded. They do not  vary with your spending limits,  or support level.  e.g.Even if you’re UnlimitedEnterprise Support,  or have a fixed (and very high) spending limit, you’re capped.
  • You can, apparently,  not delete Docker images.  Which makes this whole thing remarkably goofy and useless as a general purpose registry replacement
2 Likes

Currently stuck on the same step.

Did you manage to delete it?

I think they must have changed their API… I am getting the same error, It’s pretty unusable. I switched to using ECR, but was hoping to clean up all the storage on the account, but I can’t even do that.

They have changed the API, about 10 days ago to be precise.

An automated cleanup script that would remove uploaded Docker images > 2 days old has stopped functioning with “Version deletion is currently unsupported for docker.”

We’re slowly running out of available space for hosting our Docker images and this change is going to require a switch back to ECR for hosting the images - the last thing I want to be working on at the moment.

1 Like

Could it be related to https://github.community/t5/GitHub-Actions/Overwrite-Docker-image-in-GitHub-Package-Registry/m-p/42266/highlight/true#M4928 otherwise the docker image registery is useless for my use case. Will try to confirm the above and report back here later today.