GitHub Package Registry tag docker-base-layer is missing a manifest

Hi,

It looks like the first time you use the GitHub Package Registry it creates a “docker-base-layer” tag which is missing a manifest. Unfortunately one of the tools I’m using, fluxcd, queries the registry for new tags but throws the error:

manifest for tag docker-base-layer missing in repository

I also get error doing a docker pull:

manifest for docker.pkg.github.com/...:docker-base-layer not found: manifest unknown: Docker image reference docker-base-layer not found under repo

I’m not sure if this is going to cause problems down the road.

Thanks,

Eli

Thank for reporting this issue.

I tried _docker pull docker.pkg.github.com/org/dockertest/docker-image:docker-base-layer. _I got the same error as yours. 

I am trying to contact package registry experts for your issue. When they give me any response, I will let you know at the first time.   

I got response from Github Package Registry team, they said that docker-base-layer  is kind of a meta tag that we use. it’s not meant to be pulled directly. 

They are working on hide it in the page UI and make docker image with tag docker-base-layer  inaccessible.  

As one of the maintainers of FluxCD - the tool this issue was reported for - my advice would be to not just hide it on the page UI and make it inaccessible, but also prevent the registry API from listing the tag.

We heavily rely on the availability of manifests for all tags returned by the registry to be able to perform safe automatic image upgrades, and corrupt or missing manifests for a tag that is listed will result in our safe guard getting triggered because we are unable to get a full picture of the available images and their timestamps.

In addition to this being a specific issue for Flux, listing something that can not be accessed by the requester or is meant for internal purposes only does not make much sense to me.

4 Likes

@hiddeco Can you share the registry API to get tags of an image stored in Github Package Registry? I can not find a right one. 

@yanjingzhuthe API endpoint is as defined in the Docker Registry and OCI Distrubition specs.

For the dummy repository I used for testing, the following returns all tags available for this image.

$ DOCKER_AUTH=$(jq -r ".[\"auths\"][\"docker.pkg.github.com\"][\"auth\"]" $HOME/.docker/config.json)
$ curl -H "Authorization: Basic ${DOCKER_AUTH}" https://docker.pkg.github.com/v2/hiddeco/podinfo/podinfo/tags/list
{"name":"podinfo","tags":["something","docker-base-layer"]

As you can see this response includes the ‘docker-base-layer’, attempting to retrieve information for this tag results however in a ‘MANIFEST_UNKNOWN’ error code.

$ curl -H "Authorization: Basic ${DOCKER_AUTH}" https://docker.pkg.github.com/v2/hiddeco/podinfo/podinfo/manifests/docker-base-layer
{"errors":[{"code":"MANIFEST_UNKNOWN","message":"Docker image reference podinfo:docker-base-layer not found under repo \"hiddeco/podinfo\""}]}

While the ‘something’ tag yields a result (as expected):

$ curl -H "Authorization: Basic ${DOCKER_AUTH}" https://docker.pkg.github.com/v2/hiddeco/podinfo/podinfo/manifests/something
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 3028,
      "digest": "sha256:ec2f218e3268a10cb18cf7f83035d261d84a960baacdda5acbfd51ac7bb121c1"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 2786963,
         "digest": "sha256:4167d3e149762ea326c26fc2fd4e36fdeb7d4e639408ad30f37b8f25ac285a98"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 1033228,
         "digest": "sha256:8cfb1360a8bd40f34244e6359ed67a23c48824ea27b0f4a674157693974a7639"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 8348713,
         "digest": "sha256:fd2936a4f0d8c83ab424e2f482e982cdffb461db3a45e73c7791d07ca1bf64d5"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 5411084,
         "digest": "sha256:a2655b8ac6e0a642f823a7fadd3e5aa23cdf2f56916fecdae6edbdaf296f194f"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 2073,
         "digest": "sha256:1127b599dc1dffb926fbf9e5048e69bb0515305908ac82601aaa57341681236d"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 8350871,
         "digest": "sha256:3cef76885c1ed36568c1cf5b93cebdb727b83dbcda24a26a0a9d8e9143f5dd45"
      }
   ]
}

@yanjingzhu 

My original comment was flagged as spam, so here is a second attempt. The endpoint is as defined in the Docker Registry and OCI distribution specifications:

GET /v2/\<name\>/tags/list  
Host: docker.pkg.github.com  
Authorization: Basic \<token\>  

Using the command line, while being logged in to the registry using docker login (assumes presence of curl and jq ):

$ DOCKER_AUTH=$(jq -r ".[\"auths\"][\"docker.pkg.github.com\"][\"auth\"]" $HOME/.docker/config.json)
$ curl -H "Authorization: Basic ${DOCKER_AUTH}" https://docker.pkg.github.com/v2/hiddeco/podinfo/podinfo/tags/list
{"name":"podinfo","tags":["something","docker-base-layer"]}
$ curl -H "Authorization: Basic ${DOCKER_AUTH}" https://docker.pkg.github.com/v2/hiddeco/podinfo/podinfo/manifests/docker-base-layer
{"errors":[{"code":"MANIFEST_UNKNOWN","message":"Docker image reference podinfo:docker-base-layer not found under repo \"hiddeco/podinfo\""}]}
$ curl -H "Authorization: Basic ${DOCKER_AUTH}" https://docker.pkg.github.com/v2/hiddeco/podinfo/podinfo/manifests/something
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 3028,
      "digest": "sha256:ec2f218e3268a10cb18cf7f83035d261d84a960baacdda5acbfd51ac7bb121c1"
   },
...

Thank you for the detail method to use registry API. Yes, you are right, the “docker-base-layer” is still showing on the tags list.  I have directed your advice to the Github Package Registry team for further evaluation. When they give me any feedback, I will update here soon. 

4 Likes

Any update yet? This issue also breaks our custom app which retrieves tags via the Github V4 API.

@mtttcgcg Sorry for any inconvenience. Now the docker-base-layer issue is in lower priority, I have added your scenario in our internal ticket for this issue. It will help to improve the priority.