Mutable -latest tag

It looks like when pushing a container image to ghcr that the -latest tag isn’t mutable.

I am getting 401 errors when attempting to push branch-latest when the latest tag already exists.

Dropping the -latest tag and the issue goes away. Are all ghcr.io tags immutable so there is no way to use a -latest tag?

Is there a default -latest tag and I shouldn’t need to do this with ghcr.io?

Thanks!

The tags are mutable when using ghcr.io and the latest tag is a tag you can re-use. I think you probably have an issue independent of this, maybe a token validity issue that doesn’t have the correct permissions – I guess it’s possible that you need delete permissions to re-tag, because you’re deleting the old (but that’s just a guess!).

Here’s an example of it working

dev:~$ echo "FROM alpine" > Dockerfile
dev:~$ docker build -t ghcr.io/shrink/tag-mutability:latest .
Sending build context to Docker daemon  4.096kB
Step 1/1 : FROM alpine
 ---> a24bb4013296
Successfully built a24bb4013296
Successfully tagged ghcr.io/shrink/tag-mutability:latest
dev:~$ docker push ghcr.io/shrink/tag-mutability:latest
The push refers to repository [ghcr.io/shrink/tag-mutability]
50644c29ef5a: Layer already exists
latest: digest: sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65 size: 528
dev:~$ echo "RUN echo date" >> Dockerfile
dev:~$ docker build -t ghcr.io/shrink/tag-mutability:latest .
Sending build context to Docker daemon  4.096kB
Step 1/2 : FROM alpine
 ---> a24bb4013296
Step 2/2 : RUN echo date
 ---> Using cache
 ---> 650391d6851f
Successfully built 650391d6851f
Successfully tagged ghcr.io/shrink/tag-mutability:latest
dev:~$ % docker push ghcr.io/shrink/tag-mutability:latest
The push refers to repository [ghcr.io/shrink/tag-mutability]
50644c29ef5a: Layer already exists
latest: digest: sha256:dc8fa8c07374cf05b6a40963177f8a8a828a753679959794407578b6f0ec227c size: 528

Via the GitHub website I can see during this process:

  1. A new image ID is created for the latest tag (ID on github.com is 53479)
  2. A new image ID created for the new latest tag (ID on github.com is 53486)
  3. I return to the page for the first tag (53479) and I see the tag latest is gone and now it’s just showing the sha256... value

Can you share the command you’re using?

Not sure what’s going on but delete scope isn’t necessary for re-writing a new tag. Only deleting an entire image.

1 Like

Yep. Realized it may be more related to docker buildx than ghcr.io.

My action is here:

Command is:
docker buildx build --platform linux/arm/v6,linux/arm/v7,linux/arm64,linux/amd64 --file Dockerfile --tag ghcr.io/chriskinsman/tilloo:${GITHUB_REF##*/}-3.1.$GITHUB_RUN_NUMBER --output "type=image,push=true" .

Error is:

#56 ERROR: failed commit on ref "manifest-sha256:5ce43fed2875c44d7eddf396ac74719cbe43f61999b20d09af86dc8566ef5160": unexpected status: 401 Unauthorized
------ > exporting to image:
------ failed to solve: rpc error: code = Unknown desc = failed commit on ref “manifest-sha256:5ce43fed2875c44d7eddf396ac74719cbe43f61999b20d09af86dc8566ef5160”: unexpected status: 401 Unauthorized
##[error]Process completed with exit code 1.

Opened a bug with the buildx team also. This works great if I drop the -latest tag out of buildx.

The problem is then adding the tag to a multi-arch image.

If I do a pull, tag, push I end up with a single arch image.

@chriskinsman,

I’m experiencing a similar unexpected status: 401 Unauthorized error here:

It seems to only happen when pushing a new container package. If the container package already exists, the workflow completes fine:

Could you try creating the container package before pushing using buildx?

You can do something like this:

docker pull alpine
docker tag alpine ${{ env.IMAGE }}:pre-push
docker push ${{ env.IMAGE }}:pre-push

The pre-push tag I’m using here can be anything. ${{ env.IMAGE }} is the location of your container package.

Please let me know if that makes a difference!

In my case the container does exist.

I suspect my issue is related to: https://github.com/docker/build-push-action/issues/126

That doesn’t work as that will pull down the original image for a single architecture that you are pulling it on and then push back a single-arch image breaking the multi-arch image in the repo.

The fix for https://github.com/docker/build-push-action/issues/126 fixed my issue also.

Hi @chriskinsman,

I’m struggling to find what changed. My workflows are now publishing successfully. Do you know what and where the fix is?

Thanks,
Jamie.

Not sure. For me I had to specify a newer moby/buildkit image. I suspect they added a patch to buildkit that fixed the issue that was unreleased. If it is now working for everyone they may have promoted that to latest so it is automatically picked up.