GHCR: Cannot pull docker image without auth in Kubernetes

Code to Cloud GitHub Packages
#1

I have created a public docker image ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0.

Using docker without being logged in to ghcr.io I can pull that image using:

docker pull ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0

When I use that image without image pull secrets in Kubernetes I get the error:

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  46s                default-scheduler  Successfully assigned default/mypod to manjaro
  Normal   BackOff    21s (x2 over 44s)  kubelet            Back-off pulling image "ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0"
  Warning  Failed     21s (x2 over 44s)  kubelet            Error: ImagePullBackOff
  Normal   Pulling    9s (x3 over 46s)   kubelet            Pulling image "ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0"
  Warning  Failed     9s (x3 over 44s)   kubelet            Failed to pull image "ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0": rpc error: code = Unknown desc = failed to pull and unpack image "ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0": failed to resolve reference "ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0": failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden
  Warning  Failed     9s (x3 over 44s)   kubelet            Error: ErrImagePull

Steps to reproduce:

curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644
k3s kubectl run -it mypod --image=ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0 --restart=Never -- sh
#2

Hi @clarkbw,

I just saw your comment in Download from Github Package Registry without authentication so is my report a bug?

#3

:wave: @windsource I was able to pull your image into my cluster ok.

Locally I have kubernetes running with minikube.
I was getting a similar error, but found that doing this fixed my issue.

Perhaps you have a similar issue? The “anonymous token” error message also isn’t one coming from Container Registry (I validated :smile:) I think it’s coming from k8s

Events:
Type Reason Age From Message

Normal Scheduled 2m55s default-scheduler Successfully assigned default/windsource-6bb75db69f-f7tz9 to minikube
Normal Pulling 2m55s kubelet, minikube Pulling image “ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0”
Normal Pulled 2m50s kubelet, minikube Successfully pulled image “ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0” in 4.629080496s
Normal Created 77s (x5 over 2m50s) kubelet, minikube Created container nextcloud-influxdb-tracks-importer
Normal Started 77s (x5 over 2m50s) kubelet, minikube Started container nextcloud-influxdb-tracks-importer
Normal Pulled 77s (x4 over 2m50s) kubelet, minikube Container image “ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0” already present on machine

#4

Hi @whitneyimura, I also think that the problem might not be in GHCR but either in k3s or containerd. Switching to docker driver as you have described it is no solution for me as I want to get away from docker driver on Kubernetes.
I have created an issue in k3s.

#5

:wave: Hi @windsource from the GH Container Registry team! We just pushed a change this morning that should fix the issue you were seeing. It was a bug on our end with how we were handling certain requests from containerd. I’ll post an update on the k3s issue you created as well

$ sudo ctr image pull ghcr.io/windsource/nextcloud-influxdb-tracks-importer:latest
ghcr.io/windsource/nextcloud-influxdb-tracks-importer:latest:                     resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:a9a6214e87593bac0f0345f04025799c14120855bfded59c35a3a7bf727b77b0: done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:3c96a94aa10c342b7ac8dc8f854d14dbabe57cabc5999dbf2796d5eeb942edc4:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:e8780064e542093a966d0a52204f730019c6b087029b53b1a111268815d296cc:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:58c8db755c6dc5035b49eaa57b4696e6074db57d623fd2568d3d02751b092781:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:6dfe9eecf7268812efdbc6f9b16239dd7d4c1b6a74aaa74fc4a55c2bdc43d21e:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 0.4 s                                                                    total:   0.0 B (0.0 B/s)                                         
unpacking linux/amd64 sha256:a9a6214e87593bac0f0345f04025799c14120855bfded59c35a3a7bf727b77b0...
done
1 Like
#6

Hi @markphelps, i installed k3s a month ago. Can you suggest me a link about an article or blog post on how to upgrade k3s ( ubuntu 20.04 on digital ocean ) to make this change available on my droplet? Thanks in advance

#7

Hi @batok, the change was not in k3s but in the Github Container Registry (GHCR) itself. The change was performed by Github.

1 Like