GHCR: Cannot pull docker image without auth in Kubernetes #26427

windsource · 2020-10-05T16:54:26Z

windsource
Oct 5, 2020

I have created a public docker image ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0.

Using docker without being logged in to ghcr.io I can pull that image using:

docker pull ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0

When I use that image without image pull secrets in Kubernetes I get the error:

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  46s                default-scheduler  Successfully assigned default/mypod to manjaro
  Normal   BackOff    21s (x2 over 44s)  kubelet            Back-off pulling image "ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0"
  Warning  Failed     21s (x2 over 44s)  kubelet            Error: ImagePullBackOff
  Normal   Pulling    9s (x3 over 46s)   kubelet            Pulling image "ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0"
  Warning  Failed     9s (x3 over 44s)   kubelet            Failed to pull image "ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0": rpc error: code = Unknown desc = failed to pull and unpack image "ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0": failed to resolve reference "ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0": failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden
  Warning  Failed     9s (x3 over 44s)   kubelet            Error: ErrImagePull

Steps to reproduce:

curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644
k3s kubectl run -it mypod --image=ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0 --restart=Never -- sh

Answered by markphelps

Oct 28, 2020

👋 Hi @windsource from the GH Container Registry team! We just pushed a change this morning that should fix the issue you were seeing. It was a bug on our end with how we were handling certain requests from containerd. I’ll post an update on the k3s issue you created as well

$ sudo ctr image pull ghcr.io/windsource/nextcloud-influxdb-tracks-importer:latest
ghcr.io/windsource/nextcloud-influxdb-tracks-importer:latest:                     resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:a9a6214e87593bac0f0345f04025799c14120855bfded59c35a3a7bf727b77b0: done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:3c96a94aa10c342b7ac8dc8f854d14dbabe57cabc5999dbf…

View full answer

windsource · 2020-10-05T19:13:00Z

windsource
Oct 5, 2020
Author

Hi @clarkbw,

I just saw your comment in Download from Github Package Registry without authentication so is my report a bug?

0 replies

whitneyimura · 2020-10-16T19:33:43Z

whitneyimura
Oct 16, 2020

👋 @windsource I was able to pull your image into my cluster ok.

Locally I have kubernetes running with minikube.
I was getting a similar error, but found that doing this fixed my issue.

Perhaps you have a similar issue? The “anonymous token” error message also isn’t one coming from Container Registry (I validated 😄) I think it’s coming from k8s

Events:
Type Reason Age From Message

Normal Scheduled 2m55s default-scheduler Successfully assigned default/windsource-6bb75db69f-f7tz9 to minikube
Normal Pulling 2m55s kubelet, minikube Pulling image “ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0”
Normal Pulled 2m50s kubelet, minikube Successfully pulled image “ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0” in 4.629080496s
Normal Created 77s (x5 over 2m50s) kubelet, minikube Created container nextcloud-influxdb-tracks-importer
Normal Started 77s (x5 over 2m50s) kubelet, minikube Started container nextcloud-influxdb-tracks-importer
Normal Pulled 77s (x4 over 2m50s) kubelet, minikube Container image “ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0” already present on machine

0 replies

windsource · 2020-10-17T15:45:41Z

windsource
Oct 17, 2020
Author

Hi @whitneyimura, I also think that the problem might not be in GHCR but either in k3s or containerd. Switching to docker driver as you have described it is no solution for me as I want to get away from docker driver on Kubernetes.
I have created an issue in k3s.

0 replies

markphelps · 2020-10-28T13:49:17Z

markphelps
Oct 28, 2020

👋 Hi @windsource from the GH Container Registry team! We just pushed a change this morning that should fix the issue you were seeing. It was a bug on our end with how we were handling certain requests from containerd. I’ll post an update on the k3s issue you created as well

$ sudo ctr image pull ghcr.io/windsource/nextcloud-influxdb-tracks-importer:latest
ghcr.io/windsource/nextcloud-influxdb-tracks-importer:latest:                     resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:a9a6214e87593bac0f0345f04025799c14120855bfded59c35a3a7bf727b77b0: done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:3c96a94aa10c342b7ac8dc8f854d14dbabe57cabc5999dbf2796d5eeb942edc4:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:e8780064e542093a966d0a52204f730019c6b087029b53b1a111268815d296cc:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:58c8db755c6dc5035b49eaa57b4696e6074db57d623fd2568d3d02751b092781:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:6dfe9eecf7268812efdbc6f9b16239dd7d4c1b6a74aaa74fc4a55c2bdc43d21e:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 0.4 s                                                                    total:   0.0 B (0.0 B/s)                                         
unpacking linux/amd64 sha256:a9a6214e87593bac0f0345f04025799c14120855bfded59c35a3a7bf727b77b0...
done

0 replies

batok · 2020-10-30T18:13:37Z

batok
Oct 30, 2020

Hi @markphelps, i installed k3s a month ago. Can you suggest me a link about an article or blog post on how to upgrade k3s ( ubuntu 20.04 on digital ocean ) to make this change available on my droplet? Thanks in advance

0 replies

windsource · 2020-10-30T18:27:32Z

windsource
Oct 30, 2020
Author

Hi @batok, the change was not in k3s but in the Github Container Registry (GHCR) itself. The change was performed by Github.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GitHub Community

GHCR: Cannot pull docker image without auth in Kubernetes #26427

{{title}}

Replies: 6 comments

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

GitHub Community

GHCR: Cannot pull docker image without auth in Kubernetes #26427

windsource Oct 5, 2020

Replies: 6 comments

windsource Oct 5, 2020 Author

whitneyimura Oct 16, 2020

windsource Oct 17, 2020 Author

markphelps Oct 28, 2020

batok Oct 30, 2020

windsource Oct 30, 2020 Author

windsource
Oct 5, 2020

windsource
Oct 5, 2020
Author

whitneyimura
Oct 16, 2020

windsource
Oct 17, 2020
Author

markphelps
Oct 28, 2020

batok
Oct 30, 2020

windsource
Oct 30, 2020
Author