Can't pull private ghcr.io image using GITHUB_TOKEN from non-default branch

I followed these instructions: Using GitHub Packages with GitHub Actions - GitHub Docs

I am able to push new images to ghcr.io/<org>/<image> from the original repository using username=<org> and password=GITHUB_TOKEN.

I am unable to pull those images from other repositories that were added with read (or write) access in https://github.com/orgs/<org>/packages/container/<image>/settings/actions_access using the same username+GITHUB_TOKEN

Pulling with PAT still works.

How can we pull docker images from ghcr.io cross-organisation?

Error message when pulling the image (login is succesfull)

Login Succeeded
Error response from daemon: pull access denied for ghcr.io/<org>/<image>, repository does not exist or may require 'docker login': denied: installation not allowed to Read organization package

It appears the GITHUB_TOKEN only has the assigned permissions on the default branch.
I was testing this in a PR that changed from PAT to GITHUB_TOKEN.

Even if the default branch has the exact same workflow, the GITHUB_TOKEN will not grant you the correct permissions for using the docker image in PR workflows.

I am using the docker image as a service in my workflow during CI, this prohibits me from moving away from PAT for pulling the docker image.