Can't pull private image using GITHUB_TOKEN from non-default branch

I followed these instructions: Using GitHub Packages with GitHub Actions - GitHub Docs

I am able to push new images to<org>/<image> from the original repository using username=<org> and password=GITHUB_TOKEN.

I am unable to pull those images from other repositories that were added with read (or write) access in<org>/packages/container/<image>/settings/actions_access using the same username+GITHUB_TOKEN

Pulling with PAT still works.

How can we pull docker images from cross-organisation?

Error message when pulling the image (login is succesfull)

Login Succeeded
Error response from daemon: pull access denied for<org>/<image>, repository does not exist or may require 'docker login': denied: installation not allowed to Read organization package

It appears the GITHUB_TOKEN only has the assigned permissions on the default branch.
I was testing this in a PR that changed from PAT to GITHUB_TOKEN.

Even if the default branch has the exact same workflow, the GITHUB_TOKEN will not grant you the correct permissions for using the docker image in PR workflows.

I am using the docker image as a service in my workflow during CI, this prohibits me from moving away from PAT for pulling the docker image.