Copying docker images via Skopeo fails

When trying to copy an image from one repo to GHCR it fails with the message:

Writing manifest using preferred type application/vnd.docker.distribution.manifest.v2+json failed: writing manifest "{\n   \"schemaVersion\": 2,\n   \"mediaType\": \"application/vnd.docker.distribution.manifest.v2+json\",\n   \"config\": {\n      \"mediaType\": \"application/vnd.docker.container.image.v1+json\",\n      \"size\": 1471,\n      \"digest\": \"sha256:c059bfaa849c4d8e4aecaeb3a10c2d9b3d85f5165c66ad3a4d937758128c4d18\"\n   },\n   \"layers\": [\n      {\n         \"mediaType\": \"application/vnd.docker.image.rootfs.diff.tar.gzip\",\n         \"size\": 2818413,\n         \"digest\": \"sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3\"\n      }\n   ]\n}": uploading manifest latest to denied: installation not allowed to Write organization package

Not sure where to problem is since the Action’s Repo is explicitly allowed to Write the Package.
Also, it works fine with PAT - this error only happens when using GITHUB_TOKEN.