Pip install from git repo inside docker build with github actions

I’m working on segregation of common modules into dedicated repositories for our github organization. Use pip install from git repo in Dockerfile to install shared modules developed inside the organization

RUN pip3 install -r requirements.txt

where git repo dependency referenced like

git+https://github.com/org/repo.git@master

The faced issue is that I can’t make pip3 install to authenticate against organisation private repository when running as github action with pip3 install inside Dockerfile . I want to avoid creating private access token (PAT) for one of the dev as want to be user-agnostic and don’t maintain tokens for leaving team members. Tried to use ${{ secrets.GITHUB_TOKEN }} but with deeper reading realized that the token has access to repository where github action is initiated (link)

The token’s permissions are limited to the repository that contains your workflow

Is there a way to make pip3 install working in github actions without PAT?

Error getting in one of many iterations:

Collecting git+https://****@github.com/org/repo.git@master (from -r requirements.txt (line 17))
  Cloning https://****@github.com/org/repo.git (to revision master) to /tmp/pip-req-build-mnge3zvd
  Running command git clone -q 'https://****@github.com/org/repo.git' /tmp/pip-req-build-mnge3zvd
  fatal: could not read Password for 'https://${GITHUB_TOKEN}@github.com': No such device or address
WARNING: Discarding git+https://****@github.com/org/repo.git@master. Command errored out with exit status 128: git clone -q 'https://****@github.com/org/repo.git' /tmp/pip-req-build-mnge3zvd Check the logs for full command output.
ERROR: Command errored out with exit status 128: git clone -q 'https://****@github.com/org/repo.git' /tmp/pip-req-build-mnge3zvd Check the logs for full command output.