How to pull the same private repository inside another action?

I try to automate Infrastructure management with terraform in a kind of mono-repo.

In the same private repository, I store terraform modules and several terraform-configuration entry-points pointing to different products/environments. Simplified the project structure is:

src/
    dev/
        main.tf
    prod/
        main.tf
    tf_modules
        module1
        module2

Each environment/product reuse tf_modules and I want to reference them as git sources. Thus, the reference looks similar to "git::git@github.com:private_org/private_repo.git//src/tf_modules/module1?ref=v1.0".

According to terraform documentation:

Terraform installs modules from Git repositories by running git clone, and so it will respect any local Git configuration set on your system, including credentials. To access a non-public Git repository, configure Git with suitable credentials for that repository.

When I try to run it, terraform fails to initialize modules complaining about not sufficient permissions:

git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

But if checkout-action is able to pull the private repository, I hope, it should be possible to configure another action doing the same. For me, it looks like, I have to prepare local git configuration before running Terraform.

Does anyone managed to accomplish something similar? Could you point me to which direction I’d better start diving?

Hi @v1r7u,

Glad to see you in Github Community Forum!
If your Terraform module is in a private Git repository, you will need to ensure the SSH keys has been configured correctly and allow Terraform to access that repository. In other words, please make sure you can git clone that URL firstly:

git clone ssh://git@github.com/foo/modules.git

If that command fails, you need to set up your SSH keys first. GitHub has excellent documentation on how to do that.

For the private repository, please check the usage in the similar ticket here.

Thanks.

1 Like

Thanks for the reply!
I do not have any problems running terraform locally, my problem is in running it inside a github action.

My terraform configuration reference the same repository, where it is located. Thus, if Github Action is able to start and checkout the repository - it’s also should be able to pull modules, as they are hosted inside the same repo, but reference just another commit-sha.

And my problem is that git-checkout step somehow could pull the repo, while the next steps could not do git clone of the same repository.

Thus, I do not see how ssh-keys could help me.

The referenced issue-discussion has a suggestion to run git config --global http.https://github.com/.extraheader 'AUTHORIZATION: basic $GITHUB_TOKEN, which sounds interesting and could solve the problem. Let me try it in the wild once I have a chance.

After several trials, I did not manage to use terraform modules referenced as git over HTTPS

Let me dump what I’ve checked so far, maybe it would help someone who would try to investigate it further.

  1. I tried to run git config --global http.https://github.com/.extraheader 'AUTHORIZATION: basic ${{ secrets.GITHUB_TOKEN }} before running terraform init. Terraform fails with the git’s exit-code 128:

    Could not download module "module1" (src/dev/main.tf:91) source code from
    "github.com/private-org/private-repo?ref=tag1": error downloading 'https://github.com/private-org/private-repo.git?ref=tag1':
    /usr/bin/git exited with 128: Cloning into '.terraform/modules/module1'...
    fatal: could not read Username for 'https://github.com': No such device or address
    
  2. I tried to use a part of the script from deprecated terraform-github-actions repository, but with no success either.

  3. After that I decided to take a look what is happening in official github action. Github’s checkout-action does auth-configuration based on GITHUB_TOKEN. Source code and sample output:

    2020-06-23T13:36:42.4564859Z ##[group]Disabling automatic garbage collection
    2020-06-23T13:36:42.4565210Z [command]/usr/bin/git config --local gc.auto 0
    2020-06-23T13:36:42.4565393Z ##[endgroup]
    2020-06-23T13:36:42.4566570Z ##[group]Setting up auth
    2020-06-23T13:36:42.4566987Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
    2020-06-23T13:36:42.4567488Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
    2020-06-23T13:36:42.4572730Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
    2020-06-23T13:36:42.4573365Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
    2020-06-23T13:36:42.4574310Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
    2020-06-23T13:36:42.4574699Z ##[endgroup]
    2020-06-23T13:36:42.4575010Z ##[group]Fetching the repository
    2020-06-23T13:36:42.4575659Z [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +git_sha:refs/remotes/pull/15/merge
    
  4. Based on the outcome above, I thought that if git is configured locally, I could try to clone the repo on my own. I tried to clone the same repo in another step, but git clone failed with exit code 128

At this point, I gave up, as it’s not vital for our project at this point of time.

If we would come back to this, the next thing I would try is creating dedicated ssh-keys for a technical account:

  • create a technical github-account and register an ssh-key for it;
  • give this account read-access to repos with terraform modules;
  • store the ssh-key as github-secret in the repo with github-action;
  • load the ssh-key from secrets to local .ssh directory during a pipeline run;
  • configure terraform to pull modules over SSH.

If anyone finds out how to accomplish the task, without creating a technical user and managing a separate set of SSH keys for it, I’d love to hear the story :slight_smile:

Hi @v1r7u,

Thanks for your reply!
actions/checkout@v2 will automatically append http extraheader and persist GITHUB_TOKEN for the latter git operations. Could you please add ‘persist-credentials: false’ to the action for a try:

        uses: actions/checkout@v2
        with:
          persist-credentials: false

If possible please share a sample repo for further investigation.

Thanks.

I tried checkout action with and without persist-credentials property (even with true and false values). While repo is public - action works fine. Once I make repo private - it fails.

Sample action:

name: CI

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

jobs:
  ubuntu:
    runs-on: ubuntu-latest

    steps:    
    - uses: actions/checkout@v2
      with:
        persist-credentials: false

    - name: Run a script
      run: |
        git clone https://github.com/v1r7u/github_actions_sandbox.git test_repo_dir
        ls -la test_repo_dir

I used this repo: https://github.com/v1r7u/github_actions_sandbox/tree/2efd7e1784cb5bb64e94b03f888f5512ef72a532

Hi @v1r7u,

Sorry for late response! To clone private repository, please use below format.

git clone https://name:password@github.com/org/repo.git

Please refer to similar ticket for more details.

Thanks.

I want to clone the same repository, not another private repo. If the github-action is able to clone it, likely, I should be able to do the same. Take a look on the example in my previous comment: it tries to clone the repo, which triggered the action.

The route with username/password or ssh key is doable, but then I have to use someone’s account or create a technical one. It’s definitely a more complicated way.