Need a way to authenticate to AWS beyond IAM users

Hi,

love  GitHub Actions and I am keen to migrate my employer’s CI/CD processes to it. Right now there are a couple of unfortunate blockers stopping that. I thought it would be worth making a feature request here for a) visibility and b) to see if anyone else is in this situation.

  1. As far as I can see, it is only possible to set secrets at a per-repo level. Ideally this would be at a per-org level as we have 1000+ repos and repo-level management becomes unwieldy/impossible at that level. EDIT : I see a request for this already.

  2. Right now, the only way to authenticate to our AWS environments is to create an IAM user and store its access key ID and secret access key as secrets in Github. This is a blocker for us as we don’t use IAM users at all. We want to have granular roles and different repos should have different levels of access - creating a huge number of IAM users to cater to this won’t work.

Ideally  GitHub Actions would provide an environment variable with a JWT that identifies a handful of properties about the job. Things like repo name, org name, etc - basically the event.json  signed by Github. We would then be able to use this JWT to retrieve AWS credentials through the AssumeRoleWithWebIdentity API. We’d then be able to create policies limiting access, etc - and not need to store any secrets in the repo.

Interestingly, there is already a JWT in the Actions runtime environment named ACTIONS_RUNTIME_TOKEN. However, this doesn’t seem to have many useful fields and is primarily used for authentication to the internal cache?

Additionally, self-hosted runners are something we are keen to avoid - and don’t _really _solve the problem. While they would give us an instance profile to start with, we still wouldn’t have a Github-signed copy of the event.json – so we’d have to trust developers not to play funny games with that file. And as a cheeky developer, I know that can’t be relied upon. Also, Github-hosted runners are a big part of the appeal of Github Actions.

Is anyone else in this situation? Is this a use case GitHub Actions aims to support?

13 Likes

Hello @aidansteele,

I am one of the product managers for GitHub Actions and improvements in continuious delivery workflows is something we are focued on for the next couple of quarters.  One of the specific areas that I am looking at right now is a better mechansim to make certian types of secrets available to the workflow particularly for clould deployment. 

We have been looking at signing the event payload for some other scenarios so getting some feedback that it would be useful here is great.

Another idea I had which I believe would provide additional security is to enable certian workflows to run only on approval from members of a particualr team.  Upon approval the workflow would aquire credentials and then pass those credentials to a deployment workflow.  In this model you could have the same deployment workflow for dev and test environments as you do for production but certian settings like credentials would vary based on the environment and approvals.

  Thoughts?

5 Likes

@chrispat Thanks for the very speedy and detailed response!

It’s great to hear that this is something you’re already thinking about. With respect to signing the event payload, that’s great to hear. While I mentioned a specific AWS API in my first post, I wouldn’t want perfect to be the enemy of good - I’d be happy with  any  form of signed event payload, because we can always do some plumbing on our end and create an API to verify the payload and vend credentials to the action.

That idea you mention about approvals being part of a workflow would be  fantastic. I haven’t given it too much thought yet, but it feels like a good fit for integrating with Github’s existing (but a bit barebones) support for “deployments”. Perhaps deployment targets could be configured (as workflow files?) and they specify the name of a team allowed to execute the workflow. The workflow could then update the status in the Deployments tab.

I guess where this ties in with the first idea is that when the privileged team member starts the workflow, the signed payload would include the  actor that approved it and the  name  of the approved step. Maybe with some metadata too (e.g. maybe they would type in a message with a link to a Jira ticket, etc). Our aforementioned API could then see “it’s been approved by person X from team Y, therefore it is ok to vend the highly privileged credentials”.

^ That was all a bit wordy, let me know if I failed to get the idea across!

2 Likes

Hi @chrispat 

We share our use case with @aidansteele where we want to invoke the theAssumeRoleWithWebIdentity API to obtain STS AWS credentials. It would also be great if the tokens were compatible with how AWS manages claim permissions.

Having a signed JWT when performing an action would be immensely valuable to us. It would eliminate the need to have to manually load secrets into Github, which is a time-consuming manual process since we have hundreds of repos to manage.

Currently (it appears) secrets in Github are not restricted by branch, so someone with the ability to create PRs could extract the secrets. With a signed payload, you could embed both the branch and action type in the claim, e.g. the sub field could be “https://github.com/facebook/react:bugfix-branch:pull-request”, where there would be limited rights associated with the IAM role. The IAM role associated with sub “https://github.com/facebook/react:master:merge” could however have rights to upload to S3.

Thanks

Anand

4 Likes

I was also looking for something similar. Is that jwt signing available or have people found workarounds in the meanwhile?

I think the approach that Aidan described is the most elegant one, maybe a bit geared towards AWS.
Another more generic approach i could think of would be signing the github/workflow/event.json file within the runner with a key by github (maybe global, could also be per organization).
This could work in a similar way like AWS does it, or given that actions run on Azure, similar to what they do with attested metadata (Azure Instance Metadata Service for Windows - Azure Virtual Machines | Microsoft Docs)

With that signed and trusted information one could implement a small API that hands out credentials. This is still less favorable than the other approach, but more generic and could work for many other use cases.

I have tried a few more options, that are sort of working, but are still quite hacky. Sharing that others don’t have to try.

What we tried is having a GitHub App, that reacts on workflow_run webhooks. The webhooks can be verified. The App (in the test case running in an AWS lambda) would hand out an intermediate AWS role with the repository attached as tags. It writes the temporary credentials as repository secrets, and actions could pick it up from there, and do an actual assume role. The target role in AWS then checks for the session tag, and therefore and identify the repo. More granularity isn’t possible with this approach.
This works, but it looks like a workflow run takes all the settings and secrets at exactly the start time, so there is no way the lambda behind the webhook is fast enough to issue the credentials in time for the run.
So your first run would always fail. Not really what you want for your CI/CD.

You can work around by that with creating one dummy workflow (note that a second job within a workflow does not work). This basically triggers the writing of credentials. You then have a second workflow that checks for the end of the first one and runs the actual things you want to run.
This again works, but you loose the initial event, and are also running actions in another way than everyone else does.
on: workflow_run: workflows: ["trigger-wf"] branches: [master] types: - completed

In the end it again boils down to what @aidansteele already posted. We need some sort of signed metadata and then it all works. Everything else is currently just more or less quirky workarounds

1 Like

Event signing :+1:, Approval Workflows :-1:.

We also need 100% automation, a JWT signed with all the event data that can be verified by using a publicly available JWK is really the only solution that will work for us. This JWT needs to be generated for every run including Pull requests from forks, but of course should have indicators that it is coming from a fork.

Just need the token to have:

  • KID
  • Signature
  • Issuer
  • Target Org/Repo/Branch
  • Source Org/Repo/Branch
  • Run Number
  • Url to rest of job data

Update for those following along at home: this feature has almost shipped. Here’s an example of how to use it: https://github.com/patterson-io/oidc-test/blob/4e4d11d629bd5fdc9a47f04a2f2e5743109fe029/.github/workflows/oidc.yml. Not yet documented, but I imagine that will happen when it’s officially launched.

Thanks so much @chrispat and team!