GitHub actions are severely limited on PRs

Hi there,

I think GitHub Actions are ruined by a very tiny detail which should be fixed ASAP. More specifically, GitHub actions running on PRs are so severerly limited they often become useless.

On forks and PRs

I think everyone will agree that PRs from forks are pretty much the de facto standard way of collaborating on GitHub. For instance, let’s say you want to contribute to Ruby on Rails. The recommended way is to :

On Actions

Now, what’s the point of actions? Well, they can solve some of those problems:

  • Running some checks on the code base (for instance, a linter)
  • Running a test suite
  • Getting some metrics about the code base (e.g. sloc count)
  • Benchmarking the code (finding out how fast the code goes)
  • Measuring the products (what’s the size of the binary once built, or the size of the JS once mifinied/gzipped)

As you can see, some problems answer a boolean question : for instance the test suite either passes or not. That can be the case of a linter too, which either has some work to do or doesn’t need to. For those problems, actions are fine, they work just great.

Now there are other problems which do not have a boolean answer. For instance a measuring the speed of the code or the memory usage of a library isn’t a yes/no kind of question. Those problems simply cannot be currently solved by actions.

The problem

Funnily enough, the problem is not answering the question (e.g. how fast is the code), it’s about reporting the answer. Indeed, there are two ways to output something from a GitHub action:

* First, using a return code. If you return a non-zero code, then the action is considered a failure. Otherwise, it’s a pass. That works great for boolean problems!

* Otherwise, you can use the GitHub API (using a provided GITHUB_TOKEN) to do pretty much whatever you want (create a new issue, write a comment, add a label, request a review, etc…).

However, for security reasons, the team at GitHub has decided to issue a GITHUB_TOKEN that has read-only access to the entire GitHub repository when the action is run from a fork.

In other words, when you create a pull request on Ruby on Rails, any action running on your PR cannot do anything at all on the Ruby on Rails repository. That includes doing anything at all on your very own PR. I repeat: the only output the actions running on your PR can have are “fail/pass”. That’s it. Of course it’s a good idea to limit what those actions could do, but at the moment they simply cannot do anything at all apart from returning a pass/fail status. Namely, they cannot comment on your PR, they cannot add labels to your PR, etc…

That’s a really bad problem, that’s been reported several times, and nothing has been done about this. It badly limits the usefulness of GitHub actions. By the way, “HttpError: Resource not accessible by integration” is the message you’ll get when you’ll be bitten by this, as an action developer.

Some possible solutions

I can imagine two solutions to this:

  • Provide actions running on PR with a GitHub token that can act on the PR itself only.
  • Allow Actions to return more than a “fail/pass” result. Maybe at least a string that would be commented on the PR would be a good start?

Links

32 Likes

Currently, the GITHUB_TOKEN only has read permission to pull requests when access by forked repos. 

This is refer to official document: https://help.github.com/en/actions/configuring-and-managing-workflows/authenticating-with-the-github_token#permissions-for-the-github_token 

For all your requirements on PR from forked repo, I would encourage you sharing those in the Feedback form for GitHub Actions. Thank you for your understanding. 

Seriously? A canned answer? I obviously did read the documentation. You’ve missed the entire point. That’s shameful.

3 Likes

Hi @ecco , 

I really understand your scenario and you are looking forward to a write permission of GITHUB_TOKEN to act on pull request from fork.  

But I am so sorry that this is not supported now. 

Feedback form for GitHub Actions is a more suitable site to share your idea : adding some additional scopes to the GITHUB_TOKEN to enable commenting , labeling on PRs. The feedback will be checked by GitHub Engineering team directly. 

>>Allow Actions to return more than a “fail/pass” result. Maybe at least a string that would be commented on the PR would be a good start?

There is a set-output command which could set output variable for a step with string value.  You could use the output variable in the next steps of current job. https://github.com/actions/toolkit/blob/master/docs/commands.md#set-outputs 

But unfortunately,  it still could not be commented on the PR.  

2 Likes

May this be a working workaround for this problem?

https://github.com/nyurik/auto_pr_comments_from_forks

OK, here’s an experiment I made where I was sure it would work and be a workaround for the problem you describe. But I was wrong…

First, create an access token in your settings with sufficient “repo” permissions on the repo you have set up your actions.

Store that token as secret, e.g. RW_TOKEN, in the repo you have set up your actions.

Create an action YML:

name: create issue on PR/push

on: [pull_request,push]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - name: Run a multi-line script
      env:
        RW_TOKEN: ${{ secrets.RW_TOKEN }}
      shell: bash
      run: |
        curl --request POST \
        --url https://api.github.com/repos/myaccount/mytestrepo/issues \
        --header 'authorization: Bearer'" $RW_TOKEN" \
        --header 'content-type: application/json' \
        --data '{
          "title": "Automated issue for commit: ${{ github.sha }}",
          "body": "This issue was automatically created by the GitHub Action workflow **${{ github.workflow }}**. \n\n The commit hash was: _${{ github.sha }}_."
          }'

This code, using the RW_TOKEN copied and run in a terminal on my machine executes just fine. Triggered by a push, the action runs just fine and create the issue. Triggered by a PR from a forked repo , it fails with:

{
  "message": "Bad credentials",
  "documentation_url": "https://developer.github.com/v3"
}

THIS IS JUST CRAZY, isn’t it??? 

As frustrating as this is, it does make sense that the same issue would apply to secrets as it does for the built in GitHub Token. Otherwise that would be a major security flaw. However this is a MAJOR oversight on GitHub’s part. There needs to be some solution to handle this extremely common workflow.

1 Like

This actually makes a majority of use cases that I have for GitHub Actions useless and would force me to look to external integrations. Can someone from GitHub comment on whether this is being investigated? Or is this “just the way things are going to be”?

I have seen mention elsewhere that this is (was) being looked into. I’ll see if I can dig the details up again.

Agreed - the #1 use for github actions in my mind is around making the contribution process simpler, automated, and bullet proof. But since we can’t support any features that matter with contributed updates then it is useless.

All we need is the ability to automatically use the workflow from the TARGET branch instead of the PR itself – but giving it access to the branch being merged. It really doesn’t seem that difficult? Although I am sure I am missing something?

I’ve updated the add-pr-comment action I maintain to make this as easy as possible until GitHub comes up with a better solution. I created a simple proxy app which can be deployed via a Cloud Run Button, and allows specifying the proxy URL as an override in the action. I imagine most use will fall in the Cloud Run free tier.

1 Like

I accomplished this without a github app actually. @mshick this actually maeks PR comments on forked PR as well as run full checks / linting on them so it may be a good example of how you could accomplish your action as yours would be 10,000x simpler :slight_smile:

So the artifacts generated from the fork are available to the scheduled run on the upstream / original repo? Great discovery, thanks!

I’ll look at incorporating something similar, could definitely be useful for those who don’t mind the lag time and would rather not host a proxy.

1 Like

Only when using the REST API - the @actions/artifact specifically restricts it to the current workflow run only and doesn’t make it available to the workflow as a whole.

I was planning at some point to bundle it up into a easy workflow to use, if possible.