I think GitHub Actions are ruined by a very tiny detail which should be fixed ASAP. More specifically, GitHub actions running on PRs are so severerly limited they often become useless.
On forks and PRs
I think everyone will agree that PRs from forks are pretty much the de facto standard way of collaborating on GitHub. For instance, let’s say you want to contribute to Ruby on Rails. The recommended way is to :
- Go on github.com/rails/rails
- Click the fork button
- Work on your own repository
- Eventually create a PR at github.com/rails/rails from your own github.com/foobar/rails repository
Now, what’s the point of actions? Well, they can solve some of those problems:
- Running some checks on the code base (for instance, a linter)
- Running a test suite
- Getting some metrics about the code base (e.g. sloc count)
- Benchmarking the code (finding out how fast the code goes)
- Measuring the products (what’s the size of the binary once built, or the size of the JS once mifinied/gzipped)
As you can see, some problems answer a boolean question : for instance the test suite either passes or not. That can be the case of a linter too, which either has some work to do or doesn’t need to. For those problems, actions are fine, they work just great.
Now there are other problems which do not have a boolean answer. For instance a measuring the speed of the code or the memory usage of a library isn’t a yes/no kind of question. Those problems simply cannot be currently solved by actions.
Funnily enough, the problem is not answering the question (e.g. how fast is the code), it’s about reporting the answer. Indeed, there are two ways to output something from a GitHub action:
* First, using a return code. If you return a non-zero code, then the action is considered a failure. Otherwise, it’s a pass. That works great for boolean problems!
* Otherwise, you can use the GitHub API (using a provided GITHUB_TOKEN) to do pretty much whatever you want (create a new issue, write a comment, add a label, request a review, etc…).
However, for security reasons, the team at GitHub has decided to issue a GITHUB_TOKEN that has read-only access to the entire GitHub repository when the action is run from a fork.
In other words, when you create a pull request on Ruby on Rails, any action running on your PR cannot do anything at all on the Ruby on Rails repository. That includes doing anything at all on your very own PR. I repeat: the only output the actions running on your PR can have are “fail/pass”. That’s it. Of course it’s a good idea to limit what those actions could do, but at the moment they simply cannot do anything at all apart from returning a pass/fail status. Namely, they cannot comment on your PR, they cannot add labels to your PR, etc…
That’s a really bad problem, that’s been reported several times, and nothing has been done about this. It badly limits the usefulness of GitHub actions. By the way, “HttpError: Resource not accessible by integration” is the message you’ll get when you’ll be bitten by this, as an action developer.
Some possible solutions
I can imagine two solutions to this:
- Provide actions running on PR with a GitHub token that can act on the PR itself only.
- Allow Actions to return more than a “fail/pass” result. Maybe at least a string that would be commented on the PR would be a good start?