No check suite event for foreign pull reuqests

I noticed that the check_suite event that GitHub apps can subscribe to is only triggered for new code push to the repo the app is installed on. The event is not created when another user creates a fork to submit a patch to your repo.

Here is an example, where this happend:

https://github.com/jazzband/pip-tools/pull/784#issuecomment-480476249

Everything is setup following this guide:

https://developer.github.com/apps/quickstart-guides/creating-ci-tests-with-the-checks-api/

I think it’t a stretch to say, that you want your CI to run for contributions from other users too.

How do I make this work for all pull requests no matter the origin, or better, can we please just sent the check_suite event for all pull requests?

1 Like

The team is definitely aware of this problem and friction point, and it’s something they want to improve and resolve. The problem is not trivial, so a full solution from our end will probably not happen in the near future.

For now, the recommended workaround is that in addition to listening for check_suite events – your App also listens to pull_request events which are triggered when a pull request is updated (the synchronize event). If you receive the event for a pull request made from a fork – you can use that to manually create check runs for the latest commit in the pull request.

I hope that helps!

1 Like

For those like me who might find this and wonder how the suggested workaround affects the tutorial.

In Step 1.1 Updating app permissions, perform the following additional steps:

  • In the “Permissions” section, find “Pull requests”, and select  Read-only  in the Access dropdown next to it.
  • In the “Subscribe to events” section, select  Pull Request  to subscribe to that event.

In Step 1.2 Add Event Handling, paste this modified version of the check_suite route so that it can also handle a pull_request event.

case request.env['HTTP_X_GITHUB_EVENT']
when 'check_suite'
  # A new check_suite has been created. Create a new check run with status queued
  if @payload['action'] == 'requested' || @payload['action'] == 'rerequested'
    create_check_run
  end
when 'pull_request'
  # A new check_suite has been created. Create a new check run with status queued                                                         
  if @payload['action'] == 'opened' || @payload['action'] == 'updated'
    create_check_run_from_pull_request
  end
end

In Step 1.3 Creating a check run, under helpers do, add this create_check_run_from_pull_request method:

# Create a new check run with the status queued
def create_check_run_from_pull_request
  # # At the time of writing, Octokit does not support the Checks API yet, but
  # it does provide generic HTTP methods you can use:
  # https://developer.github.com/v3/checks/runs/#create-a-check-run
  check_run = @installation_client.post(
    "repos/#{@payload['repository']['full_name']}/check-runs",
    {
      # This header allows for beta access to Checks API
      accept: 'application/vnd.github.antiope-preview+json',
      # The name of your check run.
      name: 'Octo RuboCop',
      # The payload structure differs depending on whether a check run or a check suite event occurred.
      head_sha: @payload['pull_request']['head']['sha']
    }
  )
end

In Troubleshooting, encourage Github to acknowledge this limitation and workaround.

4 Likes

@lee-dohm do I read this correctly that it’s the same behavior for “normal pull requests” - that is standard PRs open from the main repo itself (no forks).

I’m basically experiencing the same behavior, that is not receiving any check_suite events when the PR Is created; only when a new code is pushed to the corresponding branch (which misses the very first time the PR is initially created).

Great alternative!
Just one thing, the action should be “synchronized” instead of “updated” I think.

Thanks,
Pablo

Doing this, can you rerun a check_run individual ?

Because I do not have a way to checkout files from a revision number, when this revision belongs to a fork.
Whereas if I do a commit, the pull_request event is trigger and I can do git pull origin pull/{pr_number}/head. and update the status of the check_run.

Any one found a solution for this problem?