No check suite event for foreign pull reuqests

[codingjoe]

I noticed that the check_suite event that GitHub apps can subscribe to is only triggered for new code push to the repo the app is installed on. The event is not created when another user creates a fork to submit a patch to your repo.

Here is an example, where this happend:

https://github.com/jazzband/pip-tools/pull/784#issuecomment-480476249

Everything is setup following this guide:

https://developer.github.com/apps/quickstart-guides/creating-ci-tests-with-the-checks-api/

I think it't a stretch to say, that you want your CI to run for contributions from other users too.

How do I make this work for all pull requests no matter the origin, or better, can we please just sent the check_suite event for all pull requests?

[lee-dohm]

The team is definitely aware of this problem and friction point, and it's something they want to improve and resolve. The problem is not trivial, so a full solution from our end will probably not happen in the near future.

For now, the recommended workaround is that in addition to listening for check_suite events -- your App also listens to pull_request events which are triggered when a pull request is updated (the synchronize event). If you receive the event for a pull request made from a fork -- you can use that to manually create check runs for the latest commit in the pull request.

I hope that helps!

[dblevins]

For those like me who might find this and wonder how the suggested workaround affects the tutorial.

In Step 1.1 Updating app permissions, perform the following additional steps:

• In the "Permissions" section, find "Pull requests", and select Read-only in the Access dropdown next to it.
• In the "Subscribe to events" section, select Pull Request to subscribe to that event.

In Step 1.2 Add Event Handling, paste this modified version of the check_suite route so that it can also handle a pull_request event.

case request.env['HTTP_X_GITHUB_EVENT']
when 'check_suite'
  # A new check_suite has been created. Create a new check run with status queued
  if @payload['action'] == 'requested' || @payload['action'] == 'rerequested'
    create_check_run
  end
when 'pull_request'
  # A new check_suite has been created. Create a new check run with status queued                                                         
  if @payload['action'] == 'opened' || @payload['action'] == 'updated'
    create_check_run_from_pull_request
  end
end

In Step 1.3 Creating a check run, under helpers do, add this create_check_run_from_pull_request method:

# Create a new check run with the status queued                                                                                           
def create_check_run_from_pull_request
  # # At the time of writing, Octokit does not support the Checks API yet, but                                                            
  # it does provide generic HTTP methods you can use:                                                                                     
  # https://developer.github.com/v3/checks/runs/#create-a-check-run                                                                       
  check_run = @installation_client.post(
    "repos/#{@payload['repository']['full_name']}/check-runs",
    {
      # This header allows for beta access to Checks API                                                                                  
      accept: 'application/vnd.github.antiope-preview+json',
      # The name of your check run.                                                                                                       
      name: 'Octo RuboCop',
      # The payload structure differs depending on whether a check run or a check suite event occurred.                                   
      head_sha: @payload['pull_request']['head']['sha']
    }
  )
end

In Troubleshooting, encourage Github to acknowledge this limitation and workaround.