I noticed that the check_suite event that GitHub apps can subscribe to is only triggered for new code push to the repo the app is installed on. The event is not created when another user creates a fork to submit a patch to your repo.
Here is an example, where this happend:
Everything is setup following this guide:
I think it't a stretch to say, that you want your CI to run for contributions from other users too.
How do I make this work for all pull requests no matter the origin, or better, can we please just sent the check_suite event for all pull requests?
The team is definitely aware of this problem and friction point, and it's something they want to improve and resolve. The problem is not trivial, so a full solution from our end will probably not happen in the near future.
For now, the recommended workaround is that in addition to listening for check_suite events -- your App also listens to pull_request events which are triggered when a pull request is updated (the synchronize event). If you receive the event for a pull request made from a fork -- you can use that to manually create check runs for the latest commit in the pull request.
I hope that helps!