You are correct: with such an approach, we do not find previous issues. But that is not the goal: we want to find issues from what is sent. This is the approach taken by usual code review tools, such as phabricator (https://www.phacility.com/phabricator/) or reviewboard (https://www.reviewboard.org/).
The reason is the following: somebody might start to install my GitHub app on a old, legacy repository that already has 10,000 issues. No one wants to have all these issues flagged at each code review. Instead, one wants to make sure they are not adding more issues at each commit. This is why we are only inspecting the diff.
So I am staying with the current solutions but might still think of integrating the check_suite event when a change is not part of a pull request (for example, when somebody commit a change on a branch without a pull request.
Many kudos for your help and time to clarify how check_run and check_suite work
The example that you gave doesn't seem to be correct.
Let me give a simple example:
- Step1: Alice starts a new branch. Alice add a file (code.py) that contains errors. She commits, pushes the branch and start a pull request. The check run results in a lot of warning, it does not pass.
- Step2: Alice commit a new file on the branch and adds a new file (code2.py) that does not contains any error. The check passes.
In this example, the check on Step2 will fail unless you have corrected the errors included in code.py in the first step. This is because commits are layered sequentially and don't stand independent from one another. That's why only the last commit in a chain is checked.
Hope that helps!