In https://github.com/web-platform-tests/wpt we do some amount of testing for each push to the master branch, but also have epochs/daily and epochs/weekly branches trailing the master branch which we've used to trigger slightly different jobs. (The project is for testing browsers, and we test stable/beta versions less often than dev versions of the browser.)
This setup is working with Mozilla's Taskcluster CI system, which is currently using the old Status API integration. However, I'm unable to do the equivalent thing with Azure Pipelines which is using the Checks API integration. https://github.com/web-platform-tests/wpt/issues/14836 has the details of what I have attempted.
It appears that because checks are triggered for a commit sha and not a symbolic ref (branch name), that if the push to master has already triggered checks, later pushing another branch pointing to the same commit will not create a new check suite or cause any check runs to rerun.
I would like to confirm that my understanding of the Checks API is correct, and if this is intentional. Are there any known workarounds? It does appear that some uses of https://docs.microsoft.com/en-us/azure/devops/pipelines/build/triggers are fundamentally incompatible with the Checks API model, but I'd like to confirm this before reporting to the Azure Pipelines team.
Solved! Solved! Go to Solution.
Yes, Checks run against an individual commit. To my knowledge, they do not take into account the number of symbolic refs that may or may not refer to that individual commit because that will change over time. However, since that commit is immutable, so long as the Checks are looking at the contents of the repository and not anything else, new runs of the Checks should create the same result. However, a Check Suite can be manually rerun through the UI or requested through the API. You could potentially create an Action or a Probot App to create the workflow you desire by requesting a rerun of a Check Suite when a new branch is pushed and the Suite is already completed?
We have a workaround now where we don't rely on Checks at all but instead have a scheduled build in Azure Pipelines.
However, I noticed while playing with GitHub Actions that the problem doesn't seem to be intrinsic to Checks. In https://github.com/foolip/wpt/runs/56065527, I pushed the same ref to two different branches, and that resulted in two different check suites which ran at different times. Screenshot:
Is GitHub Actions using internal APIs available only to GitHub, or could any CI system achieve the same?
According to the main.workflow that I'm looking at in your sample repository, the workflow triggers on `push`. Checks works slightly differently in that it creates a Check Suite object for a commit when it is pushed and then notifies the GitHub App via a `check_suite` event, rather than a `push` event.
I hope that helps!
It's all a matter of subscribing to the events you want when you create or update your GitHub App:
Alright, thanks for your help. I'll point developers at Azure Pipelines and Taskcluster at this thread when suggested that they match GitHub Actions's behavior on this.