We have a workaround now where we don't rely on Checks at all but instead have a scheduled build in Azure Pipelines. However, I noticed while playing with GitHub Actions that the problem doesn't seem to be intrinsic to Checks. In https://github.com/foolip/wpt/runs/56065527, I pushed the same ref to two different branches, and that resulted in two different check suites which ran at different times. Screenshot: Is GitHub Actions using internal APIs available only to GitHub, or could any CI system achieve the same?
... View more
In https://github.com/web-platform-tests/wpt we do some amount of testing for each push to the master branch, but also have epochs/daily and epochs/weekly branches trailing the master branch which we've used to trigger slightly different jobs. (The project is for testing browsers, and we test stable/beta versions less often than dev versions of the browser.) This setup is working with Mozilla's Taskcluster CI system, which is currently using the old Status API integration. However, I'm unable to do the equivalent thing with Azure Pipelines which is using the Checks API integration. https://github.com/web-platform-tests/wpt/issues/14836 has the details of what I have attempted. It appears that because checks are triggered for a commit sha and not a symbolic ref (branch name), that if the push to master has already triggered checks, later pushing another branch pointing to the same commit will not create a new check suite or cause any check runs to rerun. I would like to confirm that my understanding of the Checks API is correct, and if this is intentional. Are there any known workarounds? It does appear that some uses of https://docs.microsoft.com/en-us/azure/devops/pipelines/build/triggers are fundamentally incompatible with the Checks API model, but I'd like to confirm this before reporting to the Azure Pipelines team.
... View more