Specify check suite when creating a checkrun #24616

jwgmeligmeyling · 2020-06-16T22:48:39Z

jwgmeligmeyling
Jun 16, 2020

When creating a check run through the API, a check suite is newly created unless a check suite for the requesting application has already been made, in which case that check suite seems to be reused.

This leads to issues however: due to Github Actions workflows, and each Workflow Run (workflow/event) having its own check suite, multiple check suites may exist for a given commit SHA. As a result, annotations generated in a Github action during a step executed in the “pull request” event, end up in the “workflow” for the "push’ event.

Through hoops, it is possible to find the check run ID during a Github Action: one can look up the Workflow Run by the run_id and then extract the Check Suite ID from the checkSuiteUrl. However, there is no way to specify the Check Suite ID when creating a new check run (its ignored in the request) and neither there seems to be any dedicated endpoint for creating check runs for a particular check suite.

It would be great if it were possible to more easily find the corresponding Check Suite for a Workflow Run, and if it were possible to create Check Runs for this particular Check Suite rather than have one picked “randomly” for you.

See also the issue thread jwgmeligmeyling/pmd-github-action#4 and proof of concept code in jwgmeligmeyling/pmd-github-action#3

Answered by francisfuzz

Jun 26, 2020

jwgmeligmeyling:

It would be great if it were possible to more easily find the corresponding Check Suite for a Workflow Run, and if it were possible to create Check Runs for this particular Check Suite rather than have one picked “randomly” for you.

@jwgmeligmeyling - Welcome back to the Support Community and thanks for this feedback! 🙇‍♂️

After reviewing your particular case, I’d like to confirm here that it is not currently possible to create a check run associated with a particular check suite with the GitHub Actions-provided GITHUB_TOKEN. I acknowledge that it is something that would be valuable given your use case.

We’re always working to improve GitHub and the GitHub Support Co…

View full answer

francisfuzz · 2020-06-26T00:08:01Z

francisfuzz
Jun 26, 2020

jwgmeligmeyling:

It would be great if it were possible to more easily find the corresponding Check Suite for a Workflow Run, and if it were possible to create Check Runs for this particular Check Suite rather than have one picked “randomly” for you.

@jwgmeligmeyling - Welcome back to the Support Community and thanks for this feedback! 🙇‍♂️

After reviewing your particular case, I’d like to confirm here that it is not currently possible to create a check run associated with a particular check suite with the GitHub Actions-provided GITHUB_TOKEN. I acknowledge that it is something that would be valuable given your use case.

We’re always working to improve GitHub and the GitHub Support Community, and we consider every suggestion we receive. Would you mind submitting this through our official product feedback form so that our product team can track your request? That’s the best place to share requests like these in consideration for future iterations of GitHub features, including Actions.

3 replies

carlspring Mar 29, 2023

@francisfuzz ,

Ping? Please, see the comments below.

ruben-hoenle-fnt Jul 26, 2023

@francisfuzz
Ping?! Please, see the comments below!

samus-aran Aug 7, 2023
Maintainer

Hi @ruben-hoenle-fnt 👋

Thanks for sharing and we appreciate your patience. You can check our public roadmap to see the kinds of work we are focusing on for the coming year. Smaller fixes won’t be listed, but are often included when work is happening adjacent to it.

While it has been a while since we have updated on this thread, the absence of an update is simply that - there has been no update because there is so far nothing to report. That doesn't mean it isn't being worked on - it might simply be that we're unable to share details about work that isn't ready for release.

As a GitHub user, the most you can do is to share your feedback on this Discussion space with the correct tags (include examples), and trust that we are always working to make GitHub as awesome as possible. We can't guarantee that every bit of feedback sees action, but we are paying attention.

jwgmeligmeyling · 2020-06-26T08:25:49Z

jwgmeligmeyling
Jun 26, 2020
Author

Thanks for your response! I’ve submitted the product feedback form.

4 replies

This comment was marked as spam.

Sign in to view

bewuethr Aug 12, 2023

Is it really useful to copy-paste ChatGPT answers here ("last knowledge update in September 2021")?

queenofcorgis Aug 14, 2023
Maintainer

@Akashh2151 While we encourage non-AI answers to questions in community, we also love experimenting and using new technologies. We ask if you do use ChatGPT or similar programs, please disclose what software and model version number you used to generate the response you used in your reply in order to promote transparency in the community.

ChatGPT answers in a tone that sounds like a tech support professional and while we support trying out this software, it is important to verify that they are solutions before posting otherwise it may mislead users into thinking that they are receiving official responses when they are not.

Per ChatGPT's website, the program does have limitations like:

"May occasionally generate incorrect information"
"May occasionally produce harmful instructions"
"Limited knowledge of world and events after 2021"

jwgmeligmeyling Aug 14, 2023
Author

So instead of reporting this junk as spam we encourage it and ask for the version number? This whole thread is getting more insane by the day.

EnricoMi · 2020-11-30T09:36:35Z

EnricoMi
Nov 30, 2020

Is this product feedback submitted by @jwgmeligmeyling publicly visible? What is the best way to track progress of this issue?

0 replies

jwgmeligmeyling · 2020-11-30T10:33:06Z

jwgmeligmeyling
Nov 30, 2020
Author

No I don’t think its publicly available. I didn’t receive any response as well. I’ve filed another support ticket, maybe that gets things going.

0 replies

francisfuzz · 2020-11-30T19:15:03Z

francisfuzz
Nov 30, 2020

@EnricoMi - Great question! Because product feedback submitted through this form isn’t publicly visible, it’s not possible to track its progress. However, it may be worth sharing that our team announced the GitHub public roadmap to give everyone more transparency into what each of our teams are planning and to help everyone understand our product vision. It’s publicly viewable if you’re interested in following what our team is working on next, though it doesn’t currently show any feedback submitted from that form. I hope this helps!

0 replies

jonico · 2020-12-22T08:14:54Z

jonico
Dec 22, 2020

It might be worth trying out to use the token of a separate GitHub App when creating the check run. Here is an action that retrieves a temporary token from a GitHub App created just for this purpose: https://github.com/peter-murray/workflow-application-token-action

0 replies

NasAmin · 2021-01-19T09:51:14Z

NasAmin
Jan 19, 2021

Yes I have the same problem.
When you have multiple workflows triggered on a pull_request event and one of the workflows is creating check runs using the Create Check API, the check runs end up randomly in one of the workflows.

I have also submitted a suggestion on the feedback form.

0 replies

dobegor · 2021-01-23T09:40:23Z

dobegor
Jan 23, 2021

@francisfuzz is there any update on this issue? I wasn’t able to find it on the roadmap you’ve mentioned.

0 replies

eblot · 2021-01-26T08:15:01Z

eblot
Jan 26, 2021

I was expecting to use several runners to perform distinct, unrelated checks, but the results appear in a randomly picked workflow tab. Is there any known workaround for now? Thanks.

(note: it sounds weird to have this thread flagged as “solved” :-))

0 replies

francisfuzz · 2021-02-10T00:58:36Z

francisfuzz
Feb 10, 2021

@dobegor - I don’t have any updates to share on my part nor am I able to provide any in the future. The public roadmap is still the best place to keep up with upcoming features.

0 replies

lukash · 2021-03-10T16:12:04Z

lukash
Mar 10, 2021

Hello @francisfuzz, this seems to be a “minor” bug from the roadmap PoV, and is unlikely to ever be listed there. Hence this response from you is not really useful at all, as we don’t even know whether the issue is still being tracked by Github internally or it just “fell through the cracks”.

It’s a shame especially because at least on the surface this seems like it should be an easy fix to actually use the Check Suite ID. For my use case it’s causing this: Report check item appears under the wrong workflow name · Issue #39 · ScaCap/action-surefire-report · GitHub and again, it’s a shame, because that action allows reasonable and very simple test result reporting via workflow annotations, which is clear and easy to find/read even by outside contributors. There are very few options with regards to test result reporting that don’t require outside resources!

I have submitted another feedback form for this, hopefully it can get some more attention. Thank you.

0 replies

thegecko · 2021-05-10T11:47:28Z

thegecko
May 10, 2021

This issue makes it difficult to locate checks for specific workflows and is serious enough for us to consider moving away from GH actions 😦

0 replies

inket · 2021-05-27T06:27:37Z

inket
May 27, 2021

This is a BREAKING issue for GitHub Actions, and it’s honestly mindblowing that it’s come down to “submit feedback and get no answer”. We have zero acknowledgment that our feedback was even seen by anyone working on GitHub Actions.

It’s especially sad since we all know this would be a very easy fix. You can literally put an intern on it they’ll get it done in less than a day.

1 reply

radiomix Oct 19, 2022

It reminds me of the old 🪟 behavior It's not a bug. It's a feature

jwgmeligmeyling · 2021-06-07T21:38:16Z

jwgmeligmeyling
Jun 7, 2021
Author

Sad to see this was still not communicated to the appropriate folks. It is a trivial fix. A dozen people have requested this feature.

And annotations are practically useless as it is.

Someone made a stinky API decision and successfully managed to sweep it under the carpet. Of course we want to be able to specify to which check run annotations should be added, rather than have one picked randomly for us! It’s just common sense.

0 replies

MostefaKamalLala · 2021-07-29T16:42:04Z

MostefaKamalLala
Jul 29, 2021

This is seriously needed by my team. We moved to github actions and we are even paying for the service. We cannot have test results, annotations spread all over the place randomly. When will we get an update on the API?

0 replies

IanMoroney · 2022-05-27T08:56:48Z

IanMoroney
May 27, 2022

If people upvote the following discussion, github might get around to fixing this problem:

  <a href="https://github.com/github/feedback/discussions?discussions_q=Allow%2Bspecifying%2Bworkflow%2Brun%2Bwhen%2Bcreating%2Ba%2Bcheckrun%2Bfrom%2Ba%2BGitHub%2Bworkflow" target="_blank" rel="noopener nofollow ugc">GitHub</a>

Discussions · github/feedback

Public feedback discussions for: GitHub for Mobile, GitHub Discussions, GitHub Codespaces, GitHub Sponsors, GitHub Issues and more! - Discussions · github/feedback

The more votes we have, hopefully the higher visibility it’ll get

0 replies

gabsmprocha · 2022-08-09T20:11:55Z

gabsmprocha
Aug 9, 2022

Any update about this problem?

4 replies

jwgmeligmeyling Aug 9, 2022
Author

Not that I am aware of.

gabsmprocha Aug 9, 2022

oh my god, two years later, sad 😢

jwgmeligmeyling Aug 9, 2022
Author

Yea I was one of the early adopters of the annotations API back when it was in beta. If it even ever left beta 🤣

eblot Aug 9, 2022

Update is something like "we just don't care". As with many other features (such as the terrible PR review workflow), the implementation is half-complete, half-debugged, half-usable. The only thing that is not halved is the price per seat.

devstrat · 2022-10-04T12:38:00Z

devstrat
Oct 4, 2022

I guess I want too much: having test results properly attached to the correct workflow while having parallel runs, this technology is too advanced for 2022. We're paying customers btw. 2 years and nothing

1 reply

radiomix Oct 19, 2022

💯 LOL!

austinpray-mixpanel · 2022-10-04T14:14:03Z

austinpray-mixpanel
Oct 4, 2022

You can see how confusing this is for our users. The "flake8" check is a python linting check:

0 replies

dobegor · 2022-10-04T14:27:42Z

dobegor
Oct 4, 2022

@francisfuzz is it possible for Github to re-evaluate the priority of this issue?

3 replies

francisfuzz Oct 5, 2022

👋 @dobegor,

While I understand this feature is high priority for you, the Product team takes several factors into consideration when prioritizing features. The Product team is aware of this feedback as it lives in this discussion, but I can't make any guarantees about this feature's prioritization.

sifive-eblot Oct 5, 2022

This is not a "Feature" request. It is about fixing a nasty bug that plague an existing feature for over 2 years.

As long as "making things usable" is considered as a "new feature" we can wait forever. This existing feature is deeply broken - randomly linking reports - and the whole point of this issue is not to add something new but making it work...

radiomix Oct 19, 2022

Thread started on Jun 17, 2020
Now at Oct. 19, 2022 LOL

Feels like the old MS user experience: Well come back to the world of

vstirbu · 2022-10-24T16:33:16Z

vstirbu
Oct 24, 2022

This is the proper way to add feedback in actions, equivalent with the check runs in GitHub Apps: https://github.blog/2022-05-09-supercharging-github-actions-with-job-summaries/

10 replies

zolex Oct 24, 2022

Depending on the complexity of your workflow files you can require the workflow to merge. See below the modelcard-test is the name of the workflow:

this is what I mean, which is not possible when just putting text in summary, as suggested above ;)

vstirbu Oct 24, 2022

The workflow in the screenshot has a step that outputs the summary.

zolex Oct 24, 2022

This whole topic is about programmatically added checks...

jwgmeligmeyling Oct 24, 2022
Author

Wouldn't call the suggested approach the proper way. It might work fine for your goals though. However, the Checks and Annotations API was announced with quite the buzz (I even remember a mail campaign) for exactly this purpose: https://github.blog/2018-05-07-introducing-checks-api/

bbrzoska Nov 18, 2022

A single Github Action can create multiple checks. Some can pass, others fail, and others still be neutral in outcome. Failures of some may be set as blocking merges, others not. Using the output summary isn't a solution for this problem.

AlexPerathoner · 2022-12-26T21:29:49Z

AlexPerathoner
Dec 26, 2022

Same problem on my side. Definitely a problem, or "missing feature", if you prefer.

0 replies

leviplj · 2023-01-04T17:52:32Z

leviplj
Jan 4, 2023

Is GH going to fix this or should we give up on the checks?

1 reply

jwgmeligmeyling Jan 4, 2023
Author

You should give up. Ive waited for years.

Talked to my GH Enterprise rep about it, they push on sarif uploads trough a Github Advanced Security upsell. Mind you, it can process a dozen files at most before crashing on you. The annotation API actually works better.

JustinOhms · 2023-03-09T01:00:12Z

JustinOhms
Mar 9, 2023

Well this seems to explain why my check runs keep keep ending up associate with my push action and not the action driven by the PR. I can't believe this is an issue. If you cannot control what the check run is actually associated with it's next to useless.

1 reply

sarnold Aug 21, 2023

I can't believe I'm still reading this...

carlspring · 2023-03-28T19:04:41Z

carlspring
Mar 28, 2023

@francisfuzz , @maxprit ,

Can we raise the severity of this? It's not just a desired feature, it's a total misimplementation of how it should work. As you can see, quite a few other developers feel this way and are affected by it.

What do we need to do in order to get the attention of the Github developers/managers? Is there a issue tracker where this can be reported? How do escalations work?

1 reply

steve-todorov Mar 28, 2023

We are also experiencing this issue and it is quite disappointing to see such a rookie mistake being made on such an important feature.
To be honest this feature makes absolutely no sense to even exist or be available to the general public, because practically it doesn't work.

Nobody ever has just a hello world project - people have 1, 2 sometimes 3 workflows that deal with the different phases / stages.
I really hope this gets more attention.

dev-hanz-ops · 2023-04-13T18:29:37Z

dev-hanz-ops
Apr 13, 2023

There is a workaround. It's possible to publish the check-run in a dedicated, separate check-suite by using a token for a GitHub app, instead of the regular GITHUB_TOKEN.
As example, we use this action:

create a GitHub App (permissions e.g. Read and write access to actions, checks, code, discussions, and pull requests - possibly too broad for your case)
create a private key for it and store it in your GH Actions secrets
install the app to your org / repo

      - id: app_token
        uses: tibdex/github-app-token@b62528385c34dbc9f38e5f4225ac829252d1ea92
        if: always()
        env:
          OPENSSL_CONF: /dev/null  # https://github.com/tibdex/github-app-token/issues/54
        with:
          app_id: your-app-id
          private_key: ${{ secrets.PRIVATE_KEY_YOUR_APP }}

      - name: Publish Test Results
        uses: EnricoMi/publish-unit-test-result-action@v2
        if: always()
        with:
          # when using the regular GITHUB_TOKEN, the check-run created by this step will be assigned to a
          # random workflow in the GH UI. to prevent this, we can force the check-run to be created in a separate
          # check-suite, which is created automatically if we use the credentials of a GitHub App
          github_token: ${{ steps.app_token.outputs.token }}

update: your workflow should create a comment like the default one using the github-actions user, as otherwise the action will create one comment per workflow run, instead of updating the first:

      - name: create test-results comment if not exists
        if:
          always() && (github.event_name == 'workflow_dispatch' || steps.changes.outputs.src == 'true')
        env:
          GH_ENTERPRISE_TOKEN: ${{ secrets.GITHUB_TOKEN }}    # choose one
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}               # choose one
          GH_HOST: your-enterprise-github.com                 # leave blank if using github.com
        run: |
          set -o xtrace
          short_ref=$(IFS='/' read -r -a REF <<< "$GITHUB_REF" && echo ${REF[2]})
          test_result_comment=$(gh pr view $short_ref --comments \
            --json comments \
            --jq '.comments[]
              | select(.author.login == "github-actions")
              | select(.body | startswith("## Test Results"))')
          [[ -z "$test_result_comment" ]] && gh pr comment $short_ref --body $'## Test Results\nResults for commit ' || true

result

It comes at the cost of not having the published message as part of the GH Actions run - this would otherwise be randomly assigned.

9 replies

austinpray-mixpanel Apr 19, 2023

We already have GitHub actions workflows that publish checks under a github app. The feedback we have been getting in a large repo is that the UX is bad because it disconnects the job the generates the checks from the checks. Also you have to be super careful multiple github actions (pull_request, push, schedule) writing to the same check suite stomping all over each other.

Conceptually it's a github action that generates checks. Why not be able to colocate the checks with the thing that caused them?

steve-todorov Apr 19, 2023

I completely agree. We noticed it as well, but there is simply no "going around it". 😞

@francisfuzz will get annoyed at some point if we ping him enough times. :)

carlspring Apr 19, 2023

@steve-todorov

@francisfuzz will get annoyed at some point if we ping him enough times. :)

Unless he's muted notifications from this thread... :)

@maxprit,

Could anyone at Github comment on this? It's obviously an issue for quite a few people.

EnricoMi Apr 19, 2023

Maybe raising individual issues at Github Support may get their attention.

jwgmeligmeyling Apr 20, 2023
Author

I am not sure why the sudden influx on this thread. I appreciate y'alls trust in getting this resolved, but after pushing this issue for three years now I can only recommend to put your time elsewhere

EnricoMi · 2023-06-05T09:20:12Z

EnricoMi
Jun 5, 2023

Github support's response is:

Using the Checks API to generate check_runs at runtime from an Actions workflow is not a supported use scenario.

The API is meant to be called from a Github app, but not from Githubs very own Github app called Github Actions.

The proper use in Github Actions is to create your own Github app, install that app in your repos, get an installation token and use that with the Checks API (see https://github.com/orgs/community/discussions/24616#discussioncomment-5607870 above).

Very user-friendly.

1 reply

carlspring Jun 5, 2023

This discussion sounds like a front row seat at a stand-up comedy show! 😀

mutantcornholio · 2023-08-02T14:07:26Z

mutantcornholio
Aug 2, 2023

Just to clear something out: if you have a single workflow that is triggered by serveral different webhook events, you can't have it resulting in a single status on a commit?

An example:

an action is triggered on a pull_request event
then somebody reviews a PR (triggering a pull_request_review event)

So the action can't create a commit status in first case, which would be updated in the second?

1 reply

samrees Oct 3, 2023

I spent all day on this before I found this thread, trying to merge my status between those two events.

This comment was marked as off-topic.

Sign in to view

Specify check suite when creating a checkrun #24616

Replies: 33 comments · 40 replies

samus-aran Aug 7, 2023 Maintainer

jwgmeligmeyling Jun 26, 2020 Author

This comment was marked as spam.

queenofcorgis Aug 14, 2023 Maintainer

jwgmeligmeyling Aug 14, 2023 Author

jwgmeligmeyling Nov 30, 2020 Author

jwgmeligmeyling Jun 7, 2021 Author

jwgmeligmeyling Aug 9, 2022 Author

jwgmeligmeyling Aug 9, 2022 Author

jwgmeligmeyling Oct 24, 2022 Author

jwgmeligmeyling Jan 4, 2023 Author

This comment was marked as off-topic.

This comment was marked as off-topic.

jwgmeligmeyling Apr 20, 2023 Author

Replies: 33 comments 40 replies

samus-aran Aug 7, 2023
Maintainer

jwgmeligmeyling
Jun 26, 2020
Author

queenofcorgis Aug 14, 2023
Maintainer

jwgmeligmeyling Aug 14, 2023
Author

jwgmeligmeyling
Nov 30, 2020
Author

jwgmeligmeyling
Jun 7, 2021
Author

jwgmeligmeyling Aug 9, 2022
Author

jwgmeligmeyling Aug 9, 2022
Author

jwgmeligmeyling Oct 24, 2022
Author

jwgmeligmeyling Jan 4, 2023
Author

jwgmeligmeyling Apr 20, 2023
Author