Publishing Test Results

Does anyone have a way to do this yet? Or is this planned for the future? I’m thinking of the Tests tab in CICD platforms like AppVeyor or Azure DevOps that display test results (if they’re converted into a specific format like NUnit or similar) so it’s super easy to see which tests failed and click on each individual one for more details. 

35 Likes

Hi,

I have tested several ways to deploy tests:

  1. Using the GitHub Status API to set a status to the Commit / PR

Pro: The little :white_check_mark: or X icon you get for successful or failing tests

Con:

  • The description only has very limited space to display text

  • You can add a link and link somewhere for more information. I have not found a way to link to the result/output of the Action yet

  • If you have a matrix build, you need a status for every matrix build. Or you find a way to merge the test results.

  1. Using the GitHub Comment API to add a comment to the Commit / PR

Pro: You can display as many characters as you want

Con: For matrix builds you have the same problems when using the status. Either you merge the test results, or you have one comment for every matrix build. Or you find and update the comment.  

  1. Using the GitHub Checks API

Pro: You can display the error of the failing test inline inside the file right next to the line where the error happens.

Con:

  • You have to parse the test results

  • I have not found a way to add a test summary only to add annotations to failing tests

  • Annotations can only be displayed on changed files. If you have a file that was not changed but the test is failing for that file you cannot display an annotation there.

  1. Uploading the test results as an artifact

Pro: There’s an action for that and it’s easy to use

Con: You have to manually download the artifact unzip it and check the content.

  1. Just letting the Actions do their job

Pro: You do not have to do anything. You get the check or x status and the “Details” Link links right to the failing action

Con: It’s only an okayish user experience to scan the test results in the console output

  1. Using a marketplace app

Pro: All the features the marketplace app has (maybe it has merging features, possibly it has hosting features)

Con: There’s quite possibly costs involved


I have done the following for our tests:

  1. Run the tests
  2. Add a status with a summary of the test run
  3. Parse the test output and add the annotations
  4. (optional) add a comment
  5. zip the test result
  6. deploy the test result on a static server (behind our firewall)

(we currently do not use matrix builds)

If you want to check the action out. You can find it here

You can find some more of my actions like coverage and lincense-check in the organization. But they are all for a npm / Jest workflow. And they are very specific to our needs. But maybe you find something interesting in there.

Maybe I should write a blog post about my experience with this. 

13 Likes

Yeah, for now I’m just doing your fifth suggestion, @ankri . I’m using Pester for PowerShell tests, and those are all outputted, so its at least possible to see it. 

I may look into utilizing some of the Github APIs later. Thanks for your suggestions. 

I still think this should be an official part of the CICD stuff with GitHub Actions though.

3 Likes

Hey @mgeorgebrown89 and @ankri thanks for taking the time to write this feedback, I’ve taken your suggestion and passed it along to the appropriate team. In the meantime, we’re looking forward to seeing you around!

5 Likes

I’d love to see this.  Right now it is a pain to scroll through the stdout to see which test(s) failed.  I’d like a tests tab that shows all failed tests and requires xunit formatting like typical CI solutions.  So far only gripe, I love this service thus far

9 Likes

I had the same issue where we had to display annotations and failures for unit test.  I used ava testing framework and in order for me to display the errors I had implemented in the following way,

Step 1 => Produced ava test results into TAP format.

Step 2 => Wrote custom parser to parse the ava --tap output to generate the annotations similiar to format which is been accepted by checks api.

Something similar to 

['##[error] 3:1 error "test is not defined" path:test/sum.spec.js name:ReferenceError',
  '##[error] 21:1 error "Duplicate test title: Should pass" path:test.spec.js name:Error',
  '##[error] 4:4 error "Test failed via `t.fail()`" path:test2.spec.js name:AssertionError',
  { total: 8, failure: 5, success: 3 } ]

Annotations output looks like 

Inline errors for the tests which failed will also be shown, 

example:

 

Hope this helps someone for now as a workaround till github supports :smiley:

4 Likes

Was looking for the same thing and came across this: https://github.com/orta/danger-junit

As long as you have your tests in junit xml format tou could use a ruby image to post the results. Not perfect but seems simple enough

2 Likes

Would be very great, at least for the test coverage to be able to publish artifact somewhere, where it is possible to open it as HTML, without downloading zip archive, extract it and view.

Another solution might be, if github pages will provide possibility to view their page only to those who are logged in, means “private github pages” - which isn’t supported yet.

In that case using the GitHub Status API it will be possible to set URL to this private github page or visible artifact (with content-disposition: inline or something similar).

8 Likes

I have similar requirement and more.

i;d like to publish not only the test report, code coverage report, but also:

  • load test reports, the JMeter generated html report
  • AWS ECR docker image scan report in html format
  • more custom metric reports

so i’d like to see a sub-tab on actions tab to show up these kind reports.

something like what gitlab does:

it’s the gap preventing me from migrating all our Jenkins jobs to Github actions

1 Like

Recently this new plugin by @jmisur has been published https://github.com/ScaCap/action-surefire-report

2 Likes

I have a question for the GitHub Actions development team - is there any plan on the roadmap for functionality specifically aimed at recording and viewing test results?

ankri gives an excellent summary of what is currently possible. In terms of making test failures prominent in the UI, annotations is the best thing we currently have, but tbh it feels a bit like we’re abusing a feature that’s designed for something different (i.e. annotating lines changed in a commit… rather than reporting test failures which may or correspond to zero, 1 or many changes in the commit).

Before I go off an implement something that creates annotations for my testing tool’s results (I’m thinking one annotation per test failure, up to the per-step annotation limit of 10) it’d be great to hear if there is anything in the pipeline that’s specifically designed for capturing test results? I’m hoping this will be something that works from workflow command stdout rather than something needing a custom JavaScript action.

Features that would help:

  • a way to report a single string summarizing the output of a job (or step) prominently in the UI, e.g. “5 failures, 100 passes” without the need to dig deeply into the logs
  • a way to add more than 10 annotations from step stdout
  • annotation file links working even on files that haven’t changed
  • a way to add non-warn/failure annotations from workflow stdout
  • a dedicated UI for showing test results (test id, outcome, outcome summary string, link to position in workflow stdout or to artifact with more detail). Needs to work from workflow command stdout. From the CI providers I’ve experimented with TeamCity’s API and UI for this work well (though even something much simpler would still be appreciated :slight_smile: ) https://www.jetbrains.com/help/teamcity/build-script-interaction-with-teamcity.html#BuildScriptInteractionwithTeamCity-ReportingTests

@andreagriffiths11 I’m also curious to know if this is on the roadmap as it’s conspicuously absent.

FWIW, adding test and code coverage results to Azure Pipelines is trivial. For example:

- task: PublishTestResults@2
  inputs:
    testRunner: VSTest
    testResultsFiles: "Artifacts/Tests/**/*.trx"
  displayName: Upload test artifacts

- task: PublishCodeCoverageResults@1
  inputs:
    codeCoverageTool: cobertura
    summaryFileLocation: "Artifacts/Tests/**/coverage.*.xml"
  displayName: Upload coverage artifacts

… integrates “Tests” and “Code Coverage” tabs directly in the pipeline UI. Perfect!

I would like to echo the comments in this thread. This seems to be a gap in current github actions implementation that would be nice to have a solution for. As mentioned Azure DevOps and/or Team City have nice ways of seeing unit test summaries without drilling through pages of logs. +1 from me to add this as a feature to github actions.