Skipped unit tests prevent build from succeeding

As the title says, I have a few skipped unit tests (phpunit), which are fine locally, and in CircleCI, but when attempting to run the same tests in Github Actions, the skipped tests prevent the build from succeeding, and it just hangs and does not continue to the next step in the build. Is there a way I can get Actions to allow skipped tests? Skipped tests are cool in CircleCi, and locally, but apparently Actions does not like them ¯_(ツ)_/¯

This is what I see in the build:

Hi @litzinger,

Glad to see you in Github Community Forum!

It should be due the command doesn’t exit properly. Please append ‘& exit 0’ to your command.

composer run-script test & exit 0

Or you can add continue-on-error: true to the step.

      - name: Run test suite
        continue-on-error: true
        run: |
          composer run-script test

Thanks.

1 Like

Thanks or the suggestions, but neither of those worked. & exit 0 terminated the tests too early (only the first 63 tests executed) and continue-on-error: true didn’t change the behavior either.

Thanks for your reply, @litzinger.

It seems your script returned exit code during the test, which caused it’s executed partically. Please have a check on your code.
If possible please share your repo/sample repo for further investigation.

Thanks

It’s a commercial product/proprietary repository, so I can’t share it. None of the tests have a return or exit statement, they all end with $this->assert...

Hi @litzinger,

Could you please enable debug logging, append ‘$ exit 0’ to your command, provide the error log for investigation?

Thanks

I think it says the operation was canceled b/c I had to manually cancel it. Again, it just hangs after completing the tests.

##[group]Run composer run-script test
2020-06-26T13:20:46.8808242Z e[36;1mcomposer run-script teste[0m
2020-06-26T13:20:46.8849003Z shell: /bin/bash -e {0}
2020-06-26T13:20:46.8849131Z ##[endgroup]
2020-06-26T13:20:46.8891108Z ##[debug]/bin/bash -e /home/runner/work/_temp/d6bd5b9c-f136-4df5-9aee-c673ac2fc1e4.sh
2020-06-26T13:20:47.1462707Z > phpunit
2020-06-26T13:20:47.3518336Z PHPUnit 4.8.36 by Sebastian Bergmann and contributors.
2020-06-26T13:20:47.3518807Z 
2020-06-26T13:20:47.5661672Z ...............................................................  63 / 367 ( 17%)
2020-06-26T13:20:53.6349105Z ..........S.................................................... 126 / 367 ( 34%)
2020-06-26T13:20:55.7890050Z ............................................................... 189 / 367 ( 51%)
2020-06-26T13:21:01.9171896Z ............................................................... 252 / 367 ( 68%)
2020-06-26T13:21:07.0069394Z ....................................................S.S...S.S.. 315 / 367 ( 85%)
2020-06-26T13:21:17.0738086Z .SS....SS...........................................
2020-06-26T13:21:17.0738632Z 
2020-06-26T13:21:17.0739016Z Time: 29.88 seconds, Memory: 16.00MB
2020-06-26T13:21:17.0739277Z 
2020-06-26T13:21:17.0740755Z OK, but incomplete, skipped, or risky tests!
2020-06-26T13:21:17.0741129Z Tests: 367, Assertions: 1950, Skipped: 9.
2020-06-26T14:20:57.9710709Z ##[debug]Re-evaluate condition on job cancellation for step: 'Run test suite'.
2020-06-26T14:21:10.5539896Z ##[error]The operation was canceled.
2020-06-26T14:21:10.5545995Z ##[debug]System.OperationCanceledException: The operation was canceled.
2020-06-26T14:21:10.5546178Z ##[debug]   at System.Threading.CancellationToken.ThrowOperationCanceledException()
2020-06-26T14:21:10.5546425Z ##[debug]   at GitHub.Runner.Sdk.ProcessInvoker.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, Channel`1 redirectStandardIn, Boolean inheritConsoleHandler, Boolean keepStandardInOpen, Boolean highPriorityProcess, CancellationToken cancellationToken)
2020-06-26T14:21:10.5546741Z ##[debug]   at GitHub.Runner.Common.ProcessInvokerWrapper.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, Channel`1 redirectStandardIn, Boolean inheritConsoleHandler, Boolean keepStandardInOpen, Boolean highPriorityProcess, CancellationToken cancellationToken)
2020-06-26T14:21:10.5547041Z ##[debug]   at GitHub.Runner.Worker.Handlers.DefaultStepHost.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, Boolean inheritConsoleHandler, CancellationToken cancellationToken)
2020-06-26T14:21:10.5547243Z ##[debug]   at GitHub.Runner.Worker.Handlers.ScriptHandler.RunAsync(ActionRunStage stage)
2020-06-26T14:21:10.5547406Z ##[debug]   at GitHub.Runner.Worker.ActionRunner.RunAsync()
2020-06-26T14:21:10.5547572Z ##[debug]   at GitHub.Runner.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)
2020-06-26T14:21:10.5549197Z ##[debug]Finishing: Run test suite

Hi @litzinger,

This is a code issue, after the tests complete, it should return an exit code to github runner, otherwise the step will hang.

Thanks.

That doesn’t make sense to me. I have another project running tests the exact same way and it succeeds just fine in Actions. The only difference is that it does not have any skipped tests. It also doesn’t make sense that Github Actions is the only environment in which I run this projects tests that it hangs.

It sounds like the composer executable is not exiting.

Can you try adding something like the below script to confirm:

      - name: Run test suite
        run: |
          composer run-script test
          if [ "$?" = "0" ]; then
            echo Successfully finished
          else
            echo Failed
            exit 1
          fi