continue-on-error (allow failure), UI indication

Currently, using ‘continue-on-error:’ in a step flags the step with a green check and allows the job to continue.  The only way to see if the step failed is to look at the step’s log.

Other CI systems allow the UI indication of the failure to propagate up to the job level, so it’s easy to see if the there was an ‘allowed failure’.

Since ‘continue-on-error:’ is step scoped, it would be helpful to have something similar to ‘allow-failure’ at the job level, and have the job level UI indication reflect whether there was a failure.

I suspect normal use would use an expression based on matrix values.

A common use of ‘allow-failure’ is to test ‘packages’ against their respective language’s master branch.

54 Likes

Instead of using continue-on-error, you may want to execute subsequent steps even if there was a failure. 

This isn’t on the job level, but will preserve the failure of a particular step but allow subsequent steps to complete.  You can do this with conditionals.

For example:

steps:
- run: false
- run: echo runs on success or failure
  if: succeeded() || failed()
- run: echo runs always, even if the prior step was cancelled
  if: always()
6 Likes

@ethomson 

Thanks for the response.  Apologies, I wasn’t clear.

Here’s a build on Travis:

https://travis-ci.org/puma/puma/builds/607851562

It has 'Allowed Failures jobs shown, and some failed.  The commit that the build is associated with shows a green check.  So, we're one or two UI clicks away from determining whether any 'Allowed Failures jobs actually failed.

I don’t know what the interface is between a CI provider and the UI that displays whether a commit or PR passed the CI, but, at present, determining the status of an 'Allowed Failures` job with Actions is a PITA.

15 Likes

I would like to +1 this; this would be convenient for upgrade paths that should not break the build but give a warning. For example, when you are working towards improving your coding standards or testing against experimental branches of libraries or languages.

16 Likes

I’d also like to point out that this is very needed. :slight_smile:

I’m testing some experimental functionality, and I don’t want it to hold up my build, but I do want it part of my CI, so that I can keep the experiment up to date

5 Likes

If I understand correctly, another use for this would be to flag linting errors.  In general I want all PRs to come with nice clean linting reports, but I don’t want to fail the build and prevent artifacts from uploading just because someone missed a single white space somewhere.

I don’t know, I’m relatively new to workflows so maybe I missed another way to do this.  But it’s important to understand that there is a difference between an error and a warning.  Errors do mean something has broken,  warnings do not necessarily show something has broken, but should be presented somewhere obvious so they can be assessed.

8 Likes

You’d actually see an entry in the “Annotations” snippet on the bottom right of the page, but unfortunately the step is still marked with the green check mark for success, instead of something indicating “allowed failure”. Also, the annotation entry links to .github:1, which is of course unhelpful for finding where the error has occurred.

This is a mandatory feature as, as mentioned, it is essential during upgrade paths.

I use Symfony (PHP) and have many libraries that relies on its components.

Symfony has a release calendar very clear: four minor release and then a breaking change major release.

So, for example, while using components of version 5 I’m sure my library works.

But when version 6 will be released, I will have to make my library compatible with both 5 and 6.

Before version 6 is released, I want to start to upgrade the library.

This requires I have to be able to test against version 5 of components, but also against the future version 6 (master branch).

So, if the tests fail with version 5 there is something wrong that I have to fix: the library is not stable.

But if something fails using the master branch of the components I want the build being green, but be alerted in some way that my library has something not compatible with the coming new version.

The same happens with apps: I want the build has to be pass using stable dependencies, but I want to test it against master branches or very new versions, sonI can gently upgrade it without breaking anything.

@ethomson do you have this in your roadmap for the future? As pointed out, it is a very useful aspect of CI

3 Likes

Hello, in a suite of job variants per matrix variable combinations, this would indeed by nice, to test the tester.

On the one hand, if a workflow fails, all the matrix combinations are blocked. Some options could allow all of the other combinations to keep running.

The continue-on-error option puts a green tick to a step, it could some orange tick instead…

Related non-github-internal project-specific issue: https://github.com/douglaskastle/blender-addon-tester/issues/16#issuecomment-605697324

Thank you!

2 Likes

This is the one feature that is holding me back from switching some of my projects from Travis CI to GitHub Actions.

I often have a case where I run a test suite against my project with:

  - n-1 stable version

  - current stable version

  - latest / master

And for the latest/master, I want it to run but not fail the entire build if the run fails. And in the UI I’d like to see that called out at a glance—overall a green check for the build, but the step that failed should be an orange check or exclamation point, something like that.

Here’s the way it looks on Travis CI, for example: https://travis-ci.org/github/jrgm/allowed_failures/builds/61378523

5 Likes

I think this is blocked on Checks API statuses. Failed statuses would often block PRs which you don’t want and “failed but okay” would need to be green for that not to happen. When other CIs have their own UI, they can allow themselves to use whatever visual markers they want and still report green to GitHub while GitHub doesn’t have such a separate place (except maybe for Checks pages where you can put anything you want really).

This is the issue on GitHub: please, vote also there to point out this is a really mandatory feature.

1 Like