Help
cancel
Showing results for 
Search instead for 
Did you mean: 
Pilot Lvl 1
Message 1 of 10

continue-on-error (allow failure), UI indication

Currently, using 'continue-on-error:' in a step flags the step with a green check and allows the job to continue.  The only way to see if the step failed is to look at the step's log.

 

Other CI systems allow the UI indication of the failure to propagate up to the job level, so it's easy to see if the there was an 'allowed failure'.

 

Since 'continue-on-error:' is step scoped, it would be helpful to have something similar to 'allow-failure' at the job level, and have the job level UI indication reflect whether there was a failure.

 

I suspect normal use would use an expression based on matrix values.

 

A common use of 'allow-failure' is to test 'packages' against their respective language's master branch.

9 Replies
Highlighted
GitHub Staff
Message 2 of 10

Re: continue-on-error (allow failure), UI indication

Instead of using `continue-on-error`, you may want to execute subsequent steps even if there was a failure. 

 

This isn't on the job level, but will preserve the failure of a particular step but allow subsequent steps to complete.  You can do this with conditionals.

 

For example:

 

steps:
- run: false
- run: echo runs on success or failure
  if: succeeded() || failed()
- run: echo runs always, even if the prior step was cancelled
  if: always()
Highlighted
Pilot Lvl 1
Message 3 of 10

Re: continue-on-error (allow failure), UI indication

@ethomson 

 

Thanks for the response.  Apologies, I wasn't clear.

 

Here's a build on Travis:

https://travis-ci.org/puma/puma/builds/607851562

 

It has 'Allowed Failures` jobs shown, and some failed.  The commit that the build is associated with shows a green check.  So, we're one or two UI clicks away from determining whether any 'Allowed Failures` jobs actually failed.

 

I don't know what the interface is between a CI provider and the UI that displays whether a commit or PR passed the CI, but, at present, determining the status of an 'Allowed Failures` job with Actions is a PITA.

Highlighted
Copilot Lvl 2
Message 4 of 10

Re: continue-on-error (allow failure), UI indication

@ethomson do you have this in your roadmap for the future? As pointed out, it is a very useful aspect of CI

Highlighted
Copilot Lvl 2
Message 5 of 10

Re: continue-on-error (allow failure), UI indication

I would like to +1 this; this would be convenient for upgrade paths that should not break the build but give a warning. For example, when you are working towards improving your coding standards or testing against experimental branches of libraries or languages.

Highlighted
Copilot Lvl 3
Message 6 of 10

Re: continue-on-error (allow failure), UI indication

I'd also like to point out that this is very needed. :)

I'm testing some experimental functionality, and I don't want it to hold up my build, but I do want it part of my CI, so that I can keep the experiment up to date

 

Demonstration of failure, but step is green, due to continue-on-errorDemonstration of failure, but step is green, due to continue-on-error

Highlighted
Ground Controller Lvl 1
Message 7 of 10

Re: continue-on-error (allow failure), UI indication

If I understand correctly, another use for this would be to flag linting errors.  In general I want all PRs to come with nice clean linting reports, but I don't want to fail the build and prevent artifacts from uploading just because someone missed a single white space somewhere.

 

I don't know, I'm relatively new to workflows so maybe I missed another way to do this.  But it's important to understand that there is a difference between an error and a warning.  Errors do mean something has broken,  warnings do not necessarily show something has broken, but should be presented somewhere obvious so they can be assessed.

Highlighted
Pilot Lvl 1
Message 8 of 10

Re: continue-on-error (allow failure), UI indication

You'd actually see an entry in the "Annotations" snippet on the bottom right of the page, but unfortunately the step is still marked with the green check mark for success, instead of something indicating "allowed failure". Also, the annotation entry links to .github:1, which is of course unhelpful for finding where the error has occurred.

Highlighted
Copilot Lvl 2
Message 9 of 10

Re: continue-on-error (allow failure), UI indication

This is a mandatory feature as, as mentioned, it is essential during upgrade paths.

I use Symfony (PHP) and have many libraries that relies on its components.

Symfony has a release calendar very clear: four minor release and then a breaking change major release.

So, for example, while using components of version 5 I’m sure my library works.

But when version 6 will be released, I will have to make my library compatible with both 5 and 6.

Before version 6 is released, I want to start to upgrade the library.

This requires I have to be able to test against version 5 of components, but also against the future version 6 (master branch).

So, if the tests fail with version 5 there is something wrong that I have to fix: the library is not stable.

But if something fails using the master branch of the components I want the build being green, but be alerted in some way that my library has something not compatible with the coming new version.

The same happens with apps: I want the build has to be pass using stable dependencies, but I want to test it against master branches or very new versions, sonI can gently upgrade it without breaking anything.
Highlighted
Ground Controller Lvl 1
Message 10 of 10

Re: continue-on-error (allow failure), UI indication

 

Hello, in a suite of job variants per matrix variable combinations, this would indeed by nice, to test the tester.

On the one hand, if a workflow fails, all the matrix combinations are blocked. Some options could allow all of the other combinations to keep running.

The continue-on-error option puts a green tick to a step, it could some orange tick instead...

Related non-github-internal project-specific issue: https://github.com/douglaskastle/blender-addon-tester/issues/16#issuecomment-605697324

Thank you!