Curated test outcomes and controlling the big red X

We have our Github set up to trigger automated testing using Jenkins and deliver back the test outcome; this works fine.

However, our Github testing is not a simple build, it is a build and test on real hardware with real third-party entities (servers, radio connection, etc.). We are now at the point where the most common cause of failure is the third party, hence the test failures are always curated by someone who knows what they’re looking at to determine whether the cause of failure is “us” or “them”.

My problem is that all one ever sees from a Github perspective in the failure case is the big red X. Is there an easy way of controlling this, making it turn into a nice green tick under moderation? FYI, the test infrastructure is all internal, so clicking on the X would always result in “404 not found” for our public users; we can’t put a “don’t worry” indication behind there.

The red checkmark and green tick refer to the exit code status of whichever script handles the build tests.

Do you mean by this a way to manually alter the status icon on the GitHub WebUI? i.e. overriding the exit code of the script and forcing the icon to be green?

Hi there: yes indeed, a way of manually altering the status icon and forcing it to be green, effectively saying “this failure can be ignored, all is actually good.”

1 Like

That would be useful in many situations (form a service being down, a script needing to be fixed, to many other practical cases). I would also suggest that manually overridden status icons should differ slightly form their auto-generated counterparts — e.g. by the addition of a small ! or * symbol in their right corner — just to ensure they are no misused (e.g. to misrepresent the working status of a project, or faking having fixed a problem at work, etc.).

But I agree 100% with the need of maintainers to be able to override this status checks, especially if their status impacts the repository work flow (e.g. prevent merging, etc.).

I wonder if there’s a temporary workaround to this, while the feature request is being addressed. The only solution that comes to my mind would be to implement some middleware between the actual services and the repository fed check-statuses — it wouldn’t solve the problem of having a WebUI-integrated menu to override the status, but the middleware solution could expose a web interface of its own.