Is it possible for
checkout@v1 to not run on the hash displayed if Github Actions jobs are run in swift succession?
I experienced a situation where a Django project’s unit tests passed unexpectedly, but were resolved using Re-run jobs.
Mock Test Setup
I created a nearly empty django project with a set of 133 mock tests. All tests are patterned like this:
def test_invalid_user_add_fails(self): self.assertTrue(True)
These are named after the tests in the real application I plan to bring into the CI later, and intentionally empty accept for a
I added a job to my GA workflow that runs these tests on commit, and verified that they all ran successfully.
I then changed one of the tests to fail on purpose to observe Github Actions report the test as failed:
def test_invalid_user_add_fails(self): self.assertTrue(False)
However, this time, the tests passed again unexpectedly.
I verified that the hash under test included the commit, and ran the tests locally using the same conditions and verified it was failing.
I added two more failing tests, along with print statements in two of the test methods like this:
def test_invalid_user_add_fails(self): print('This is a failing test') self.assertTrue(False)
These changes caused the test stage to fail, due to all three expected failing test methods.
In looking at the output, I observed the print stdout appeared at the very end.
I have not been able to determine what allowed this failing test to pass. Making almost any further changes have caused it to fail.
Eventually, I re-ran the unexpectedly passing job and it passed.
The only thing I can pinpoint is that just prior to making this commit, I had pushed a change that increased the verbosity of the django test command. I made the test fail, and pushed again.
This was so weird, I almost posted it to the Django community, though I can’t see how this would have been a Django issue. It seemed as though the job was not actually running on the commit displayed in GA.
Can anyone offer feedback on what might have caused this? It was sort of a jolt to my system, as a failing test must always fail individually and trigger the stage to fail to trust the CI.
I recognize that this is not a normal test setup, that nothing is being tested. Yet, that should not matter.
I also recognize this setup should run
checkout@v2, and am happy to update that. But it seems like still, this should never have passed.