Race Condition Possible from Rapidly executed Concurrent Github Actions?

Is it possible for checkout@v1 to not run on the hash displayed if Github Actions jobs are run in swift succession?

I experienced a situation where a Django project’s unit tests passed unexpectedly, but were resolved using Re-run jobs.

Mock Test Setup

I created a nearly empty django project with a set of 133 mock tests. All tests are patterned like this:

def test_invalid_user_add_fails(self):
    self.assertTrue(True)

These are named after the tests in the real application I plan to bring into the CI later, and intentionally empty accept for a True==True assertion.

I added a job to my GA workflow that runs these tests on commit, and verified that they all ran successfully.

I then changed one of the tests to fail on purpose to observe Github Actions report the test as failed:

def test_invalid_user_add_fails(self):
    self.assertTrue(False)

However, this time, the tests passed again unexpectedly.

I verified that the hash under test included the commit, and ran the tests locally using the same conditions and verified it was failing.

Sanity check

I added two more failing tests, along with print statements in two of the test methods like this:

 def test_invalid_user_add_fails(self):
        print('This is a failing test')
        self.assertTrue(False)

These changes caused the test stage to fail, due to all three expected failing test methods.

In looking at the output, I observed the print stdout appeared at the very end.

Why?

I have not been able to determine what allowed this failing test to pass. Making almost any further changes have caused it to fail.

Eventually, I re-ran the unexpectedly passing job and it passed.

The only thing I can pinpoint is that just prior to making this commit, I had pushed a change that increased the verbosity of the django test command. I made the test fail, and pushed again.

This was so weird, I almost posted it to the Django community, though I can’t see how this would have been a Django issue. It seemed as though the job was not actually running on the commit displayed in GA.

Can anyone offer feedback on what might have caused this? It was sort of a jolt to my system, as a failing test must always fail individually and trigger the stage to fail to trust the CI.

I recognize that this is not a normal test setup, that nothing is being tested. Yet, that should not matter.

I also recognize this setup should run checkout@v2, and am happy to update that. But it seems like still, this should never have passed.

In searching across these forums, I was able to to determine that this happened due to the parallel workflows that I had initiated.

There is no official way to prevent workflows from running concurrently and possibly behaving in unexpected ways.

However, there is a great tool called Turnstyle which specifically addresses the situation of parallel workflows.

I’ve now implemented this in the CI workflow and tested it and it works very well.

I am going to submit feedback to GH on this, since this seems like it should be addressed in the docs and have an official solution.

Here are links to other threads discussing this problem:

@banagale,

Thanks for your feedback and sharing your experience.

As far as I know, by default the checkout action fetches only a single commit that triggered the workflow run.

In your case, looks like there indeed have issues on the checkout action. I recommend that you can look for an existing issue or start a new one on the repository of the checkout action (here) to directly report the issues to the appropriate engineering team for further investigation and evaluation. This will allow you to directly interact with the appropriate engineering team.

@brightran

You’re welcome. I appreciate that additional feedback. I found issue #299, (which links to other related), and offered a link back to this thread.