Serializing workflow runs in the context of continuous deployment

I’m working on migrating continuous deployment pipelines to from concouse ci to GitHub actions. I’m having an ah-ha moment realizing that there doesn’t appear to be a way to serialize jobs across workflows or even workflows themseves. This is very problematic as two or more merges to master branch can easily interfer with the merge that came just before. An example usecase.

merge master -> test -> publish docker images -> deploy to e2e env -> run e2e tests -> deploy to prod

Since workflows are indpendent and sequential its possible for a second merge to deploy a different set of changes to an e2e testing envrionment while the previous set of tests are being run

In the context of an aws cloudformation deployment the second merges deployment could get rejected while the first merges deployment is in progress.

Have others come up with a solution for this problem? I would love to see queueing as a first class option

6 Likes

For lack of a first class solution I think I’ve been able to find a work around with this GitHub action https://github.com/softprops/turnstyle

2 Likes

In your scenario, merge master -> test -> publish docker images -> deploy to e2e env -> run e2e tests -> deploy to prod

I would suggest you put these 6 jobs in one workflow. Then add needs in each job.

For example:

needs.png

Please refer to this document : https://help.github.com/en/actions/automating-your-workflow-with-github-actions/workflow-syntax-for-github-actions#jobsjob_idneeds 

Thanks @yanjingzhu what I had listed was a simplification what what we’re using. The actual configuration we use looks a lot like the one you have listed.

The feature I’m looking for not to serialize within a workflow run, its to serialize _across_ workflow runs. 

If you want to trigger workflow B when workflow A completes , you need to add a step to create an event with personal access token in workflow A.

For example you want to trigger deploy workflow when a docker image pushed to Github Package Registry (GPR) . 

You need to push image to GPR with your PAT, then use  on: registry_package  in the deploy workflow. 

Another scenario, you could push code in workflow A using PAT then trigger workflow B with on:push  .

https://github.community/t5/GitHub-Actions/Push-from-Action-does-not-trigger-subsequent-action/m-p/37872 

> If you want to trigger workflow B when workflow A completes

The context is here is not workflow B and workflow A. Its two instances of workflow A, in this concrete case pushes to master. In a repository worked on by multiple teams and multiple engineers, pull requests that can trigger workflow A happen independently and without coordination. 

Sorry for misunderstanding your requirement.  What you want is to serialize workflow executions and/or to batch successive changes if workflow is running. 

Currently , there is no way to complete it. I found a similar feature request in our internel channel. I would recommend you to share this idea in the Feedback form for GitHub Actions. It will help to increase the priority of this suggestion. 

Thank you for helping us building a better Github Actions. 

1 Like

We’ve been able to work around the limitation with https://github.com/softprops/turnstyle. It comes at the tradeoff of being billed for the time the second workflow run waits for the first to complete

1 Like

Hello, I try to use this action, https://github.com/marketplace/actions/action-turnstyle to solve the same problem, but running Github Action say [error] not found'  when I use softprops/turnstyle@v1orsoftprops/turnstyle@v0.1.1`. Is it impossible to use yet?

As a workaround, you could consider adding schedule as an event to do something like a daily release. Then within your workflow, have the deploy job run only if: github.event_name == “schedule” .

It changes up your deploy practice quite a bit, but if it can meet your requirements, you can effectively serialize the deployments.

Were you able to figure this out? I use v1 in production https://github.com/softprops/turnstyle/releases/tag/v1

A once a day release schedule is not tenable for our usages but this might work for others We do continuous deployment to gain fast feedback in production. A one day release schedule would be a very slow feedback cycle for our usecases

1 Like

In marketplace, the latest version is v.0.1.1.

So I try to use both v1 and v0.1.1

However, when Github Action is running, it say `[error] not found’ like this.

스크린샷 2020-03-09 오전 1.43.26.png

Could it be a github-token problem in env?

hi, @softprops just found your question and was wondering if the “turnstyle” action is the only workaround at the moment? we want to achive something similar to you, a CI/CD pipeline. at the moment we have to parallel builds in case of two commits/pushed that trigger the workflow twice which results in build failures when provisioning AWS via terraform twice … sucks. cheers marcel

Could you walk me though your setup in a bit more detail.

This tool focuses on serializing workflows. Typically in CI/CD, CI can happen independently but deployments need to happen serially. Workflows by nature are async, when a commit is pushed a new workflow is triggered. The idea behind turnstyle is that you can add a serialization step inside your workflow so that checks to see if there is a previous workflow run in progress, it awaits its completion or optionally continues after waiting a configurable amount of time before progressing.

If you have deployments that are triggered by different workflows (using workflow in the github actions sense) that’s a use case not handled but one I think this could be adapted to handle. If this is the case I’m interested in learning more about your setup. Typically in those cases you don’t want a different workflow but a single workflow with different sets of triggers.