"break glass" option for deploying via github actions

We are using a GitHub Actions workflow for our Continuous Integration (CI) pipeline and we’re about to make it a Continuous Delivery pipeline by automating deployment to our production environment via that same workflow. We’re pretty excited about it.

However we still have a slight sense of trepidation about automating deployments to live production environments so we’re looking for a way to quickly switch off deployments without requiring a commit should the need arise. In my head a way to do this would be to have the notion of an environment variable that is defined in the projects settings, probably at https://github.com/ourorg/ourrepo/settings/actions, which we can change as desired. Our workflow can be designed to only deploy to production if that env var is set to a certain value.

I’ve used GitLab in the past which had this notion of environment variables that could be set outside of the Workflow YAML file and hence why I think this could be a workable solution, but GitHub Actions has no such concept of env vars set outside of the Workflow.

I’m looking for ideas of how we could achieve similar. I basically want a way of quickly and easily shutting off all deployments to production, I’m using the “break glass” metaphor to describe this. Any ideas?

Hi @jamiet-msm – GitHub Actions has Environments that may have associated Secrets, and branch protection rules which could be changed to block all deploys to an environment. Are you able to speak to why this feature isn’t suitable for your use case?

1 Like

Hi,
Well firstly this isn’t a secret. I’m familiar with secrets set at the repo level and I know that the value stored therein isn’t visible to anyone after its being set. I’d like the value stored in there to be well-known. Update, have just taken a look at environments and realised the same is true there, secrets are not visible. This makes sense of course, they’re secrets after all, but the value I want to store is not a secret.

I have read a little bit about environments and read that they can be used to “block all deploys”, but I’m not clear about how that would be accomplished. In our case “a deployment” means running some bespoke scripts that we have written ourselves so GitHub Actions doesn’t know that its doing a deployment, it just knows that it is running some scripts. Hence, how could an environment be used to “block a deployment”?

1 Like

GitLab’s “Environment Variables” and GitHub’s “Secrets” are functionally equivalent, however, GitLab’s “Environment Variables” are optionally “masked” whereas GitHub’s “Secrets” are always masked.

An Environment on GitHub is a collection of Secrets that are available if a set of Environment protection rules pass. You can “break glass” by changing the protection rules to deny permission to access the Production secrets.

Are you dynamically determining which environment to deploy to based on branch name, some other logic or do you have dedicated workflows for production deploys that you can hardcode the environment name into?

1 Like

OK that’s fair, and useful. Thanks. I personally think its rather handy to have masking be optional. I want to see the value that’s in there however I accept that GitLab’s env vars and GitHub’s secrets are functionally equivalent.

I think “some other logic” is probably the closest description. We essentially have a python script which does our deployment for us (we do it this way so that we can run the deployment from both the CI pipeline and from our local machines). The python script takes an argument that tells it where to deploy to. Hence in our workflow YAML we’ve got something akin to:

      - name: deploy-to-test
        run: |
          python deploy.py TEST

then later in the workflow we’ll have a job to deploy to production

      - name: deploy-to-prod
        run: |
          python deploy.py PRODUCTION

(its actually a bit more complicated than that but for the purposes of explaining, that’s essentially what it does.)

So, we have a single workflow that:

  • deploys to TEST environment
  • runs some tests
  • deploy to PROD (if the tests are successful)

As I think you’ve gathered, I want to be able to easily toggle that last job on or off should we need to.

GitHub Actions is flexible enough to support a variety of approaches, so your current approach will work, but you will be able to better integrate the full range of Actions features (including Environments) if you integrate using the as-documented approach, where Jobs are used for distinct units of work.

The following example assumes a push to the branch called main is a deploy to the production environment, and a push to any other branch is a deploy to the staging environment.

on:
  push:
    branches:
      - "**"
      
jobs:
  test:
    runs-on: ubuntu-latest
    environment: testing
    steps:
    - run: python deploy.py TEST --some-secret=${{ secrets.MY_SECRET_KEY }}
  environment: # "environment" here is an arbitrary name for a job that uses an action to calculate the environment name for your deployment from the current branch name
    runs-on: ubuntu-latest
    outputs:
      name: "${{ steps.environment.outputs.value }}"
    steps:
    - uses: haya14busa/action-cond@v1
      id: environment
      with:
        cond: ${{ github.ref == 'refs/heads/main' }}
        if_true: "production"
        if_false: "staging"
  deploy:
    runs-on: ubuntu-latest
    needs: [test, environment]
    environment:
      name: ${{ needs.environment.outputs.name }}
    steps:
    - run: python deploy.py ${{ needs.environment.outputs.name }} --some-secret=${{ secrets.MY_SECRET_KEY }}

You’re now deferring all responsibility for environments to the Environments feature. If you need to “break glass” you can temporarily enable one of the various Environment protection rules, like required approval:

Rather than their Run fail, the developer will see that their deployment is “waiting” pending approval. As a designated approver, once you want to un-break glass, you can disable the approval requirement and then either reject the pending deployments or approve them.

Let me know if there’s any aspects of your requirements that I’ve missed, or if any aspect of the example needs further explanation.