I’m trying to create CI that does the following:
terraform plan -out=plan.out to generate a Terraform plan.
- After looking at the Terraform plan output in Github actions, I can manually run another job or workflow that calls
terraform apply plan.out with the previously generated plan.
I’ve looked online for some examples of this but all the examples of this I can find just run
terraform apply without actually allowing someone to verify the plan output.
Is this something that’s possible to do in Github Actions? How should I structure the workflows?
I have achieved this by first splitting into two workflows, a CI workflow and a CD workflow. The CI workflow prepares the actual software artefacts to be deployed: a small json manifest of images & tags found in a container registry for deployment into Kubernetes.
The CD workflow uses the Terraform HCL sources straight from the PR/branch being deployed, as opposed to a CI artefact containing Terraform sources. You could use either approach here. I sometimes skip CI stage when it is just CD code changing, and all sources (application and infrastructure) reside in the same repo. Whether CI builds are skipped is controlled by PR labels in my case. We also scan for the last successful CI workflow on the PR when the CI is skipped. The CI build actually adds a label to the PR to kick off the CD workflow as, at least when I originally authored this example workflow,
workflow_call did not correlate subsequent workflows to the triggering PR.
The CD workflow is where I then deploy into all DEV/UAT/PROD stages, all in the same workflow. For this arrangement, we have 3 “environments” set up in Github, which are important when considering deployment jobs, as environments can be used to invoke a manual approval wait step during deployment. We use this approval step to examine and approve the plan.
The broadest strokes of this CD workflow is:
- Check build job to fetch the correct application artefacts
- Plan job split into a matrix strategy based on stages: DEV/UAT/PROD, each job outputs a plan artefact for each stage. DEV is always planned, UAT is only planned if PR is non-draft, PROD is only planned if PR is approved, etc.
- Report job - aggregate and assess the plan outputs and posts a comment on the PR detailing planned changes (65k character limit mind, big plans are abbreviated and must be looked up in the job’s check run output)
- Apply job again split into a matrix strategy of stages, each stage downloads relevant artefact from the corresponding plan job, DEV is applied automatically if the PR is not a draft, UAT and PROD are paused awaiting manual approval courtesy of “environment” based “deployment jobs”, only if there was a plan produced that included changes.
Hope this sheds light on the subject. You have to be quite well versed in how actions workflows behave to have any effective control over using this kind of setup for release management. Distinct lack of dashboards, a bug or two (with how check runs are listed & reported in PRs), plus no real good way surfacing important build/deployment metadata through the checks user interfaces makes it all quite cumbersome to use IMO. It does work tho.
I was able to come up with a solution that worked for us.
We have branches that represent each environment. When a branch is merged to the environment, a workflow runs
terraform plan and saves the plan output to an S3 bucket. This plan is referenced by the commit hash, so it’s impossible to accidentally run a workflow for a commit that doesn’t belong to a particular environment, or that hasn’t been reviewed.
When we’re ready to apply, we have a workflow that runs on
workflow_dispatch which pulls the plan by the commit hash and applies it.