I made a CLI tool for templating GitHub workflows

I was getting tired of duplicating yaml files for workflows and jobs, so I made a CLI tool to streamline the process. From discussions I’ve seen elsewhere on these forums I figured it may be of interest to others.

The repo is here: https://github.com/jbrunton/gflows

It doesn’t do anything you couldn’t do with existing templating tools and a bunch of bash scripting, but the aim was to make it extremely simple to use. It basically integrates a few existing tools:

  • Currently, ytt or Jsonnet as the templating engines.
  • A json schema to validate the generated workflow.
  • A command that validates workflows are up to date with their source templates (run as a CI step).
  • A watch option, so you can get fast feedback when refactoring workflows.

It doesn’t magically make the yaml in .github/workflows disappear, but it does mean you can use these other templating engines to share code, break up files and DRY up your configuration.

Hopefully it will be of some use to others. I’d be keen to hear any feedback, and if anyone’s interested in collaborating (e.g. to add further templating engines) that would be great.

1 Like

Thanks @jbrunton I’ll give that a try!
I was waiting for composite run steps to land to remove duplication in my workflows. Can you comment on how your solution compares to that?

@alpeb: good question! I must admit, although I’d searched to see if there was anything else that may solve my use cases, I didn’t know about that ADR. It’s good to see GitHub are working to provide some built in solutions to this problem.

From my reading of the ADR, I think these are likely some of the major differences.

  1. Composite Run Steps will be supported by GitHub – obviously a big one!

  2. Composite run steps, being steps, can only DRY up specific parts of a job. This means there’s no support for templating other constructs like the on object, or actual jobs. While I expect composite run steps will solve most use cases for many users, I’ve had cases where I needed to parameterize a job. (Example of how I was able to do that with Jsonnet here, and usage here.) And I like being able to factor out the on object for PRs (example using ytt here, usage here).

  3. GFlows is a workflow generator, whereas composite run steps are executed as part of a workflow. The implication here is that if underlying library/action files change, GFlows would require a gflows update + commit to re-generate the workflows, whereas I assume composite run steps would simply use the latest changes. Depending on your use cases there are pros and cons to either approach, but it does mean that one has to be more careful about maintaining a non-breaking API for composite steps (whereas GFlows would have a little more admin overhead to pushing out updates, though they’d be easy to script).

  4. Choice of templating tools. This is a big one for me: I’ve been getting rather tired of yaml for configuration purposes given how easy it is for an erroneous indent to produce subtle changes and errors, or the number of times my IDE messes up a copy and paste because it can’t infer the desired level of indentation. I found refactoring with Jsonnet a breeze partly for that reason. Moreover, Jsonnet and ytt both provide all the language constructs (including variables, functions, iteration) you’d expect for configuration templating, so they’re much more expressive than raw yaml. For simple workflows this obviously unnecessary, but for complex cases they’re very helpful. The architecture of GFLows is also intended to make it easy to add other engines – I have my eye on CUE, for example.

  5. Related to the above: because these tools are designed for configuration management, they include built in language constructs for overlaying customizations on defaults. For example, this is how I can customize the default pull request triggers for a specific workflow using Jsonnet:

    on: workflows.triggers.pull_request_defaults {
      push+: {
        "paths-ignore": ["deployments/**"],
      }
    }
  1. Speed of feedback when refactoring. Because GFlows is a generator, it can diff the changes of the generated output against the actual existing workflow file. In general, this feedback is given in milliseconds (though I did observe occasional latency in the library I was using to implement watch functionality). This is very helpful when breaking up existing workflows, although it obviously doesn’t give you any benefit when creating new code.

I suspect for many users composite run steps would be a sensible way to DRY up workflows significantly, but if you’re writing complex workflows and getting frustrated with the limitations of yaml or the restrictions of composite steps, then GFlows opens up the possibility of some much more sophisticated templating tools – with its own set of trade-offs.

2 Likes