Variable Management in Github

Currently moving from an on-premise pipeline composed of Bitbucket, TeamCity and Octopus Deploy to Github and loving having a single platform where I can define everything as code - so far so good. But to my surprise and coming from Octopus Deploy, variable management is really a big pain point and pretty much a deal breaker for many teams. And I’m a little puzzled as why this is not discussed more in the community (I may have missed it), making me thing we might have missed something.

The main features I’m missing from Octopus Deploy

What we are currently doing in our CICD pipeline in GH is explicitely declaring environment variables for each and every environment (job), using repo and org. secrets for the bits we do not want to expose. This gets even worse when you have adhoc workflows for registrating e.g. idenity metadata for API’s and clients, therefor duplicating the exact same variables into additional workflows.

Questions:

  1. Is this really the only approach at the moment?
  2. Are there any active development (or plans) in terms of an actual variable management framework?
  3. Are there any active discussions regarding this, we could chip into?
  4. Any community contributions/actions trying to solve this issue worth looking into?

Any information is greatly appreciated!

5 Likes

Those features, at least when it comes to managing secrets, are available, but only with GitHub Enterprise Cloud.

@spol-vt could you provide a link with reading material in the docs to that extended variable management capabilities?

@jnus now that composed actions have much more flexibility (announced a few weeks back) - I am wondering if you would maybe want to create a sort of repository of configuration where all the data lives in JS files, and use an action that works based on that. Kind of like in the concept of GitOps. You’ll manage those variables simply by editing those JSON files.

3 Likes

@lirantal Environments - GitHub Docs

Environment protection rules and environment secrets are only available on private repositories with GitHub Enterprise Cloud. For more information about how you can try GitHub Enterprise Cloud for free, see “[Setting up a trial of GitHub Enterprise Cloud]”

You can centrally manage your secrets within an organization, and then make them available to selected repositories. This also means that you can update a secret in one location, and have the change apply to all repository workflows that use the secret.

There is nothing for classic clear txt vars, just secrets, however since it is basically same difference, you can use it for that purpose. With Enterprise Could one can create Org level, Repo level and then Environment level secrets + there is permission management around access to secrets, etc. Functionality is not as extensive as in products mentioned by @jnus, but there is something that goes towards it.

3 Likes

The secrets per environment are nice if you an enterprise plan. The thing I miss the most is that you can’t have variables that are readable… Why only have secrets? :sweat_smile:

1 Like

@spol-vt - So some of the feature set is partly there, in terms of secrets, but not on an organization level though where you can’t scope to environments. We share a ton of variables today via Octopus Deploy, scoped for the various environments, ready to use. This could be resolved in others ways than inheritage (submoduling, npm/nupkg etc), so I’m not seeing this as the biggest challange. Not having a unified and context specific ‘view’ of you variables in you workflow(s), in the really big pain point, where you have to declare all environment specific varibles per environment prior to deploying.

@lirantal Having all variables as secrets will not be easy to maintain, plus I really applaud having everything in vcs.

@ThomasReyskens - It’s a good question - why only have the options to scope secrets per environment and not clear text? My guess is that the GH team also loves having everything in vsc, including configuration and giving the option to save clear text variables on the GH controlplane, violates this principal. A variable framework should definitely based on having the configuration in vsc and having secrets outside, is just a necessary evil to keep sensitive information safe.

1 Like

@lirantal - regarding composed actions, that might be an idea going forward. Depending on what kind of initiative is going on around the community, I might start to build a variable framework based on the ideas of Octopus Deploy - e.g. variables in a json structure, scopes to environment/workflow/jobs/steps, supports variable substitution and context aware depending on the environment you are deploying to, workflow etc. But I really want to make sure this is not on the backlog for GH, as I want to use vanilla GH if possible :expressionless:

2 Likes

Makes sense, yeah :slight_smile:

I ended up building a “pre-processing” tool for GHA workflows tailored to my company. This included variables, an equivalent to YAML anchors and pre-populating jobs with common steps. Downside is we have to remember to re-run this process every time we change the input workflow, but it makes it much simpler to manage.

2 Likes

@dorner interesting way to go about it. If you are allowed to share any details about your current implementation (publicly/privately), that would be greatly appreciated for inspirational purposes.

Still really puzzled to why this is not on the backlog for the GH team, with all the focus Microsoft is giving GitHub, as this is a big big pain point with larger deployment setups.

1 Like

@jnus you can see a cleaned-up version of it here: https://github.com/dorner/gha-config This version probably doesn’t work per se because I scrubbed some of it, but that’s the general approach.

1 Like