`needs` based on `matrix`

I build my SW using a matrix, let’s say:

    os: [windows, linux]
    configuration: [Release, Debug]
    architecture: [amd64, arm64]

Then, when I want to test the SW I use another matrix:

    os: [windows, linux]
    configuration: [Release, Debug]
    architecture: [amd64, arm64]
    tests: [foo, bar, baz, ....]

All the “tests” are big collections of tests that must be run in parallel (only limited by free runners) since they are time-consuming.

Testing must wait for the corresponding build to be done so I would like to do:

needs: build-${{ matrix.os }}-${{ matrix.configuration }}-${{ matrix.architecture }}

However, this results in an error, all I can do is:

needs: build

This puts in an unnecessary barrier in the pipeline that hits me twice. First, all testing must wait until all building is done. Then, once all building is done all tests are fired at once resulting in unnecessary queues.

Other job properties like runs-on and steps can depend on a matrix, so this seems like an oversight that should be fixed.

  • Are there any known workarounds for this? Matrix inside matrix or code composition would probably work, but both those features are also missing.
  • Is this a feature to be expected soon?
  • Is there any bug report I can subscribe to?

@nedrebo ,

There are few points you need to understand:

  1. The value of the ‘needs’ should be a job ID or an array of job IDs. The string or each element in the array will be directly considered as a job ID, any variables or expressions contained in the string will not be recognized as their corresponding values.

  2. In a workflow, each job has an unique ID that you need to explicitly set it in the workflow file, and the ID can’t be changed dynamically during the workflow run. All the jobs generated from a job matrix share a same job ID.

  3. The usage of the matrix context:

    • You can use the matrix context to set the value of the ‘name’, ‘runs-on’ and ‘with’ keys, or in the step’s if conditional, or in the run step’s command lines.

    • You can’t use the matrix context to set the value if the ‘id’ (job ID, step ID) and ‘needs’ keys, and also can’t be used in the job’s if conditional.

If you really need the feature that generate a job ID for each job in the matrix, and a job can depend (needs) a specified job in the matrix, I recommend that you can directly report a feature request here.
That will allow you to directly interact with the appropriate engineering team, and make it more convenient for the engineering team to collect and categorize your suggestions.

Thx for the detailed answer. I have tried using feature requests, but tbh both the response time and the quality of answers I get at github.community is so much better.

We ended up adding a compiler for YAML files that add things like this that we miss to the YAML automatically. This way we can make extensions to the language while we wait for upstream to catch up.

We also implemented support for Workflow wide fail fast and to have a “finalizer” job that needs all other jobs, without manually maintaining the needs list. Used for deploying. So this seems to work well for us whenever GA features are too limited.

All in all, this compiler reduces our workflow file from 2500 lines to 500 lines so I really hope GA gains more features for code reuse and advanced workflows.


Using matrix to create build/test variations other than OS flavors and tool versions that all need to succeed would be really valuable.

For example, being able to build a suite of services that can be built and tested independently would be very valuable. Without being able to reference the matrix variations in the needs list you need to repeat entire jobs for each service, and the needs list grows with each new service in the suite. It’s not impossible, but for a two-service suite my workflow file is ~450 lines with lots of repetition. If something like the original request were possible, i could make my workflow about half the size, and significantly more readable.

I don’t want to deal with workflows for more than a few services using the current syntax, but the pattern would be extemely valuable for larger service suites.


Things that I’ve tried and didn’t work/weren’t maintainable:

  1. YAML anchors are not allowed, and it feels like an “anti-pattern” to the DevOps culture/engineers, which got used to keeping YAML files DRY with YAML anchors, or with provider-specific workarounds.
  2. GitHub Composite Action - if and sub-actions are not allowed in this context, so that’s definitely not an option.
  3. Makefile / Bash scripts - Adds another layer of complexity to the workflow since the code is nested in a separate file and it’s very hard to read and maintain this type of structure.
  4. Pre-compiling YAML (as the author @nedrebo mentioned) - This means we need to develop and maintain another tool, which doesn’t make sense.

The only thing that “works” is copy-pasting code snippets, which is not an acceptable solution.

Bottom line, I have to pay in time and money for a whole matrix to complete, instead of pinpointing the relevant job that I’d like to proceed with. The expected behavior that @nedrebo described here is exactly what I would expect.

So, has anyone found a better way, other than the ones that I’ve mentioned above, to create a readable, maintainble workflow.yml file?

Note: I’m really frustrated because we moved from drone.io (allows YAML anchors) to GitHub Actions and I’m starting to regret this move. It feels like I’m hacking my way to do my day-to-day job. I just need this missing piece of YAML anchors, and then GitHub Actions would be the ultimate and perfect CI/CD service.