`needs` based on `matrix`

I build my SW using a matrix, let’s say:

matrix:
    os: [windows, linux]
    configuration: [Release, Debug]
    architecture: [amd64, arm64]

Then, when I want to test the SW I use another matrix:

matrix:
    os: [windows, linux]
    configuration: [Release, Debug]
    architecture: [amd64, arm64]
    tests: [foo, bar, baz, ....]

All the “tests” are big collections of tests that must be run in parallel (only limited by free runners) since they are time-consuming.

Testing must wait for the corresponding build to be done so I would like to do:

needs: build-${{ matrix.os }}-${{ matrix.configuration }}-${{ matrix.architecture }}

However, this results in an error, all I can do is:

needs: build

This puts in an unnecessary barrier in the pipeline that hits me twice. First, all testing must wait until all building is done. Then, once all building is done all tests are fired at once resulting in unnecessary queues.

Other job properties like runs-on and steps can depend on a matrix, so this seems like an oversight that should be fixed.

  • Are there any known workarounds for this? Matrix inside matrix or code composition would probably work, but both those features are also missing.
  • Is this a feature to be expected soon?
  • Is there any bug report I can subscribe to?
1 Like

@nedrebo ,

There are few points you need to understand:

  1. The value of the ‘needs’ should be a job ID or an array of job IDs. The string or each element in the array will be directly considered as a job ID, any variables or expressions contained in the string will not be recognized as their corresponding values.

  2. In a workflow, each job has an unique ID that you need to explicitly set it in the workflow file, and the ID can’t be changed dynamically during the workflow run. All the jobs generated from a job matrix share a same job ID.

  3. The usage of the matrix context:

    • You can use the matrix context to set the value of the ‘name’, ‘runs-on’ and ‘with’ keys, or in the step’s if conditional, or in the run step’s command lines.

    • You can’t use the matrix context to set the value if the ‘id’ (job ID, step ID) and ‘needs’ keys, and also can’t be used in the job’s if conditional.

If you really need the feature that generate a job ID for each job in the matrix, and a job can depend (needs) a specified job in the matrix, I recommend that you can directly report a feature request here.
That will allow you to directly interact with the appropriate engineering team, and make it more convenient for the engineering team to collect and categorize your suggestions.

Thx for the detailed answer. I have tried using feature requests, but tbh both the response time and the quality of answers I get at github.community is so much better.

We ended up adding a compiler for YAML files that add things like this that we miss to the YAML automatically. This way we can make extensions to the language while we wait for upstream to catch up.

We also implemented support for Workflow wide fail fast and to have a “finalizer” job that needs all other jobs, without manually maintaining the needs list. Used for deploying. So this seems to work well for us whenever GA features are too limited.

All in all, this compiler reduces our workflow file from 2500 lines to 500 lines so I really hope GA gains more features for code reuse and advanced workflows.