How to share matrix between jobs #26284

marten-seemann · 2020-08-22T05:13:02Z

marten-seemann
Aug 22, 2020

I’m trying to build a regression test suite that basically builds a client and a server binary from different release versions, and then runs them against each other.

builder:
  strategy:
    matrix:
      cfg:
        - { go: 1.13, commit: v1.0.0 }
        - { go: 1.14, commit: v1.2.0 }
  steps:
    // upload binary named "bin-${{ matrix.cfg.go }}-${{ matrix.cfg.commit }}"
runner:
  strategy:
    matrix:
      server:
        - { go: 1.13, commit: v1.0.0 }
        - { go: 1.14, commit: v1.2.0 }
      client:
        - { go: 1.13, commit: v1.0.0 }
        - { go: 1.14, commit: v1.2.0 }
  steps:
     - // download binary
     - // run each server against each client

As you can see, I have repeat the build matrix three times in my configuration, which is error prone when I remove / add versions there.

My first idea was to just use YAML anchors here, but it turns out that GitHub Actions doesn’t support those for whatever reason.

I also tried to define the matrix as an output in the builder job:

builder:
  outputs:
    matrix: ${{ toJson(matrix) }}

And then use it in the runner job:

runner:
  strategy:
    matrix:
      client: ${{ fromJson(needs.builder.outputs.matrix) }}
      server: ${{ fromJson(needs.builder.outputs.matrix) }}

This just results in a “A mapping was not expected” error message.

Any idea how I can solve this?

Answered by marten-seemann

Aug 25, 2020

I think I figured out a way to do this: I place the configuration in a separate file, matrix.json:

[
  { "go": "1.13", "commit": "v1.0.0" },
  { "go": "1.14", "commit": "v1.2.0" },
]

The config uses a separate matrix step to read this JSON file from disk:

matrix:
  runs-on: ubuntu-latest
  outputs:
    matrix: ${{ steps.set-matrix.outputs.matrix }}
  steps:
    - uses: actions/checkout@v2
    - id: set-matrix
      run: |
        TASKS=$(echo $(cat .github/workflows/matrix.json) | sed 's/ //g' )
        echo "::set-output name=matrix::$TASKS"
builder:
  needs: [ matrix ]
  strategy:
    matrix:
      cfg: ${{ fromJson(needs.matrix.outputs.matrix) }}
  steps:
    // upload binary named "b…

View full answer

weide-zhou · 2020-08-24T06:36:36Z

weide-zhou
Aug 24, 2020

Hi @marten-seemann,

Glad to see you in Github Community Forum!

To share matrix between jobs, please follow the usage of doc here , in your code the runner job cannot get the value from ${{ toJson(matrix) }} since format issue.

Please try below code for your case:

  job1:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set-matrix.outputs.matrix }}
    steps:
    - id: set-matrix
      run: |
        echo "::set-output name=matrix::[{\"go\":\"1.13\",\"commit\":\"v1.0.0\"},{\"go\":\"1.14\",\"commit\":\"v1.2.0\"}]"
  builder:
    needs: job1
    runs-on: ubuntu-latest
    strategy:
      matrix: 
        cfg: 
          - ${{fromJson(needs.job1.outputs.matrix)}}
    steps:
    - run: |
        echo bin-${{ matrix.cfg.go }}-${{ matrix.cfg.commit }}
  runner:
    needs: [job1,builder]
    runs-on: ubuntu-latest
    strategy:
      matrix:
        server: 
          - ${{fromJson(needs.job1.outputs.matrix)}}
        client:
          - ${{fromJson(needs.job1.outputs.matrix)}}
    steps:
    - run: |
          echo ${{ matrix.server.go }}-${{ matrix.server.commit }}
          echo ${{ matrix.client.go }}-${{ matrix.client.commit }}

You can refer to my workflow for the details.

Thanks

0 replies

marten-seemann · 2020-08-24T06:53:22Z

marten-seemann
Aug 24, 2020
Author

Hi @weide-zhou, thank you for your reply.

I was hoping to avoid the

echo "::set-output name=matrix::[{\"go\":\"1.13\",\"commit\":\"v1.0.0\"},{\"go\":\"1.14\",\"commit\":\"v1.2.0\"}]"

This code is not very readable, especially when I want to add 10 or so more configurations there.

0 replies

weide-zhou · 2020-08-24T08:44:48Z

weide-zhou
Aug 24, 2020

Hi @marten-seemann,

Since the job outputs must be string format(doc here), i’m afraid you have to pull all configurations in a line.

Thanks

0 replies

marten-seemann · 2020-08-25T09:31:28Z

marten-seemann
Aug 25, 2020
Author

Hi @weide-zhou,

I’m wondering if it’s somehow possible to use the toJson function to feed a human-readable JSON object into the configuration file, something like this:

builder:
  strategy:
    matrix:
      cfg:
        - { go: 1.13, commit: v1.0.0 }
        - { go: 1.14, commit: v1.2.0 }
  outputs:
    matrix: ${{ steps.set-matrix.outputs.matrix }}
  steps:
    - id: set-matrix
      run: echo "::set-output name=matrix::${{ toJson(matrix.cfg) }}"
    // upload binary named "bin-${{ matrix.cfg.go }}-${{ matrix.cfg.commit }}"
runner:
  needs: [ builder ]
  strategy:
    matrix:
      server:
        - ${{ fromJson(needs.builder.outputs.matrix) }}
      client:
        - ${{ fromJson(needs.builder.outputs.matrix) }}
  steps:
     - // download binary
     - // run each server against each client

Unfortunately, this generates an error: Error when evaluating 'strategy' for job 'runner. (Line: xx, Col: xx): Error reading JObject from JsonReader. Path '', line 1, position 1.,(Line: xx, Col: xx): Error reading JObject from JsonReader. Path '', line 1, position 1.

And it looks like there’s no way to inspect the matrix output that was passed between the two jobs, so I’m having trouble debugging this error message.

0 replies

weide-zhou · 2020-08-25T09:52:49Z

weide-zhou
Aug 25, 2020

Hi @marten-seemann,

You cannot use toJson function transfer the job outputs, because:

format is not correct, set-output doesn’t accept format below, and miss \ to avoid escape characters.
The output is determined by the last check run if it’s a matrix job, cannot get accumulated value. Please check ticket below:
[BUG] Jobs output should return a list for a matrix job

Hence, please put all configuration in a line with \.

Thanks

0 replies

marten-seemann · 2020-08-25T10:12:58Z

marten-seemann
Aug 25, 2020
Author

Does that mean it’s not possible to have matrix jobs generated by a script? That seems like… a pretty basic thing for a CI, and I’d surprised to hear that it’s not possible on GitHub Actions.

I was thinking of putting the jobs in a separate file (as JSON), and read that file from disk. But that escaping requirement sounds like I can’t do that either (at least not without a script that would add the \s).

0 replies

marten-seemann · 2020-08-25T14:18:12Z

marten-seemann
Aug 25, 2020
Author

I think I figured out a way to do this: I place the configuration in a separate file, matrix.json:

[
  { "go": "1.13", "commit": "v1.0.0" },
  { "go": "1.14", "commit": "v1.2.0" },
]

The config uses a separate matrix step to read this JSON file from disk:

matrix:
  runs-on: ubuntu-latest
  outputs:
    matrix: ${{ steps.set-matrix.outputs.matrix }}
  steps:
    - uses: actions/checkout@v2
    - id: set-matrix
      run: |
        TASKS=$(echo $(cat .github/workflows/matrix.json) | sed 's/ //g' )
        echo "::set-output name=matrix::$TASKS"
builder:
  needs: [ matrix ]
  strategy:
    matrix:
      cfg: ${{ fromJson(needs.matrix.outputs.matrix) }}
  steps:
    // upload binary named "bin-${{ matrix.cfg.go }}-${{ matrix.cfg.commit }}"
runner:
  needs: [ matrix, builder ]
  strategy:
    matrix:
      server: ${{ fromJson(needs.matrix.outputs.matrix) }}
      client: ${{ fromJson(needs.matrix.outputs.matrix) }}
  steps:
     - // download binary
     - // run each server against each client

The echo $(cat ...) remove all line breaks from the JSON file.
For some reason that I don’t fully understand it also seems necessary to remove all (?) spaces from the JSON string (sed 's/ //g').

Note that this solution also works if the matrix is dynamically generated. For example, it would be trivial now to write a script that outputs JSON, which is then used as a matrix in a later step.

0 replies

sigprof · 2022-06-25T13:21:16Z

sigprof
Jun 25, 2022

Actually this can be made to work, you just need a couple of minor adjustments to the code:

The matrix of your builder job needs to have only a single entry which contains the JS object data (represented in the YAML syntax inside the workflow). So you need to add an extra array around your data (or an object, which might be easier to represent in YAML in your case, because otherwise you would have an array of arrays there).
The set-output command supports only a single line of text, so you need to use a tool like jq to format the JSON data as a single line.
The only working way to get multiline input values (like the toJSON() output) into run is to pass them via environment variables.

builder:
  runs-on: ubuntu-latest
  strategy:
    matrix:
      cfg:
        - data:
          - { go: 1.13, commit: v1.0.0 }
          - { go: 1.14, commit: v1.2.0 }
  outputs:
    matrix: ${{ steps.set-matrix.outputs.matrix }}
  steps:
    - id: set-matrix
      env:
        CFG: ${{ toJSON(matrix.cfg) }}
      run: jq -r -n 'env.CFG | fromjson | @json "::set-output name=result::\(.data)"'
runner:

needs: [ builder ]

strategy:

matrix:

server: ${{ fromJson(needs.builder.outputs.matrix) }}

client: ${{ fromJson(needs.builder.outputs.matrix) }}

steps:

- // download binary

- // run each server against each client

At the moment such abuse of the matrix feature seems to be the only way to pass JSON data to actions while having it represented in the YAML form in the workflow file. (Of course you can wrap the data in a multiline string and pass it as an environment variable or an action input, but then you will lose any YAML syntax highlighting or other help from your editor, because it would treat that piece of data as a plain string.)

0 replies

SergiiShapoval · 2022-12-09T15:37:38Z

SergiiShapoval
Dec 9, 2022

https://vs-blogs.medium.com/dynamic-matrix-in-github-action-e516019cfdf5 (Approach 3) helped me

0 replies

cmoulliard · 2023-08-11T13:41:14Z

cmoulliard
Aug 11, 2023

It took me a while to figure the best approach and here is what I propose to todo to reuse a workflow where the configuration (YAML -> JSON) is parsed and saved as "matrix" key

Create first a reusable workflow exporting the output matrix
https://github.com/ch007m/test-github-action/blob/main/.github/workflows/reusable-config.yml

on:
  workflow_call:
    # Map the workflow outputs to job outputs
    outputs:
      matrix:
        description: "Matrix of configurations"
        value: ${{ jobs.generate-json-matrix.outputs.matrix }}
...jobs:
  generate-json-matrix:
    name: Generate output
    runs-on: ubuntu-latest
    # Map the job outputs to step outputs
    outputs:
      matrix: ${{ steps.convert.outputs.matrix }}
    steps:
      - name: Checkout to repository
        uses: actions/checkout@v3

      - id: convert
        name: Convert config from YAML to JSON and pass it as value to matrix
        run: |
          echo "matrix=$(yq -P -o=json config.yaml | jq -c .)" >> $GITHUB_OUTPUT

From a workflow use it

https://github.com/ch007m/test-github-action/blob/main/.github/workflows/consume-config-matrix.yml

name: Call reusable workflow and use its outputs
on:
  workflow_dispatch:

jobs:
  job1:
    uses: ./.github/workflows/reusable-config.yml
  job2:
    runs-on: ubuntu-latest
    needs: job1
    strategy:
      matrix: ${{ fromJSON(needs.job1.outputs.matrix) }}
...
      - name: Log Java version
        if: always()
        run: |
          echo "Java version: ${{ matrix.java }}"

Result in action: https://github.com/ch007m/test-github-action/actions/runs/5833089159

Remark: There is one caveat as we cannot copy/paste the fields defined under the steps/jobs as yaml config file should include the matrix exploded

Wrong (see array declaration)

include:
- java: ["11", "17"] 
  builder-image= "quay.io/snowdrop/my_image:1.0"

Good (array removed)

- java: "11"  
  builder-image= "quay.io/snowdrop/my_image:1.0"
- java: "17" 
  builder-image= "quay.io/snowdrop/my_image:1.0"

Feel free to use this trick and to share it with others :-)

0 replies

tfoote · 2024-01-02T21:37:47Z

tfoote
Jan 2, 2024

Yaml anchors would be a standard way to do this, but they're unfortunately not supported in github's execution.

See thread https://github.com/orgs/community/discussions/4501 and issue actions/runner#1182

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GitHub Community

How to share matrix between jobs #26284

{{title}}

Replies: 11 comments

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

How to share matrix between jobs #26284

Replies: 11 comments

marten-seemann Aug 24, 2020 Author

marten-seemann Aug 25, 2020 Author

marten-seemann Aug 25, 2020 Author

marten-seemann Aug 25, 2020 Author

marten-seemann
Aug 24, 2020
Author

marten-seemann
Aug 25, 2020
Author

marten-seemann
Aug 25, 2020
Author

marten-seemann
Aug 25, 2020
Author