Feature request and use case example to allow `matrix` in `if`s

Currently, using matrix at jobs.<job>.if is an error.

We’re building an E2E test suite at https://github.com/timberio/vector, and want to use matrices and labels to conditionally skip some of the test cases.

Here’s an example of what we’d like to have:

name: E2E Suite

on:
  push: {}
  pull_request: {}
  schedule:
    - cron: "0 4 * * *"

jobs:
  test-e2e-kubernetes:
    name: K8s ${{ matrix.kubernetes_version.version }} / ${{ matrix.container_runtime.name }}
    runs-on: ubuntu-latest
    strategy:
      matrix:
        kubernetes_version:
          - version: "v1.18.6"
            is_essential: true
          - version: "v1.17.9"
        container_runtime:
          - name: docker
            is_essential: true
          - name: containerd
    if: |
          (
            matrix.kubernetes_version.is_essential == true &&
            matrix.container_runtime.is_essential == true
          ) || (
            github.event.pull_request == null ||
            contains(github.event.pull_request.labels.*.name, 'ci-condition: k8s e2e all targets')
          )
    steps: ...

This is a simplified example, to demonstrate the point.

The idea is we’d like to have a way to only run the full test suite in a nightly build, or if the PR has a certain label (ci-condition: k8s e2e all targets in our case). We can do that, but we also want our job definition to be compact - we don’t want to duplicate it.

The listing above doesn’t work, because currently we can’t use matrix.kubernetes_version.is_essential and matrix.container_runtime.is_essential in the if: statement.

We’d like to have the access to matrix at the if level.

We’ve also considered using matrix.exclude, however that doesn’t allow us to dynamically condition those exclusions based on contains(github.event.issue.labels.*.name, 'ci-condition: k8s e2e all targets') || github.event_name == 'schedule' condition.

UPD 1:

Another notable issue we encountered as a consequence of the inability to use matrix at the job-level if: clause is the inability to alter the job execution result beyond success / failure. We attempted to move the logic into the step-level if: statement, however doing that required us to have a third outcome state for our test - something like skipped or cancelled - to indicate that the test didn’t show neither positive nor negative outcome. How can there be one if the test itself was effectively skipped? I remember there was a way to switch the job into a cancelled state, however, it seems it’s not available anymore.

Hi @mozgiii,

Glad to see you in Github Community Forum!

Currently matrix is NOT supported in job if expression, it has been already raised as an feature_request, please check similar ticket for the details.

Actually i’m a little confused with your if expression, since you use || there, there could be multiple situations with different events. It’s recommended to move push event to a new yaml.
Could you please let me know for pull_request(or schedule) event, why to evaluate matrix value?

It’s not supported to change the step outcome, but cancel the whole job, you can try to use andymckay/cancel-action@0.2

Thanks.

Hi @weide-zhou,

There was a bug in the if condition, the correct version would be

if: |
  (
    matrix.kubernetes_version.is_essential &&
    matrix.container_runtime.is_essential
  ) || (
    github.event.pull_request == null ||
    contains(github.event.pull_request.labels.*.name, 'ci-condition: k8s e2e all targets')
  )

Notice there’s a different condition at the second part of the ||.

I think this is a good way to explain the logic we’re trying to achieve:

  • if we’re executing at a PR with no labels - run only essential tests
  • if we’re executing at a PR with ci-condition: k8s e2e all targets label set - run all tests
  • if we’re executing at a non-PR (push to master or scheduled run on master) - run all tests

In that sense - we first check if the test is essential - and if it is we always run the said test - otherwise check the context we’re executing at, and sometimes run the test even if it’s not essential.

github.event.pull_request == null tells us if we’re not in a PR context - then we always want to run all tests
contains(github.event.pull_request.labels.*.name, 'ci-condition: k8s e2e all targets') - if we’re at a PR - only run if there’s a certain label

I’ve looked into andymckay/cancel-action@0.2 - however it cancels the whole workflow. We’d need to cancel just one particular job from the workflow.

Hi @mozgiii,

Thanks for your reply!

Github support dynamic matrix value now. Based on your logic, you can add a prejob and set the matrix value accordingly(events&pr labels) , transfer the matrix value to your test job. In this case, you don’t need to cancel the redundant jobs.
Code sample as below(please change the matrix value for yours):

on:
  push: 
  pull_request: 
  schedule:
    - cron: "*/6 * * * *"

jobs:
  prejob:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set-matrix1.outputs.matrix }}${{ steps.set-matrix2.outputs.matrix }}  #get the matrix value and transfer to next job
    steps:
      - name: set matrix 1   # set the matrix value for push, schedule event and pr event with expected label
        id: set-matrix1
        if: (github.event_name=='push') || (github.event_name=='schedule') || (github.event_name=='pull_request' && contains(github.event.pull_request.labels.*.name, 'ci-condition:k8s e2e all targets'))
        run: |
          echo "::set-output name=matrix::{\"include\":[{\"project\":\"foo\",\"config\":\"Debug\"},{\"project\":\"bar\",\"config\":\"Release\"}]}"
      - name: set matrix 2  # set the matrix value for pull_request event without expetected label
        id: set-matrix2
        if: github.event_name=='pull_request' && !contains(github.event.pull_request.labels.*.name, 'ci-condition:k8s e2e all targets')
        run: |
          echo "::set-output name=matrix::{\"include\":[{\"test\":\"value1\",\"config\":\"Debug1\"},{\"test\":\"value2\",\"config\":\"Release1\"}]}"
            

  test-e2e-kubernetes:
    needs: [prejob]
    runs-on: ubuntu-latest
    strategy:
      matrix: ${{fromJson(needs.prejob.outputs.matrix)}}   # get the matrix value

    steps: 
      - name: Dump matrix context
        env:
          MATRIX_CONTEXT: ${{ toJson(matrix) }}
        run: echo "$MATRIX_CONTEXT"

Confirmed it works on my side. For exmaple:
push event workflow: https://github.com/weide-zhou/ticket13/actions/runs/197233132
pull request event without expected labels: https://github.com/weide-zhou/ticket13/actions/runs/197237243
pull request with expected label: https://github.com/weide-zhou/ticket13/actions/runs/197234906

Please refer to the doc for more details.

Thanks

1 Like

Wow, amazing! A very nice solution! I didn’t have a clue to look into that direction. Github definitely should explicitly add a hint to the matrix doc that matrix can be assigned that way.

Thanks a lot, I’ll try this!

UPD: Works like a charm, thanks again!