run a job after all jobs (including failing ones)

I am adding github actions to minikube project, and I would like to do the following

  • have 4 jobs integraiton tests to run in parallel each one will upload their logs to a file

  • have one job (named upload_all_reports) that needs other 4 to finish then downloads all 4 reports puts them in one folder and then uplods one file that has all the reports.

I have manged to make it happen using job needs ! but if any of the 4 parallle jobs before the last jobs fail, it will not run the last job !

I would like the Jobs to show as Failed if they fail, but i still wanna collect their downloaded file ! 

is this possible in github actions ?

here is my yaml.

name: CI
on: [pull_request]
jobs:
  # Runs before all other jobs
  # builds the minikube binaries
  build_minikube:
    env:
      TIME_ELAPSED: time
      JOB_NAME: "Docker_Ubuntu_16_04"
      GOPOGH_RESULT: ""
    runs-on: ubuntu-18.04
    steps:
    - uses: actions/checkout@v2
    - name: build binaries
      run : |
        make minikube-linux-amd64
        make e2e-linux-amd64
        cp -r test/integration/testdata ./out
        whoami
        echo github ref $GITHUB_REF
        echo workflow $GITHUB_WORKFLOW
        echo home $HOME
        echo event name $GITHUB_EVENT_NAME
        echo workspace $GITHUB_WORKSPACE
        echo "end of debug stuff"
        echo $(which jq)
    - uses: actions/upload-artifact@v1
      with:
        name: minikube_binaries
        path: out
  unit_test:
    env:
      TIME_ELAPSED: time
      JOB_NAME: "unit_test"
      GOPOGH_RESULT: ""
    runs-on: ubuntu-18.04
    steps:
    - uses: actions/checkout@v2
    - name: install libvirt
      run : |
        sudo apt-get update
        sudo apt-get install -y libvirt-dev
    - name: lint
      env:
        TESTSUITE: lintall
      run : make test
      continue-on-error: false
    - name: unit test
      env:
        TESTSUITE: unittest
      run :
        make test
      continue-on-error: false
  # Run the following integration tests after the build_minikube
  # They will run in parallel and use the binaries in previous step
  docker_ubuntu_16_04:
    needs: [build_minikube]
    env:
      TIME_ELAPSED: time
      JOB_NAME: "Docker_Ubuntu_16_04"
      GOPOGH_RESULT: ""
    runs-on: ubuntu-16.04
    steps:
    - name: Install gopogh
      shell: bash
      run: |
        curl -LO https://github.com/medyagh/gopogh/releases/download/v0.1.16/gopogh-linux-amd64
        sudo install gopogh-linux-amd64 /usr/local/bin/gopogh
    - name: Download binaries
      uses: actions/download-artifact@v1
      with:
        name: minikube_binaries
    - name: Run integration test
      continue-on-error: true
      # bash {0} to allow test to continue to next step. in case of
      shell: bash {0}
      run: |
        cd minikube_binaries
        mkdir -p report
        mkdir -p testhome
        chmod a+x e2e-*
        chmod a+x minikube-*
        START_TIME=$(date -u +%s)
        KUBECONFIG=$(pwd)/testhome/kubeconfig MINIKUBE_HOME=$(pwd)/testhome ./e2e-linux-amd64 -minikube-start-args=--vm-driver=docker -expected-default-driver= -test.timeout=70m -test.v -binary=./minikube-linux-amd64 2>&1 | tee ./report/testout.txt
        END_TIME=$(date -u +%s)
        TIME_ELAPSED=$(($END_TIME-$START_TIME))
        min=$((${TIME_ELAPSED}/60))
        sec=$((${TIME_ELAPSED}%60))
        TIME_ELAPSED="${min} min $sec seconds "
        echo ::set-env name=TIME_ELAPSED::${TIME_ELAPSED}
    - name: Generate html report
      shell: bash
      run: |
        cd minikube_binaries
        export PATH=${PATH}:`go env GOPATH`/bin
        go tool test2json -t < ./report/testout.txt > ./report/testout.json || true
        STAT=$(gopogh -in ./report/testout.json -out ./report/testout.html -name "${JOB_NAME} ${GITHUB_REF}" -repo "${GITHUB_REPOSITORY}" -details "${GITHUB_SHA}") || true
        echo status: ${STAT}
        FailNum=$(echo $STAT | jq '.NumberOfFail')
        TestsNum=$(echo $STAT | jq '.NumberOfTests')
        GOPOGH_RESULT="${JOB_NAME} : completed with ${FailNum} / ${TestsNum} failures in ${TIME_ELAPSED}"
        echo ::set-env name=GOPOGH_RESULT::${GOPOGH_RESULT}
        echo ::set-env name=STAT::${STAT}
    - uses: actions/upload-artifact@v1
      with:
        name: docker_ubuntu_16_04
        path: minikube_binaries/report
    - name: The End Result
      shell: bash
      run: |
        echo ${GOPOGH_RESULT}
        echo "--------------------------------------------"
        echo $STAT | jq '.FailedTests' || true
        echo "--------------------------------------------"
        numberOfFailures=$(echo $STAT | jq '.NumberOfFail')
        if ["$numberOfFailures" -gt 0];then echo " ***$numberOfFailures Failed***";exit 2;fi
  docker_ubuntu_18_04:
    runs-on: ubuntu-18.04
    env:
      TIME_ELAPSED: time
      JOB_NAME: "Docker_Ubuntu_18_04"
      GOPOGH_RESULT: ""
    needs: [build_minikube]
    steps:
    - name: Install gopogh
      shell: bash
      run: |
        curl -LO https://github.com/medyagh/gopogh/releases/download/v0.1.16/gopogh-linux-amd64
        sudo install gopogh-linux-amd64 /usr/local/bin/gopogh
    - name: Download binaries
      uses: actions/download-artifact@v1
      with:
        name: minikube_binaries
    - name: Run integration test
      continue-on-error: true
      # bash {0} to allow test to continue to next step. in case of
      shell: bash {0}
      run: |
        cd minikube_binaries
        mkdir -p report
        mkdir -p testhome
        chmod a+x e2e-*
        chmod a+x minikube-*
        START_TIME=$(date -u +%s)
        KUBECONFIG=$(pwd)/testhome/kubeconfig MINIKUBE_HOME=$(pwd)/testhome ./e2e-linux-amd64 -minikube-start-args=--vm-driver=docker -expected-default-driver= -test.timeout=70m -test.v -binary=./minikube-linux-amd64 2>&1 | tee ./report/testout.txt
        END_TIME=$(date -u +%s)
        TIME_ELAPSED=$(($END_TIME-$START_TIME))
        min=$((${TIME_ELAPSED}/60))
        sec=$((${TIME_ELAPSED}%60))
        TIME_ELAPSED="${min} min $sec seconds "
        echo ::set-env name=TIME_ELAPSED::${TIME_ELAPSED}
    - name: Generate html report
      shell: bash
      run: |
        cd minikube_binaries
        export PATH=${PATH}:`go env GOPATH`/bin
        go tool test2json -t < ./report/testout.txt > ./report/testout.json || true
        STAT=$(gopogh -in ./report/testout.json -out ./report/testout.html -name "${JOB_NAME} ${GITHUB_REF}" -repo "${GITHUB_REPOSITORY}" -details "${GITHUB_SHA}") || true
        echo status: ${STAT}
        FailNum=$(echo $STAT | jq '.NumberOfFail')
        TestsNum=$(echo $STAT | jq '.NumberOfTests')
        GOPOGH_RESULT="${JOB_NAME} : completed with ${FailNum} / ${TestsNum} failures in ${TIME_ELAPSED}"
        echo ::set-env name=GOPOGH_RESULT::${GOPOGH_RESULT}
        echo ::set-env name=STAT::${STAT}
    - uses: actions/upload-artifact@v1
      with:
        name: docker_ubuntu_18_04
        path: minikube_binaries/report
    - name: The End Result
      shell: bash
      run: |
        echo ${GOPOGH_RESULT}
        echo "--------------------------------------------"
        echo $STAT | jq '.FailedTests' || true
        echo "--------------------------------------------"
        numberOfFailures=$(echo $STAT | jq '.NumberOfFail')
        if ["$numberOfFailures" -gt 0];then echo " ***$numberOfFailures Failed***";exit 2;fi
  none_ubuntu16_04:
    needs: [build_minikube]
    env:
      TIME_ELAPSED: time
      JOB_NAME: "None_Ubuntu_16_04"
      GOPOGH_RESULT: ""
    runs-on: ubuntu-16.04
    steps:
    - name: Install gopogh
      shell: bash
      run: |
        curl -LO https://github.com/medyagh/gopogh/releases/download/v0.1.16/gopogh-linux-amd64
        sudo install gopogh-linux-amd64 /usr/local/bin/gopogh
    - name: Download binaries
      uses: actions/download-artifact@v1
      with:
        name: minikube_binaries
    - name: Run integration test
      continue-on-error: true
      # bash {0} to allow test to continue to next step. in case of
      shell: bash {0}
      run: |
        cd minikube_binaries
        mkdir -p report
        mkdir -p testhome
        chmod a+x e2e-*
        chmod a+x minikube-*
        START_TIME=$(date -u +%s)
        KUBECONFIG=$(pwd)/testhome/kubeconfig MINIKUBE_HOME=$(pwd)/testhome ./e2e-linux-amd64 -minikube-start-args=--vm-driver=docker -expected-default-driver= -test.timeout=70m -test.v -binary=./minikube-linux-amd64 2>&1 | tee ./report/testout.txt
        END_TIME=$(date -u +%s)
        TIME_ELAPSED=$(($END_TIME-$START_TIME))
        min=$((${TIME_ELAPSED}/60))
        sec=$((${TIME_ELAPSED}%60))
        TIME_ELAPSED="${min} min $sec seconds "
        echo ::set-env name=TIME_ELAPSED::${TIME_ELAPSED}
    - name: Generate html report
      shell: bash
      run: |
        cd minikube_binaries
        export PATH=${PATH}:`go env GOPATH`/bin
        go tool test2json -t < ./report/testout.txt > ./report/testout.json || true
        STAT=$(gopogh -in ./report/testout.json -out ./report/testout.html -name "${JOB_NAME} ${GITHUB_REF}" -repo "${GITHUB_REPOSITORY}" -details "${GITHUB_SHA}") || true
        echo status: ${STAT}
        FailNum=$(echo $STAT | jq '.NumberOfFail')
        TestsNum=$(echo $STAT | jq '.NumberOfTests')
        GOPOGH_RESULT="${JOB_NAME} : completed with ${FailNum} / ${TestsNum} failures in ${TIME_ELAPSED}"
        echo ::set-env name=GOPOGH_RESULT::${GOPOGH_RESULT}
        echo ::set-env name=STAT::${STAT}
    - uses: actions/upload-artifact@v1
      with:
        name: none_ubuntu16_04
        path: minikube_binaries/report
    - name: The End Result
      shell: bash
      run: |
        echo ${GOPOGH_RESULT}
        echo "--------------------------------------------"
        echo $STAT | jq '.FailedTests' || true
        echo "--------------------------------------------"
        numberOfFailures=$(echo $STAT | jq '.NumberOfFail')
        if ["$numberOfFailures" -gt 0];then echo " ***$numberOfFailures Failed***";exit 2;fi
  none_ubuntu18_04:
    needs: [build_minikube]
    env:
      TIME_ELAPSED: time
      JOB_NAME: "None_Ubuntu_18_04"
      GOPOGH_RESULT: ""
    runs-on: ubuntu-18.04
    steps:
    - name: Install gopogh
      shell: bash
      run: |
        curl -LO https://github.com/medyagh/gopogh/releases/download/v0.1.16/gopogh-linux-amd64
        sudo install gopogh-linux-amd64 /usr/local/bin/gopogh
    - name: Download binaries
      uses: actions/download-artifact@v1
      with:
        name: minikube_binaries
    - name: Run integration test
      continue-on-error: true
      # bash {0} to allow test to continue to next step. in case of
      shell: bash {0}
      run: |
        cd minikube_binaries
        mkdir -p report
        mkdir -p testhome
        chmod a+x e2e-*
        chmod a+x minikube-*
        START_TIME=$(date -u +%s)
        KUBECONFIG=$(pwd)/testhome/kubeconfig MINIKUBE_HOME=$(pwd)/testhome ./e2e-linux-amd64 -minikube-start-args=--vm-driver=docker -expected-default-driver= -test.timeout=70m -test.v -binary=./minikube-linux-amd64 2>&1 | tee ./report/testout.txt
        END_TIME=$(date -u +%s)
        TIME_ELAPSED=$(($END_TIME-$START_TIME))
        min=$((${TIME_ELAPSED}/60))
        sec=$((${TIME_ELAPSED}%60))
        TIME_ELAPSED="${min} min $sec seconds "
        echo ::set-env name=TIME_ELAPSED::${TIME_ELAPSED}
    - name: Generate html report
      shell: bash
      run: |
        cd minikube_binaries
        export PATH=${PATH}:`go env GOPATH`/bin
        go tool test2json -t < ./report/testout.txt > ./report/testout.json || true
        STAT=$(gopogh -in ./report/testout.json -out ./report/testout.html -name "${JOB_NAME} ${GITHUB_REF}" -repo "${GITHUB_REPOSITORY}" -details "${GITHUB_SHA}") || true
        echo status: ${STAT}
        FailNum=$(echo $STAT | jq '.NumberOfFail')
        TestsNum=$(echo $STAT | jq '.NumberOfTests')
        GOPOGH_RESULT="${JOB_NAME} : completed with ${FailNum} / ${TestsNum} failures in ${TIME_ELAPSED}"
        echo ::set-env name=GOPOGH_RESULT::${GOPOGH_RESULT}
        echo ::set-env name=STAT::${STAT}
    - uses: actions/upload-artifact@v1
      with:
        name: none_ubuntu18_04
        path: minikube_binaries/report
    - name: The End Result
      shell: bash
      run: |
        echo ${GOPOGH_RESULT}
        echo "--------------------------------------------"
        echo $STAT | jq '.FailedTests' || true
        echo "--------------------------------------------"
        numberOfFailures=$(echo $STAT | jq '.NumberOfFail')
        if ["$numberOfFailures" -gt 0];then echo " ***$numberOfFailures Failed***";exit 2;fi
  podman_ubuntu_18_04:
      needs: [build_minikube]
      env:
        TIME_ELAPSED: time
        JOB_NAME: "Podman_Ubuntu_18_04"
        GOPOGH_RESULT: ""
      runs-on: ubuntu-18.04
      steps:
      - name: install podman
        shell: bash
        run: |
          . /etc/os-release
          sudo sh -c "echo 'deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/ /' > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list"
          wget -q https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/xUbuntu_${VERSION_ID}/Release.key -O- | sudo apt-key add -
          sudo apt-key add - < Release.key || true
          sudo apt-get update -qq
          sudo apt-get -qq -y install podman
          sudo podman version || true
          sudo podman info || true
      - name: Install gopogh
        shell: bash
        run: |
          curl -LO https://github.com/medyagh/gopogh/releases/download/v0.1.16/gopogh-linux-amd64
          sudo install gopogh-linux-amd64 /usr/local/bin/gopogh
      - name: Download binaries
        uses: actions/download-artifact@v1
        with:
          name: minikube_binaries
      - name: Run integration test
        continue-on-error: true
        # bash {0} to allow test to continue to next step. in case of
        shell: bash {0}
        run: |
          cd minikube_binaries
          mkdir -p report
          mkdir -p testhome
          chmod a+x e2e-*
          chmod a+x minikube-*
          START_TIME=$(date -u +%s)
          KUBECONFIG=$(pwd)/testhome/kubeconfig MINIKUBE_HOME=$(pwd)/testhome ./e2e-linux-amd64 -minikube-start-args=--vm-driver=docker -expected-default-driver= -test.timeout=70m -test.v -binary=./minikube-linux-amd64 2>&1 | tee ./report/testout.txt
          END_TIME=$(date -u +%s)
          TIME_ELAPSED=$(($END_TIME-$START_TIME))
          min=$((${TIME_ELAPSED}/60))
          sec=$((${TIME_ELAPSED}%60))
          TIME_ELAPSED="${min} min $sec seconds "
          echo ::set-env name=TIME_ELAPSED::${TIME_ELAPSED}
      - name: Generate html report
        shell: bash
        run: |
          cd minikube_binaries
          export PATH=${PATH}:`go env GOPATH`/bin
          go tool test2json -t < ./report/testout.txt > ./report/testout.json || true
          STAT=$(gopogh -in ./report/testout.json -out ./report/testout.html -name "${JOB_NAME} ${GITHUB_REF}" -repo "${GITHUB_REPOSITORY}" -details "${GITHUB_SHA}") || true
          echo status: ${STAT}
          FailNum=$(echo $STAT | jq '.NumberOfFail')
          TestsNum=$(echo $STAT | jq '.NumberOfTests')
          GOPOGH_RESULT="${JOB_NAME} : completed with ${FailNum} / ${TestsNum} failures in ${TIME_ELAPSED}"
          echo ::set-env name=GOPOGH_RESULT::${GOPOGH_RESULT}
          echo ::set-env name=STAT::${STAT}
      - uses: actions/upload-artifact@v1
        with:
          name: podman_ubuntu_18_04
          path: minikube_binaries/report
      - name: The End Result
        shell: bash
        run: |
          echo ${GOPOGH_RESULT}
          echo "--------------------------------------------"
          echo $STAT | jq '.FailedTests' || true
          echo "--------------------------------------------"
          numberOfFailures=$(echo $STAT | jq '.NumberOfFail')
          if ["$numberOfFailures" -gt 0];then echo " ***$numberOfFailures Failed***";exit 2;fi
  # After all 4 integration tests finished
  # collect all the reports and upload
  upload_all_reports:
    if: always()
    needs: [docker_ubuntu_16_04,docker_ubuntu_18_04,none_ubuntu16_04,none_ubuntu18_04,podman_ubuntu_18_04]
    runs-on: ubuntu-18.04
    steps:
    - name: download results docker_ubuntu_16_04
      uses: actions/download-artifact@v1
      with:
        name: docker_ubuntu_16_04
    - name: cp to all_report
      shell: bash
      run: |
        mkdir -p all_reports
        cp -r docker_ubuntu_16_04 ./all_reports/
    - name: download results docker_ubuntu_18_04
      uses: actions/download-artifact@v1
      with:
        name: docker_ubuntu_18_04
    - name: cp to all_report
      shell: bash
      run: |
        mkdir -p all_reports
        cp -r docker_ubuntu_18_04 ./all_reports/
    - name: download results none_ubuntu16_04
      uses: actions/download-artifact@v1
      with:
        name: none_ubuntu16_04
    - name: cp to all_report
      shell: bash
      run: |
        mkdir -p all_reports
        cp -r none_ubuntu16_04 ./all_reports/
    - name: download results none_ubuntu18_04
      uses: actions/download-artifact@v1
      with:
        name: none_ubuntu18_04
    - name: cp to all_report
      shell: bash
      run: |
        mkdir -p all_reports
        cp -r none_ubuntu18_04 ./all_reports/
    - name: download results podman_ubuntu_18_04
      uses: actions/download-artifact@v1
      with:
        name: podman_ubuntu_18_04
    - name: cp to all_report
      shell: bash
      run: |
        mkdir -p all_reports
        cp -r podman_ubuntu_18_04 ./all_reports/
    - uses: actions/upload-artifact@v1
      with:
        name: all_reports
        path: all_reports
1 Like

Hey! Just a general note: I suggest breaking this yml file up into smaller pieces. Right now, given its size, it’s very difficult for someone else to read (or at least for me). I would say that this approach as similar to putting all of your code in the main function vs using smaller functions to make the mail loop more broken apart. 

With that said, I think the key pieces to solve this are :"

if ["$numberOfFailures" -gt 0];then echo " ***$numberOfFailures Failed***";exit 2;fi
  none_ubuntu16_04:

 and 

if: always()

What does the always function do? Maybe it makes sense to push the artifacts after each job rather than at the end since you are letting them run asynchronously. 

1 Like

>>I have manged to make it happen using job needs ! but if any of the 4 parallle jobs before the last jobs fail, it will not run the last job !

When you add “if: always()”, the job (named upload_all_reports) will run even if the 4 dependency jobs failed. 

 What I want to prompt is that you may get "An Artifact with name “***” was not found. " error if any of the 4 parallle jobs failed at the previous steps of upload-artifacts step . 

not found.png

If you want to download all the artifacts in the final job, you need to make sure all artifacts are uploaded even if the job ( integraiton tests ) failed.  

You could add if: always() to the upload artifacts step in all those 4 jobs integraiton tests. 

For example: 

upload files.png