Generate steps dynamically by matrix/outputs

How would you write the following issue:

I have a workflow with 2 jobs (of course, this is a minimized case of my real workflow) where job build-apps (A) is compiling and building some apps with a matrix and job build-docker-image (B) should take the compiled data of all of job A apps and deploy it somewhere, let’s just say push it to docker.
Right now, I’ve done it with artifacts - I upload each of the compiled apps in job A, download all artifacts in job B so in the Dockerfile it will just copy all of the files:

jobs:
  build-apps:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        apps: [
          {
            "name": "first app",
            "basePath": "../first_app"
          },
          {
            "name": "second app",
            "basePath": "../second_app"
          }
        ]
    steps:
      - uses: actions/checkout@v2
      - name: ${{ matrix.apps.name }} cache
        uses: actions/cache@v2
        id: apps-cache
        with:
          key: ${{ matrix.apps.basePath }}/**
          path: ${{ matrix.apps.basePath }}/dist
      - name: build ${{ matrix.apps.name }}
        if: steps.apps-cache.outputs.cache-hit != 'true'
        run: build ${{ matrix.apps.name }}
      - uses: actions/upload-artifact@v2
        with:
          name: ${{ matrix.apps.name }}
          path: ${{ matrix.apps.basePath }}/dist
  build-docker-image:
    runs-on: ubuntu-latest
    needs:
      - build-apps
    steps:
      - uses: actions/checkout@v2
      - uses: actions/download-artifact@v2
      - name: Build and Push ${{ steps.vars.outputs.image-branch }}
        uses: docker/build-push-action@v2
        with:
          context: .
          push: true
          cache-from: type=registry
          cache-to: type=inline

My issue is that with large amount of apps, it takes a lot of time to download all of the artifacts. I would like to use the caching mechanism for this as well. If I could download all of the apps caches, just like in job A step apps-cache, I could use the output of it just like artifacts, and it will be much much faster.
Now, I know I can define hardcoded all the cache steps for each of the apps, but that means I will have to maintain both of job A matrix and job B steps. If anyone will add a new app, he will need to add it in both places.
How can I use something similar to matrix to build job B steps dynamically, according to the matrix of job A?
I already managed to export the job A matrix to an external file and generate it dynamically as was suggested here, but I can’t use it to generate the steps in job B as well.

What is your suggestion or solution? :slight_smile:

1 Like