Sharing steps between jobs and jobs between workflows

I’m trying to wrap my head around GitHub actions for CI/review processes after coming from an arguably much simpler environment which was GitLab CI. If I understand correctly, workflows are meant to delineate specific processes which happen in a repo. With this mind, say I have the following workflows:

  1. PR review: install deps, run lint, run tests, run code coverage.

  2. Publish: install deps, run lint, run tests, run code coverage, build, publish.

You can see that the same jobs occur in both workflows which is a lot of copy-pasting. So, my first question is whether there exists a way to share these job definitions across multiple workflows that doesn’t involve creating a custom action?

The second problem is concerned with jobs in a workflow. “Install deps” is a common job that is needed for subsequent jobs in a workflow. Say I have caching added to this as well via actions/cache. I can add a needs field to other jobs so that they execute once the “Install deps” job is done and cache is available. But, I now need to repeat these same steps from “Install deps” in other jobs as well which is again, repititive. Is there a way to share steps between jobs in the same workflow? I see that we can’t use YAML features like anchors which would’ve solved this problem but is there something else? Or if there’s a chance that I’m doing caching wrong, let me know.

If not, I would like to request to add these features which would not only make writing workflows a joy, but also avoid a lot of potential errors that come from copy-pasting or resorting to needlessly complicated methods.

6 Likes

To make a comparison to GitLab’s CI, I could simply run multiple jobs parallely on the same repo instance with packages installed at a very early step without any fuss. I don’t think that’s possible here since each job in a workflow gets its own fresh VM instance. So, I wouldn’t be able to share the workspace where the repo is checked out with npm modules installed, would I?

> 1. PR review: install deps, run lint, run tests, run code coverage.

> 2. Publish: install deps, run lint, run tests, run code coverage, build, publish.

>

> You can see that the same jobs occur in both workflows which is a lot of copy-pasting. So, my first question is whether there exists a way to share these job definitions across multiple workflows that doesn’t involve creating a custom action?

Nope. However, the way you avoid duplication is by creating a single workflow with the build and publish jobs being conditional on the event.

on:
    push:
    pull_request:
    cache_deps:
        ...
    lint:
        ...
    publish:
        name: 'Publish docker image'
        if: github.event_name == 'push'
        ...

> But, I now need to repeat these same steps from “Install deps” in other jobs as well which is again, repititive. Is there a way to share steps between jobs in the same workflow? I see that we can’t use YAML features like anchors which would’ve solved this problem but is there something else? Or if there’s a chance that I’m doing caching wrong, let me know.

Each job would have to initialize/check the actions/cache in one of its steps, yes. Each job is its own vm. I suppose you could encapsulate that logic into an action, but you’d still need to use that action within one of the steps in each job, resulting in some amount of duplication unfortunately.

2 Likes

The workspaces aren’t shared between jobs, no,

At least, they aren’t without some effort:

  • actions/cache allows you to cache a path based on a key e.g. you could cache the ~/npm_modules dir.

  • actions/upload-artifact / actions/download-artifact allow you to store some artifacts/files for a particular run. Subsequent jobs in the workflow can download the artifacts e.g. a build job uploads the executable, and the test job downloads it to test it.

2 Likes

Currently, Github actions doesn’t support a workflow template for steps or jobs. You could ask for this feature in the  Feedback form for GitHub Actions

And to share workspace with npm packages installed , I agree with @lucianposton , you could use actions/cache. What’s more, you also need to add actions/checkout and actions/cache two steps to each job. 

I understand that I’d have to duplicate the cache action between jobs. But it’s not as simply as putting uses: @action/cache, it involves config like the one mentioned here: https://github.com/actions/cache/blob/master/examples.md#node—yarn which is very brittle and copy-pasting stuff like that isn’t very reliable in the long term.

Adding core YAML features helps ease some of these pains I’m facing in my projects. I tend to trade-off parallelism for maintanable configs by squeezing all PR-related tasks in one single job these days.

2 Likes