reusing/sharing/inheriting steps between jobs declarations

I have a few setup steps that i need to make in most of my jobs
I would like to be able to delcare a set of steps onces, and reference them.
it’s a composition of other actions, i don’t want to write custom action for that.
If cutsom action chould be a set of steps - that could work for me

Thanks!

60 Likes

That is very much needed.

CircleCI allows defining commands that are a set of steps: https://circleci.com/docs/2.0/configuration-reference/#commands-requires-version-21

My first try was actually to create my own action which just consisted of the set of steps I want to reuse, imagine my disappointment when I realised they need to be written in JavaScript. Copy/Pasting my 50l YAML setup is pretty bad.

This is definitely a feature that needs to be present, either as commands, YAML anchors (hmm) or being able to define actions as a set of steps.

11 Likes

IMO this needs to be a high priority feature for the action team. Many times people want to run the same steps/jobs and then add on to them.

E.g., always build and test, then publish/deploy if it is tagged for release and the build/testing was successful.

Creating a full javascript action for that is ridiculous…

Right now there is a dirty way to do this without code duplication:

This requires adding some type of identifier to the tag so it can be checked for release. E.g., add “-release” to the end of the tag.

name: build
on: [push, pull_request]

jobs:
  # Builds, tests, and releases (if a release) the
  build_test_release:
    runs-on: ubuntu-latest
    steps:
      # ...
      # build and test
      #
      # Only runs if this is a tag and if this is a release.
# (Do not want to run this if it is a branch with '-release' in it...)
      - name: ensure release tag matches version
        if: contains(github.ref, '/tags/') && contains(github.ref, '-release')
        env:
          ref: ${{ github.ref }}
        run: |
          test $PROJECT_VERSION == $(echo $ref | awk '{n=split($1,A,"/"); split(A[n],B,"-"); print B[1]}')
      # again, have to check to make sure on release tag..
      - name: release
        if: contains(github.ref, '/tags/') && contains(github.ref, '-release')
        uses: actions/create-release@v1
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        with:
          tag_name: ${{ env.PROJECT_VERSION }}-release
          release_name: ${{ env.PROJECT_VERSION }}
          draft: false
          prerelease: false

Right now, to do this in a less hacky way, it would require duplicating the same build/test jobs/steps between workflow files, because only at the top level of the workflow can you pattern match the branch. This obviously  isn’t a very clean solution and is error prone since someone may forget to update one of the workflows. 

If allowing one workflow file to reference or depend on another is too difficult it would be nice to at least pattern match branch/tag on the job level so that one job can depend on another and then run only if it matches the pattern (or whatever requirements like is allowed at the top level of the workflow file).

4 Likes

Hello @bnaya

Thank you for your feedback! We’re always working to improve GitHub and the GitHub Community Forum, and we consider every suggestion we receive. I’ve logged your feature requests in our internal feature request list. Though I can’t guarantee anything or share a timeline for this, I can tell you that it’s been shared with the appropriate teams for consideration.

Once again, thank you for your input!

Greatly appreciated

MChevy422

4 Likes

Thanks @mchevy422 !

1 Like

One thing I forgot to mention in my post: it should also sharing the services. I don’t really want to copy/paste my redis/postgres setup on every task requiring it if possible.

2 Likes

There are some new news for this topic?

2 Likes

Any updates on this?

6 Likes

Can’t believe there’s no quick way to do this 

7 Likes

It would be great to get an update on this. Theoretically, my use case wouldn’t require shared steps if we could share the container more easily between several jobs.

Here’s what I mean: At the start of my workflow, I need to:

  1. Checkout my code.
  2. Install dependencies.
  3. Build the source code.
  4. Set up a testing environment by running a script.

After this has been accomplished, I would like to run several tasks in parallel:

  1. Running unit tests
  2. Running e2e tests
  3. Sending a build artifact to an internal service

Each of these three items depends on the first several steps having been completed. Unfortunately, the only way to support this right now is by using artifacts, which really doesn’t capture enough about what is going on to replicate the behavior. Artifacts work fine for my 3rd option, since I don’t need the dependencies around to post the build artifact somewhere. But I wish I could run options 1 and 2 in the same environment after the first section completes. And ideally they would run in parallel.

Reusable steps would definitely make this easier, but it would still take a lot longer to do the exact same thing 3 different times in parallel.

2 Likes

@thejoebourneidentity I apologise for mentioning you directly like this out of nowhere, but this is a very serious limitation and there hasn’t been any official word on it so far.

Re-using steps would ease working around other limitations, by simplifying the boilerplate of such “workarounds”.

I’m “hot” on this feature too.
Here’s my use-case:

I want to create a workflow for PRs, workflow for branches and workflow for commits to the master.

The first two install dependencies, run unit tests, e2e tests, coverages, lint and collect into an artifact the coverage report and the e2e logs from the test and coverage runs.
So far - so sweet.

The third one also publishes npm packages, reports to coveralls, and maybe even to a snyk profile, and if need be - compiles and publishes a docker to dockerhub.

So far, I could have get away with a jobs.<id>.if.

But. I need a github-actions badge that will indicate for only commits to the master that end with a published version.
Now, since actions-badges reflect the status of a workflow - I need a second file which is replicates most parts of the first, and ads on it more, and that, uh, how to put it gently… …emits an odor which is quite different… ?

Now I need to generate the workflow using templating tools so I will not have to duplicate code. Really?

Also - suppose I want to publish a version only if some paths have changed. I want to build for every change, but publish only when files that are actually shipped change - Same story here.
Although - this story could have been solved if I could limit specific steps by changes to paths like I can for entire workflows - which is another feature I wholeheartedly urge you to consider.

If I could limit jobs to path changes - I could do some output tricks, but no. the limitation is for the entire workflow, which lives in a bubble…

GitLab solved all this by using a single file for all your workflows and they not only support yaml anchors, but also have an additional mechamism to let you define resuable parts anywhere in the doc, not just on top - and use them from anywhere (a little like hoisting in javascript), so with that the result is amazing.

Can’t wait to see github-actions catch up with such capabilities - it will be a total blast!!!

IMHO - if you’re tight on resources I’d invest on one of the following two directions:
1 - require/import supbarts from relative paths
And if that’s too much - start with:
2 - support yaml anchors, which will ease the pressure and give you time

Yaml anchors are Yaml-Parser feature.
If you mutate config objects and are afraid they will be contaminated between stages - here’s a “one liner” to solve that for you:

let workflow = require('js-yaml').safeLoad(`.github/workflows/${wfName}`)
try {
   workflow = JSON.parse(JSON.stringify(worflow))
} catch(e) {
   throw new Error('You cannot use anchors for circular references')
}

ugly, yes. but considering the workflow files are not that big - it’s a bearable evil until you find the time to do it better. I’m sure it can be expressed in whatever language your infra is on.

Mmm. here’s an example of what anchors could do for us:

jobs:
  smoke-test:
    name: Node 12.x - ubuntu
    matrix:
        os: [ubuntu]
        node: [12]
        mongo: [4]
        redis: [5]
    timeout-minutes: 10
    steps: &basic-ci-steps
      - name: checkout
        uses: actions/checkout@v2
      - name: node ${{matrix.node }}
        uses: actions/setup-node@v2-beta
        with:
          node-version: ${{ matrix.node }}
      - name: mongo start
        uses: supercharge/mongodb-github-action@1.3.0
        with:
          mongodb-version: ${{ matrix.mongo }}
      - name: redis start
        uses: supercharge/redis-github-action@1.1.0
        with:
          redis-version: ${{ matrix.redis }}
      - name: Env Info
        run: npx envinfo
      - name: npm i
        run: npm i
      - name: test
        run: npm test
      - name: cover
        run: npm run cover
      - name: gather results
        run: tar -czvf ci-results.tar.gz coverage
      - name: save results
        uses: actions/upload-artifact@v1
        with:
          name: ci-results.node-${{ matrix.node }}.mongo-${{ matrix.mongo }}.redis-${{ matrix.redis }}
          path: ci-results.tar.gz

  lint:
    runs-on: ubuntu-latest
    needs: smoke-test
    steps:
      - name: checkout
        uses: actions/checkout@v2
      - name: node 12
        uses: actions/setup-node@v2-beta
        with:
          node-version: 12

      - name: lint
        run: npm run lint

  test-matrix:
    needs: [smoke-test, lint]
    strategy:
      matrix:
        os: [ubuntu, windows]
        node: [ 13, 12, 10 ]
        mongo: [ 4, 3 ]
        redis: [ 5, 4 ]
      exclude: #what's covered in smoke-test
          - os: ubuntu
            node: 12
            redis: 5
            mongo: 4
    steps: *basic-ci-steps
     
  cd:
    runs-on: ubuntu-latest
    if: github.ref == 'master'
    needs: ci
    steps:
      - name: versi
        run: ./node_modules/.bin/versi
      - name: publish
        env:
          NODE_AUTH_TOKEN: ${{secrets.PACKAGE_PUBLISH_TOKEN}}
        run: npm publish

The example above runs tests and lint on one OS & node version, and only when they pass - it enters the matrix, where each run spawns it’s own mongo db for store adapter tests and redis for cache adapter tests.

So much CPU time saved! Wouldn’t that be lovely?

(sory for the multiple edits, I’m running it over and over in my head…)

Greetings,

Actions engineering manager here. We are very aware of this pain point and are working on a feature to address this now. Expect it to release toward the end of the summer (but can’t make promises on deadlines/releases). Please keep an eye out for it soon though.

We would also like to support anchors but there are some technical/security limitations related to the parser that we need to be careful of.

You can take a look at this issue here to keep track of it in the open source runner repo: https://github.com/actions/runner/issues/438

Thanks!

16 Likes

Great to hear, thanks @hross

Hi @hross, sorry to be obtuse here, but how do composite actions (as described in the issue you linked) enable me to reuse steps? They seem to be used for something different entirely (running multiple heterogeneous steps within the same job, perhaps? The blog post was really unclear to me, but what was clear is that it serves neither the same purpose as anchors/references within a single YAML file would, nor the feature request here (declare a set of steps once, and reference them). Am I missing something - can you demonstrate for me how this feature lets us do that?

No problem @kislyuk. Composite run steps are the first step to making totally reusable composite actions as you describe them. I am going to link to some resources below that will do a better job describing:

Hopefully this makes things more clear. Let me know if there is anything else I can do to help clarify.

Hi @hross

Thanks for the clarification, specifically in regards to the lack of support of “actions within actions”.

I can see this is still in the pipeline of things you’re planning to support.

Any idea on the timeline? This one is quite critical for my team to switch to Github actions.

Can we expect support will be released later this year, or should we expect 2021?

The example in the documentation here, suggests that one can use the needs parameter to let jobs depend on a previous setup & build job. But the example actually is seems to be syntactically incorrect, the runs-on is mandatory but it is missing in the build job in the example:

Would be great to allow jobs to run in a copy of the virtual environment of a previous job, so one can build once and then run many tests in parallel in clones of the finished environment.

@hross Any news on this feature you mentioned? :slight_smile:

1 Like