Publishing from multiple workflows

My group is trying to workflows to develop a CI/CD pipeline. At this point, I just want to report data about the build as we edit it. To that end we have made several workflows that are triggered by specific code push events to gather data and then publish it on our github pages site. See the example below

---
name: YML Change Report
# Rules for yaml Lint
# https://yamllint.readthedocs.io/en/stable/rules.html

# fix truthy warning per
# https://github.com/adrienverge/yamllint/issues/158#issuecomment-586706596
# yamllint disable-line rule:truthy
on:
  push:
    paths:
      # This workflow triggers any time one of the workflow scripts is updated
      - ".github/workflows/**.yml"

jobs:
  FindChangedFiles:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1
      - name: List all files
        # This gets a JSON file with the list of all files in the commit
        # yamllint disable rule:line-length
        run: |
          curl --request GET \
          --url https://api.github.com/repos/${{ github.repository }}/commits/${{ github.sha }} \
          --header 'authorization: Bearer ${{ secrets.GITHUB_TOKEN }}' \
          --header 'content-type: application/json' > yml_change.json
        # yamllint enable rule:line-length
      - name: Parse change status
        # this runs a python script we wrote to parse the JSON and append to an existing history file
        # yamllint disable rule:line-length
        run: python .github/ciPython/yml_commit_updater.py yml_change.json ./html/cidata/yml_history.js
        # yamllint enable rule:line-length
      - name: Commit history file
        run: |
          git config --local user.email "None."
          git config --local user.name "GitHub Action"
          git add ./html/cidata/yml_history.js
          git commit -m "Auto-push yml change history" -a
      - name: Push changes
        uses: ad-m/github-push-action@master
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          branch: 'master'

The general process is

  • checkout the code, including the /html folder were our CI Data is stored
  • Run whatever task from which we want to gather data
  • Update the /html/cidata/.js
  • Use ad-m’s github-push-action to publish the CI Data update.

The problem is that if we have multiple workflows triggered by a single push (e.g. I have edited a YML file and I have edited some python code) we now have two workflows in a race to push their updates. The second one always generates an error.

hint: Updates were rejected because a pushed branch tip is behind its remote

Obviously the slower workflow is a version behind the master branch and we should be pulling that update before pushing. At least, that is what a human would do. So, I get why this is happening, but I’m hoping there is a way around it. Even if I tried to update the files right before the final push, those actions can’t be made into a single atomic action and we will just have a different race condition. I’d much rather publish the data to some other location where we can pull in all the date nightly. I just don’t know github well enough to suggest a viable alternative. Thoughts?

1 Like

I’m afraid I have another problem rather than a solution: One push may include multiple commits, and the last one (github.sha) might not even be the one (or the only one) that changed the YAML file. You should be able to get the list of commits from the push event (github.event) itself.

1 Like

You could add git pull to pull the remote changes before push changes. For example:

    - name: Commit files
      id: commit
      run: |
        git config --local user.email "action@github.com"
        git config --local user.name "github-actions"
        git add --all
        git commit -m "Add changes" -a
        git pull origin $GITHUB_REF 
    - name: Push changes
      uses: ad-m/github-push-action@master
      with:
        github_token: ${{ secrets.GITHUB_TOKEN }}
        branch: ${{github.ref}}  
3 Likes

We did consider that as an option. It still isn’t atomic, but we are going to implement this as a short term fix.
Thanks for giving us a snippet to start from.

1 Like

So we tried to implement this snippet as such:

- name: Commit files
      id: commit
      run: |
        git config --local user.email "action@github.com"
        git config --local user.name "github-actions"
        git config pull.ff only
        git add --all
        git commit -m "Add changes" -a
        git pull origin KKBranch
    - name: Push changes
      uses: ad-m/github-push-action@master
      with:
        github_token: ${{ secrets.GITHUB_TOKEN }}
        branch: 'KKBranch'

We had to use git config pull.ff only cmd due to an warning: Pulling without specifying how to reconcile divergent branches is discouraged. error message. Afterwards, we resulted in these error messages:

[KKBranch d5de0c5] Add changes
 1 file changed, 1 insertion(+), 1 deletion(-)
fatal: could not read Username for 'https://github.com': No such device or address
##[error]Process completed with exit code 1.

Any other approaches/configurations we should do?

1 Like

I checked the initial post , you are using

  - uses: actions/checkout@v1

You could consider using checkout@v2 . There are some difference between v1 and v2, you could refer to this readme file to know about checkout v2.

1 Like