Github Actions runs against wrong commit

Hi!

I created a PR, which triggers test runs via Github Actions. It runs unit tests and functional tests, but I noticed an issue with the tests and added some debugging information. When I pushed the new commit, the tests ran again, but didn’t produce the debug info added. I then changed the debugging to another format, and pushed again. The tests triggered as they should, but now I got the debugging info from the previous format.

It seems that the Actions don’t always take the latest commit in the PR and run against that for some reason. Since the error in the tests is one that can’t be reproduced locally, I’m forced to slow down testing considerably, and need to think of alternatives to Github Actions.

Any idea what could cause this?

Thanks for your feedback.
Do you have a public repo demonstrating this? Please link us the broken workflow run.
At the same time, you can also use the following steps to print out the Github context to check if the event name and commit sha are correct.

    steps:
      - name: Dump GitHub context
        env:
          GITHUB_CONTEXT: ${{ toJson(github) }}
        run: echo "$GITHUB_CONTEXT"

Hi Nico,

Unfortunately there’s no public repo with this problem, I’ll see if I have time to create a demo repo demonstrating this.

I’ll add that context check, but since it’s a PR I can’t compare the commit sha to an existing commit, because it creates the merged commit on the fly.

Here’s a somewhat redacted workflow file.

name: Long tests

on:
  push:
    branches:
      - master
  pull_request:
  schedule:
    - cron: '0 3 * * *'

env:
  REDACTED: true

jobs:
  phpunit:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        group:
          - core
          - ungrouped
          - redacted
          - additional
          - groups
    steps:
    - name: Checkout
      uses: actions/checkout@v2

    - name: Add SSH key for system clients
      uses: webfactory/ssh-agent@v0.2.0
      with:
          ssh-private-key: ${{ secrets.SYSTEMCLIENTS_SSH_KEY }}

    - name: Setup PHP version
      uses: shivammathur/setup-php@v2
      with:
        php-version: '7.2'
        extensions: mbstring, intl, imap, bcmath, bz2, calendar, exif, gd, gettext, gmp, ldap, pcntl, pdo_mysql, pspell, recode, shmop, snmp, soap, sockets, sysvmsg, tidy, wddx, xmlrpc, xsl, zip, sysvsem, sysvshm
        ini-values: date_timezone=Europe/Helsinki
        coverage: none
        tools: phpunit

    - name: Run installer
      run: ./install.sh

    - name: Install dependencies - code
      run: composer -dcode install
    
    - name: Clear cache
      run: rm -rf ./code/cache/*/

    - name: Functional tests - ${{ matrix.group }}
      run: code/vendor/phpunit/phpunit/phpunit --group ${{ matrix.group }} code/tests/Functional

Hi aleksi,

Thanks for your cooperation if you can take the time to share the demo repo. It is benefit for us to investigate this further.

What is the type of activity in your pull request, open or synchronize?
If the type is synchronize, you could check if the following two Commit SHAs are the same.

Yes, the type is synchronize and the after hash matches the commit. After I added this task the problem has not resurfaced. But looking at older tests, I think I found the issue.

When the tests ran correctly (but failed), the commit was 1d701f243f4073d9ab9bfd185cb7d5d44d2b2698. If I open the Checkout task and the Checking out the ref section, it says HEAD is now at 80afffc Merge 1d701f243f4073d9ab9bfd185cb7d5d44d2b2698 into 30f3bb799be418b35fc0b3446d9bf6e8c67067fe.

After this, I added logging, committed and pushed the commit a854e2f67d34b2dffde1007edbc34b64f2355258, but when checking the same place, it’s still HEAD is now at ec1f306 Merge 1d701f243f4073d9ab9bfd185cb7d5d44d2b2698 into 30f3bb799be418b35fc0b3446d9bf6e8c67067fe. For some reason it merges the old commit to the target branch and runs the tests, it doesn’t pick up the new commit.

For now, I’ll keep the context dump task in the workflow and see if it occurs again.