Unpredictable obsolete http blocking in Maven 3.8.1 on Github Actions

Github recently updated the default Ubuntu image used in Workflows to use Maven 3.8.1. In this version of Maven, access to repositories using http, rather than https, are blocked by default. I’ve updated the configuration in our settings.xml to unblock access to our self-hosted repository, and this has allowed builds to continue.

      - name: Setup Repo credentials
        # maven-settings-action v2.4.0
        uses: s4u/maven-settings-action@e967605742bd38de606cf6edb7c89349ce917c6a
        with:
          servers: |
            [{
                "id": "repo.releases.mirror",
                "username": "${{ secrets.REPO_USERNAME }}",
                "password": "${{ secrets.REPO_PASSWORD }}"
            }]
          mirrors: |
            [{
                "id": "repo.releases.mirror",
                "name": "repo.releases",
                "mirrorOf": "repo.releases",
                "url": "http://our-repo-url",
                "blocked": false
            }]

However, some branches have been failing during the deploy goal.

Error:  Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:3.0.0-M1:deploy (default-deploy) on project hats:
ArtifactDeployerException: Failed to retrieve remote metadata com.groupid:artifactid:version/maven-metadata.xml:
Could not transfer metadata com.groupid:artifactid:version/maven-metadata.xml from/to repo.snapshots (http://our-repo-url/):
authentication failed for http://our-repo-url/com/groupid/artifactd/version/maven-metadata.xml, status: 401 Unauthorized -> [Help 1]

These branches have no issue downloading dependencies from our repository using the unblocked http URL. But when they get to the deploy step, and try to download the maven-metadata.xml from the same internal repository, it fails with a 401 Unauthorized. Some branches don’t exhibit this behavior at all. If we recreate the branches that do have the issue, with the exact same commits, they build fine. But then when we merge in changes from the main branch they start failing with the above issue again.

I suspect the issue is possibly in Github Actions itself, perhaps in a caching layer. But So far I haven’t been able to reliably recreate the issue on demand. Some branches simply start failing because of this, and once that’s happened we’ve been unable to get that branch to successfully deploy.

Does anyone have any idea what is going on here?

So I’ve been digging into this all day and I have some more to add here.

In reality we have four repositories defined in our pom. There are releases and snapshots repositories for regular dependencies and releases and snapshots for plugin dependencies. So the actual configuration for settings.xml looks like this:

      - name: Setup Repo credentials
        # maven-settings-action v2.4.0
        uses: s4u/maven-settings-action@e967605742bd38de606cf6edb7c89349ce917c6a
        with:
          servers: |
            [{
                "id": "repo.releases.mirror",
                "username": "${{ secrets.REPO_USERNAME }}",
                "password": "${{ secrets.REPO_PASSWORD }}"
            },
            {
                "id": "repo.snapshots.mirror",
                "username": "${{ secrets.REPO_USERNAME }}",
                "password": "${{ secrets.REPO_PASSWORD }}"
            },
            {
                "id": "repo.plugin.releases.mirror",
                "username": "${{ secrets.REPO_USERNAME }}",
                "password": "${{ secrets.REPO_PASSWORD }}"
            },
            {
                "id": "repo.plugin.snapshots.mirror",
                "username": "${{ secrets.REPO_USERNAME }}",
                "password": "${{ secrets.REPO_PASSWORD }}"
            }]
          mirrors: |
            [{
                "id": "repo.releases.mirror",
                "name": "repo releases mirror",
                "mirrorOf": "repo.releases",
                "url": "http://repo/repository/releases",
                "blocked": false
            },
            {
                "id": "repo.snapshots.mirror",
                "name": "repo snapshots mirror",
                "mirrorOf": "repo snapshots mirror",
                "url": "http://repo/repository/snapshots",
                "blocked": false
            },
            {
                "id": "repo.plugin.releases.mirror",
                "name": "repo plugin releases mirror",
                "mirrorOf": "repo.plugin.releases",
                "url": "http://repo/repository/releases",
                "blocked": false
            },
            {
                "id": "repo.plugin.snapshots.mirror",
                "name": "repo plugin snapshots mirror",
                "mirrorOf": "repo plugin snapshots mirror",
                "url": "http://repo/repository/snapshots",
                "blocked": false
            }]

The repository URLs themselves are exactly the same apart from the /releases and /snapshots suffixes.

The maven portion of the build is split into two steps. The first uses the install goal, and the second uses the deploy goal and skips the tests.

      - name: Build, Test, and Install local artifacts with Maven
        env:
          VERSION: ${{ steps.buildvars.outputs.version }}
        run: mvn --batch-mode -T 1C -Drevision=${{ env.VERSION }} install


      - name: Deploy Maven Artifacts
        env:
          VERSION: ${{ steps.buildvars.outputs.version }}
        run: mvn --batch-mode -T 1C -Drevision=${{ env.VERSION }} -DskipTests deploy``

I made a series of changes to how we define the repositories, the above being what it is right now, and I did get some builds to pass. While I was doing that I combined the above two step into a single workflow step that just runs the deploy goal. When I got the passing builds, I tried to move the deploy goal back out to its own step. The build failed again, at the deploy step due to a 401 Unauthorized. No changes to the repositories were made. I immediately reversed the change and the build failed again for the same reason.

Any help understanding the issue is appreciated.

I submitted an issue for this.