Maven deploy fails with HTTP 422 Unprocessable Entity

Hi,

Trying to mvn deploy a multi-module maven project to GHPM (snapshot, not release), but I get the following error on the first file upload:

[DEBUG] Using transporter WagonTransporter with priority -1.0 for https://maven.pkg.github.com/ttomsu/spring-cloud-gcp
[DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for https://maven.pkg.github.com/ttomsu/spring-cloud-gcp with username=ttomsu, password=***
Downloading from ttomsu-spring-gcp-new: https://maven.pkg.github.com/ttomsu/spring-cloud-gcp/org/springframework/cloud/spring-cloud-gcp/2.0.1.BUILD-SNAPSHOT/maven-metadata.xml
[DEBUG] Could not find metadata org.springframework.cloud:spring-cloud-gcp:2.0.1.BUILD-SNAPSHOT/maven-metadata.xml in ttomsu-spring-gcp-new (https://maven.pkg.github.com/ttomsu/spring-cloud-gcp)
[DEBUG] Writing tracking file /home/ttomsu/.m2/repository/org/springframework/cloud/spring-cloud-gcp/2.0.1.BUILD-SNAPSHOT/resolver-status.properties
Uploading to ttomsu-spring-gcp-new: https://maven.pkg.github.com/ttomsu/spring-cloud-gcp/org/springframework/cloud/spring-cloud-gcp/2.0.1.BUILD-SNAPSHOT/spring-cloud-gcp-2.0.1.BUILD-20200624.133316-1.pom
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 13.142 s
[INFO] Finished at: 2020-06-24T09:18:41-04:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.8.2:deploy (default-deploy) on project spring-cloud-gcp: Failed to deploy artifacts: Could not transfer artifact org.springframework.cloud:spring-cloud-gcp:pom:2.0.1.BUILD-20200624.131841-1 from/to ttomsu-spring-gcp-new (https://maven.pkg.github.com/ttomsu/spring-cloud-gcp): Failed to transfer file: https://maven.pkg.github.com/ttomsu/spring-cloud-gcp/org/springframework/cloud/spring-cloud-gcp/2.0.1.BUILD-SNAPSHOT/spring-cloud-gcp-2.0.1.BUILD-20200624.131841-1.pom. Return code is: 422, ReasonPhrase: Unprocessable Entity.
...

Here are the relevant parts of my ~/.m2/settings.xml:

<profiles>
    <profile>
      <id>github</id>
      <repositories>
        <repository>
          <id>ttomsu-spring-gcp-new</id>
          <name>GitHub ttomsu Apache Maven Packages</name>
          <url>https://maven.pkg.github.com/ttomsu/spring-cloud-gcp</url>
          <releases><enabled>false</enabled></releases>
          <snapshots><enabled>true</enabled></snapshots>
        </repository>
      </repositories>
    </profile>
  </profiles>

  <servers>
    <server>
      <id>ttomsu-spring-gcp-new</id>
      <username>ttomsu</username>
      <password><!-- snip! --></password>
    </server>
  </servers>
</settings>

and the pom.xml:

<distributionManagement>
  <snapshotRepository>
    <id>ttomsu-spring-gcp-new</id>
    <name>GitHub OWNER Apache Maven Packages</name>
    <url>https://maven.pkg.github.com/ttomsu/spring-cloud-gcp</url>
  </snapshotRepository>
</distributionManagement>

As seen above the maven-deploy-plugin is at 2.8.2, and:

$ ./mvnw --version
Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T14:33:14-04:00)
Maven home: /home/ttomsu/.m2/wrapper/dists/apache-maven-3.5.4-bin/4lcg54ki11c6mp435njk296gm5/apache-maven-3.5.4
Java version: 11.0.7, vendor: Debian, runtime: /usr/lib/jvm/java-11-openjdk-amd64
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "5.5.17-1rodete4-amd64", arch: "amd64", family: "unix"

One thing that may be causing the issue is that I’m doing this on a fork - is there anything in the GHPM logic that checks this and/or the <scm> element?

I read in Problem to upload -source.jar 422 Unprocessable Entity there @clarkbw is seeing issues with SNAPSHOTs - has this been fixed? What more detail can I provide to help with a fix?

Hi @ttomsu :wave:

One thing that may be causing the issue is that I’m doing this on a fork - is there anything in the GHPM logic that checks this and/or the <scm> element?

I’ve been investigating this, but I’m afraid it isn’t obvious what’s going wrong. I was wondering if you could recreate the error in a GitHub Actions workflow and point me to your fork?

Thanks,
Jamie.

We are seeing an identical error in a pull request. But I’m unsure how to debug this: https://github.com/graphhopper/graphhopper/runs/818066668

Or is this related to the github incidents in the morning?

I get the same error, did someone solve it?

I’m having the same problem.

Likely it just not allowed for pull requests and the error message needs to be improved? https://github.com/graphhopper/graphhopper/runs/862780351

btw: additionally the first time you review the problem at GitHub actions the error log is cut off. A hard browser refresh fixed this ugliness in my case and the full log is visible.

Hi @karussell,

I think the issue might be that you’re using a non-semver compatible version string:

Could you try changing it to something like this:

mvn -B versions:set -DnewVersion=0.0.0-g${GITHUB_SHA} -DgenerateBackupPoms=false

I know semver compatible versions shouldn’t be necessary, but GitHub Packages appears to coerce them to a semver version which can cause problems!

Hi @ttomsu,

I think the problem might be that 2.0.1.BUILD-SNAPSHOT isn’t a semver compatible version (if I’m reading the output correctly). Could you try changing your version to 2.0.1-SNAPSHOT or 2.0.1-BUILD-SNAPSHOT instead, and see if that works any better?

This seems unlikely as the master branch is pushing out non-semver versions nearly every day without problems :slight_smile: but will try your suggestion.

1 Like

It still fails with the same error: https://github.com/graphhopper/graphhopper/runs/865036852

The problem goes away when I rename the artifactId of this specific package from graphhopper-navigation to e.g. graphhopper-navigationtmp.

How can we release the maven package with artifactId graphhopper-navigation under the new repo https://github.com/graphhopper/graphhopper? Is there a way to avoid the renaming of this package? It would be fine if the whole package system under https://github.com/graphhopper/graphhopper-navigation is removed, if that would help.

If you look at this, you can see what’s going wrong:

$ curl https://maven.pkg.github.com/graphhopper/graphhopper/com/graphhopper/graphhopper-navigation/maven-metadata.xml -u token:${GITHUB_TOKEN} | xmllint --format -
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1427  100  1427    0     0   1803      0 --:--:-- --:--:-- --:--:--  1801
<?xml version="1.0"?>
<metadata>
  <groupId>com.graphhopper</groupId>
  <artifactId>graphhopper-navigation</artifactId>
  <versioning>
    <latest>14092720.0.0-e145896672c40a4fe9aeba9f5222de57</latest>
    <versions>
      <version>e34946f3394bd4a2055fd89af174dd6d0fae378b</version>
      <version>a0c7994e262a35ce01c769937c4052445fa8805c</version>
      <version>bbcee6211b4699408df5d8a2ae051cca3e197aed</version>
      <version>d551eb4e7126e586f3518b70252d1ee67f35e1e4</version>
      <version>cd5716cf062274d228a1cc040d61534778aae009</version>
      <version>bb7c8dfca4d8c8d07082e6544ac5160f0af445b8</version>
      <version>a7d7ce90de4d8ffb294f118f0b1551e72c165404</version>
      <version>aff83a15f6d2473a14d18a3bb9005edff0395ce4</version>
      <version>2.0.0-f18033a14756721d3eb84b2ec0faf679de18059</version>
      <version>3.0.0-ca3e8c403c602ead45e54fff0cb40075a051ec5</version>
      <version>8.0.0-bd15b51b29576061b672ae48401dab238b87856</version>
      <version>8.0.0-be2a853584177e9c164633fe31c7afb88aa97ce</version>
      <version>8.0.0-c2a6890442def182ea88ed387d39957452b9268</version>
      <version>12.0.0-ffccae403f92d788b69c00a5fd10f687459f5b</version>
      <version>616.0.0-cb14f8f9ff5ce756ebf149ae89ca449b518f3</version>
      <version>978.0.0-a9f025427b6defc4a61283af2411c932b5002</version>
      <version>6394.0.0-aef2bb33cf61dc168af420ee5dea3f635e52</version>
      <version>62170.0.0-f8255d88258c48d4efced9f168b2df0b788</version>
      <version>14092720.0.0-e145896672c40a4fe9aeba9f5222de57</version>
    </versions>
    <lastUpdated>20200712132017</lastUpdated>
  </versioning>
</metadata>

How can we release the maven package with artifactId graphhopper-navigation under the new repo https://github.com/graphhopper/graphhopper ? Is there a way to avoid the renaming of this package?

I think you might need to change the version to something over 14092720.0.0 for latest to change.

Could you try this:

mvn -B versions:set -DnewVersion=20000000.0.0-g${GITHUB_SHA} -DgenerateBackupPoms=false

I know this isn’t a permanent solution, but I’m interested to find out if it works!

Thanks,
Jamie.

If you look at this, you can see what’s going wrong:

What do you mean? I do not see immediately what looks wrong. (Except that we never published anything else than github_sha and that now somehow I can see number+github_sha.)

Or do you mean that we are forced to use semver for Github Actions?

Even if we would use semver, why would it be a good idea to force maintainers publishing bigger versions? What is wrong with pushing out an update of an old version?

I think you might need to change the version to something over 14092720.0.0 for latest to change.

Can you explain this? Do you mean as a temporary workaround? And why isn’t the problem reported in the build - how would I know what I need to change without contacting support next time?

I think you have a serious bug here plus a usage problem. So I just grepped our versions on repo github.com/graphhopper/graphhopper-navigation and I stumbled over commit

3ca3e8c403c602ead45e54fff0cb40075a051ec5

Now if you look into your output it seems that this commit was converted (why!?) into

3.0.0-ca3e8c403c602ead45e54fff0cb40075a051ec5

Ok. So now we know why there are these strange versions. But I still do not understand why you enforce increasing number. This is IMO a security problem.

Jamie, have tried your workaround but it does not work: https://github.com/graphhopper/graphhopper/runs/866299005

Transfer failed for https://maven.pkg.github.com/graphhopper/graphhopper/com/graphhopper/graphhopper-navigation/20000000.0.0-b8952d865c3ca99df19bbe3685d9b651bf48b184/graphhopper-navigation-20000000.0.0-b8952d865c3ca99df19bbe3685d9b651bf48b184.jar 422 Unprocessable Entity

I currently see multiple problems with this situation:

  • you should not change the version. E.g. from 3ca3e8... into 3.0.0-ca3e8.... Or 8be2a8... into 8.0.0-be2a8...
  • you should not enforce semver (or be more strict and throw an error that the current version does not comply)
  • if you enforce semver you should not enforce increasing version (I think you do not do this as the workaround does not work)
  • you should make it clear that the published artifactId is not per repository but per organisation (or even global?)

Hi @karussell,

I agree with all of these.

This is a little complicated and confusing. Packages are associated with a repository when they’re published. You need to have write/read permissions for that repository when publishing or installing a package.

This is confusing because your com.graphhopper.graphhopper-navigation package will show up in all these locations:

curl https://maven.pkg.github.com/graphhopper/graphhopper-navigation/com/graphhopper/graphhopper-navigation/maven-metadata.xml -u token:${READ_PACKAGES_TOKEN} | xmllint --format -
curl https://maven.pkg.github.com/graphhopper/graphhopper/com/graphhopper/graphhopper-navigation/maven-metadata.xml -u token:${READ_PACKAGES_TOKEN} | xmllint --format -
curl https://maven.pkg.github.com/graphhopper/*/com/graphhopper/graphhopper-navigation/maven-metadata.xml -u token:${READ_PACKAGES_TOKEN} | xmllint --format -

It is however associated with graphhopper/graphhopper-navigation, which I think is why the publish to graphhopper/graphhopper is failing.

To make it work, you’ll need to:

  1. Delete all versions associated with graphhopper/graphhopper-navigation
    https://github.com/graphhopper/graphhopper-navigation/packages/245763/versions
  2. Run the publish workflow again to publish under graphhopper/graphhopper

This is complicated by the fact that you can’t delete package versions associated with a public repository. To workaround this, you can temporarily make the graphhopper/graphhopper-navigation repository private, delete all package versions, make the repository public again.

Alternatively, you could delete the graphhopper/graphhopper-navigation repository or more it under a different org (I noticed it’s deprecated).

Thanks for your patience here! I will be raising this with the GitHub Packages team. I’d first like to get it working so we understand exactly what’s going wrong!

1 Like

First of all: thanks for your constant assistance here :slight_smile: much appreciated.

I agree with all of these.

Ok :slight_smile:

We plan also to move https://github.com/graphhopper/map-matching into https://github.com/graphhopper/graphhopper and so I would like to find a simple solution that does not require the deletion of the repository. We will deprecate the old repositories but we only plan to archive them and cannot delete them. So I’ll try via temporarily making the repo graphhopper/graphhopper-navigation private.

delete all package versions, make the repository public again

Is this possible via UI or do I need to set up some API calls somehow?

I will be raising this with the GitHub Packages team.

Thanks!

@karussell,

Your pain is my pain. :+1:

This is possible using the GitHub UI, see here:
https://docs.github.com/en/packages/publishing-and-managing-packages/deleting-a-package#deleting-a-version-of-a-private-package-on-github

Alternatively, I’ve written an app that you can use. Assuming you have Node+npm installed, you can delete your package versions like this:

npx jcansdale/gpr#package delete graphhopper/graphhopper-navigation -k ${DELETE_PACKAGES_TOKEN}

This will show you the package versions that would be deleted. If you’re confident you’re targeting the correct versions, you can do:

npx jcansdale/gpr#package delete graphhopper/graphhopper-navigation -k ${DELETE_PACKAGES_TOKEN} --force

DELETE_PACKAGES_TOKEN is a PAT with the repo, read:packages, write:packages and delete:packages scopes.

Alternatively you can use Docker, like this:

docker run jcansdale/gpr delete graphhopper/graphhopper-navigation -k ${DELETE_PACKAGES_TOKEN}

Let me know how you get on!

2 Likes

Just started to make the repo private. Unfortunately we would loose all stars and watchers, which is acceptable for this particular graphhopper/graphhopper-navigation repo but not for the other repo (graphhopper/map-matching) where we plan the same in 1 or 2 months. So the last option seems to be to move it to a different org. Is it possible to move it to a different org and then move it back to graphhopper without loosing watchers and stars?

Or could you delete these packages for us? The documentation indicates that this is possible. I can send you a confirmation from our side that this is acceptable. We likely won’t break users (except us) as we introduced Github Actions + Packages only a few weeks ago (Unfortunately too early it seems :smiley: ) and have not communicated this publicly.

And are you sure a usage of graphhopper-navigation from graphhopper/graphhopper will be really possible afterwards? Because in the documentation I read

Even if an entire package is deleted, you cannot reuse the deleted package name in any repository owned by the same account.

I will need to do some tests. I’m also not sure what will happen with the duplicate package name when you move it back. :thinking:

This usually isn’t possible. You seem to have hit every bug here, so I will argue your case to make an an exception!

This documentation is out of date. People sometimes needed to associate a package with a different repository. Moving it like this is the only way.

Are you okay temporarily turning graphhopper/graphhopper-navigation private? Getting these packages manually deleted is likely to take a while.

Be sure to revert this change before you publish!