Bug - API PATCH chunk upload to ghcr.io broken

Code to Cloud GitHub Packages
#1

Chunked layer upload appears to be broken, responds with the first chunk uploaded over and over again

DEBUG org.apache.http.headers - http-outgoing-10 << HTTP/1.1 202 Accepted
DEBUG org.apache.http.headers - http-outgoing-10 << Content-Length: 0
DEBUG org.apache.http.headers - http-outgoing-10 << Content-Type: application/json
DEBUG org.apache.http.headers - http-outgoing-10 << docker-distribution-api-version: registry/2.0
DEBUG org.apache.http.headers - http-outgoing-10 << range: 0-33

For example, range on github is perpetually responding with 0-<content-length>, whereas gitlab and local docker registry respond with 0-<total length sent>.

I’ve found two cases in which chunked uploads on github do succeed; one where all of the data is uploaded in the final put and no patchs are sent or if all the data is sent in a patch, and then completed with an empty put (simple change to the for loop). Multiple patchs is where I start hitting troubles with github.

Current branch I’m testing against Comparing GoogleContainerTools:master...Sineaggi:chunked-uploads · GoogleContainerTools/jib · GitHub
The code as-is works for gitlab and a local registry (running in docker).

Note that the 34 byte chunk size is arbitrary, I’m uploading small layers to I used small chunk sizes.

Here’s an example chunk request/response

DEBUG org.apache.http.headers - http-outgoing-10 >> PATCH /v2/sineaggi/busybox/blobs/upload/cdbf883e-3ac5-44b2-a6e5-7f1395a2683f HTTP/1.1
DEBUG org.apache.http.headers - http-outgoing-10 >> Accept: 
DEBUG org.apache.http.headers - http-outgoing-10 >> Accept-Encoding: gzip
DEBUG org.apache.http.headers - http-outgoing-10 >> Authorization: Bearer -------
DEBUG org.apache.http.headers - http-outgoing-10 >> Content-Range: 49-74
DEBUG org.apache.http.headers - http-outgoing-10 >> User-Agent: jib null jib-core Google-HTTP-Java-Client/1.38.1 (gzip)
DEBUG org.apache.http.headers - http-outgoing-10 >> Content-Length: 26
DEBUG org.apache.http.headers - http-outgoing-10 >> Content-Type: application/octet-stream
DEBUG org.apache.http.headers - http-outgoing-10 >> Host: ghcr.io
DEBUG org.apache.http.headers - http-outgoing-10 >> Connection: Keep-Alive

DEBUG org.apache.http.headers - http-outgoing-10 << HTTP/1.1 202 Accepted
DEBUG org.apache.http.headers - http-outgoing-10 << Content-Length: 0
DEBUG org.apache.http.headers - http-outgoing-10 << Content-Type: application/json
DEBUG org.apache.http.headers - http-outgoing-10 << docker-distribution-api-version: registry/2.0
DEBUG org.apache.http.headers - http-outgoing-10 << docker-upload-uuid: cdbf883e-3ac5-44b2-a6e5-7f1395a2683f
DEBUG org.apache.http.headers - http-outgoing-10 << Location: /v2/sineaggi/busybox/blobs/upload/cdbf883e-3ac5-44b2-a6e5-7f1395a2683f
DEBUG org.apache.http.headers - http-outgoing-10 << range: 0-25
DEBUG org.apache.http.headers - http-outgoing-10 << Date: Mon, 02 Aug 2021 18:16:52 GMT
DEBUG org.apache.http.headers - http-outgoing-10 << X-GitHub-Request-Id: 0B20:19A0:7C33:FDEB:61083694
#2

Also it appears that checking upload progress is unsupported? Trying to use it to gauge upload progress returns

com.google.cloud.tools.jib.ChunkedTest > test FAILED
    com.google.cloud.tools.jib.registry.RegistryErrorException: Tried to push BLOB for ghcr.io/sineaggi/busybox with digest sha256:cce90f5b6cdf79da8fd2b50af1685767b660f86cb2825fbc34d49d952322a83c but failed because: other: The operation is unsupported.
        Caused by:
        com.google.cloud.tools.jib.http.ResponseException: 405 Method Not Allowed
        GET https://ghcr.io/v2/sineaggi/busybox/blobs/upload/2fa55091-ce4c-4ecf-9934-4bd44bc01bc7
        {"errors":[{"code":"UNSUPPORTED","message":"The operation is unsupported."}]}
            Caused by:
            com.google.api.client.http.HttpResponseException: 405 Method Not Allowed
            GET https://ghcr.io/v2/sineaggi/busybox/blobs/upload/2fa55091-ce4c-4ecf-9934-4bd44bc01bc7
            {"errors":[{"code":"UNSUPPORTED","message":"The operation is unsupported."}]}

when querying github. When running against a local docker registry, the response I get is

Range: 0-0
Docker-Upload-UUID: ef720427-aa56-4b76-b545-3b53b62a2ec8
Location: http://localhost:5000/v2/sineaggi/busybox/blobs/uploads/ef720427-aa56-4b76-b545-3b53b62a2ec8?_state=6KOlYSbBZwqklupoGwF9OA2Nz7KosXx6TlCkz8pg57N7Ik5hbWUiOiJzaW5lYWdnaS9idXN5Ym94IiwiVVVJRCI6ImVmNzIwNDI3LWFhNTYtNGI3Ni1iNTQ1LTNiNTNiNjJhMmVjOCIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMS0wOC0wM1QxNzo0MzoyMVoifQ%3D%3D
#3

Also I just noticed the urls are wrong? Under detail, a list of supported uris are listed.
An example location I get back from the github api is Location: /v2/sineaggi/busybox/blobs/upload/1dd11f7a-e286-43cd-b419-7836b6501b59
Instead of /v2/xxx/blobs/upload/ I would be expecting a url with /v2/xxx/blobs/uploads/
Or is blobs/upload fine?

#4

Are these issues confirmed?

#5

Bump plus additional characters

#6

Hi Sineaggi,

I am also experiencing this … too bad, I spent quite some time trying to figure out chunked uploads and it looks like I was banging my head against this issue the entire time.

I’ll try google or gitlabs implementations.

Non-chunked seemed to work well though!

#7

Can anybody confirm that this issue it being looked at?

#8

@Sineaggi, @wolfv,

I’ve had word that this has been fixed. Any chance you could try again and let me know if it’s working?