Docker.pkg.github.com intermittently returns 500 Internal Server Error on push

We occasionally receive the following errors when pushing multiple tags as part of a GitHub Actions workflow:
Firstly, we successfully push the first tag, which only has three layers that actually need to be pushed, and then immediately start pushing the next one - which should be really fast because all layers exist.
However, it fails with an unspecific error unexpected HTTP status: 500 Internal Server Error:

The push refers to repository [docker.pkg.github.com/camunda/XXXXX/YYYYYY]
46a46747d59: Preparing
f43dcee283e0: Preparing
4abf25ab291f: Preparing
69fbee19464e: Preparing
7c7455e7f503: Preparing
264dfe42a385: Preparing
777b2c648970: Preparing
264dfe42a385: Waiting
777b2c648970: Waiting
7c7455e7f503: Layer already exists
69fbee19464e: Layer already exists
264dfe42a385: Layer already exists
777b2c648970: Layer already exists
a46a46747d59: Pushed
f43dcee283e0: Pushed
4abf25ab291f: Pushed
fefaa1c09f3e955ae10229a7793e4fa5d3f9aaa2: digest: sha256:8008ea567c517980cc4ccdc7d254bcaac8a7100df38f5bdf7f244bf1c3349b79 size: 1784
The push refers to repository [docker.pkg.github.com/camunda/XXXXX/YYYYY:${COMMIT_SHA}]
a46a46747d59: Preparing
f43dcee283e0: Preparing
4abf25ab291f: Preparing
69fbee19464e: Preparing
7c7455e7f503: Preparing
264dfe42a385: Preparing
777b2c648970: Preparing
264dfe42a385: Waiting
777b2c648970: Waiting
4abf25ab291f: Layer already exists
69fbee19464e: Layer already exists
7c7455e7f503: Layer already exists
a46a46747d59: Layer already exists
f43dcee283e0: Layer already exists
264dfe42a385: Layer already exists
777b2c648970: Layer already exists
received unexpected HTTP status: 500 Internal Server Error
Error: Process completed with exit code 123.

This affects ~20 builds per day which is a bit too much imho.
We don’t yet clean up any of the produced tags in the docker registry; could this be related?

1 Like

Hi @lwille,

Thanks for reporting this. I have seen a similar error, but we weren’t able to reproduce it consistently. I’ll contact the GitHub Packages team and let you know what they find.

1 Like

Hi we have the same problem


The push refers to repository [docker.pkg.github.com/ydyle]
d1aad3ca70a3: Preparing
06e4961994dc: Preparing
27031ae57027: Preparing
faa05be6bd95: Preparing
1bf64614fbf0: Preparing
1580f513f55e: Preparing
97ee0f574dc3: Preparing
724ab166c33e: Preparing
0ab233912ea8: Preparing
4073cf0cd094: Preparing The push refers to repository [docker.pkg.github.com/ydyle]
d1aad3ca70a3: Preparing
68
06e4961994dc: Preparing
69
27031ae57027: Preparing
70
faa05be6bd95: Preparing
71
1bf64614fbf0: Preparing
72
1580f513f55e: Preparing
73
97ee0f574dc3: Preparing
74
724ab166c33e: Preparing
75
0ab233912ea8: Preparing
76
4073cf0cd094: Preparing
77
9f21278068c9: Preparing
78
0883fdc029c7: Preparing
79
8ba0b5fd2af5: Preparing
80
1baa605dde29: Preparing
81
5113fddaede5: Preparing
82
9ab15bec4b06: Preparing
83
45bbfac6f026: Preparing
84
95d1849410ed: Preparing
85
5d53caf19040: Preparing
86
cb381a32b229: Preparing
87
1580f513f55e: Waiting
88
97ee0f574dc3: Waiting
89
724ab166c33e: Waiting
90
0ab233912ea8: Waiting
91
4073cf0cd094: Waiting
92
9f21278068c9: Waiting
93
0883fdc029c7: Waiting
94
8ba0b5fd2af5: Waiting
95
1baa605dde29: Waiting
96
5113fddaede5: Waiting
97
9ab15bec4b06: Waiting
98
45bbfac6f026: Waiting
99
95d1849410ed: Waiting
100
5d53caf19040: Waiting
101
cb381a32b229: Waiting
102
06e4961994dc: Layer already exists
103
d1aad3ca70a3: Layer already exists
104
27031ae57027: Layer already exists
105
1bf64614fbf0: Layer already exists
106
faa05be6bd95: Layer already exists
107
1580f513f55e: Layer already exists
108
97ee0f574dc3: Layer already exists
109
724ab166c33e: Layer already exists
110
0ab233912ea8: Layer already exists
111
4073cf0cd094: Layer already exists
112
9f21278068c9: Layer already exists
113
0883fdc029c7: Layer already exists
114
8ba0b5fd2af5: Layer already exist
1baa605dde29: Layer already existsM
5113fddaede5: Layer already exist
45bbfac6f026: Layer already existes
95d1849410ed: Layer already exists
9ab15bec4b06: Layer already exists
5d53caf19040: Layer already exists
cb381a32b229: Layer already exists
received unexpected HTTP status: 500 Internal Server Error
9f21278068c9: Preparing
0883fdc029c7: Preparing
8ba0b5fd2af5: Preparing
1baa605dde29: Preparing
5113fddaede5: Preparing
9ab15bec4b06: Preparing
45bbfac6f026: Preparing
95d1849410ed: Preparing
5d53caf19040: Preparing
cb381a32b229: Preparing
1580f513f55e: Waiting
97ee0f574dc3: Waiting
724ab166c33e: Waiting
0ab233912ea8: Waiting
4073cf0cd094: Waiting
9f21278068c9: Waiting
0883fdc029c7: Waiting
8ba0b5fd2af5: Waiting
1baa605dde29: Waiting
5113fddaede5: Waiting
9ab15bec4b06: Waiting
45bbfac6f026: Waiting
95d1849410ed: Waiting
5d53caf19040: Waiting
cb381a32b229: Waiting
06e4961994dc: Layer already exists
d1aad3ca70a3: Layer already exists
27031ae57027: Layer already exists
1bf64614fbf0: Layer already exists
faa05be6bd95: Layer already exists
1580f513f55e: Layer already exists
97ee0f574dc3: Layer already exists
724ab166c33e: Layer already exists
0ab233912ea8: Layer already exists
4073cf0cd094: Layer already exists
9f21278068c9: Layer already exists
0883fdc029c7: Layer already exists
8ba0b5fd2af5: Layer already exists
1baa605dde29: Layer already exists
5113fddaede5: Layer already exists
45bbfac6f026: Layer already exists
95d1849410ed: Layer already exists
9ab15bec4b06: Layer already exists
5d53caf19040: Layer already exists
cb381a32b229: Layer already exists
received unexpected HTTP status: 500 Internal Server Error

@rubencherbit,

Thanks, I’ve added this to the internal issue.

How consistently do you see this error? Do you think it’s always happening on the same layer?

Very often now it’s at each build and it’s very awful… It’s random

2 Likes

Up on this topic… Now, I can’t push any images on my repo…

1 Like

I try something and it’s working, I try to create a new repo and push my image and it’s working…

This is a long shot. Were these builds triggered by depandabot?

@lwille,

I think I know what the problem is. Please could you open a ticket about this here:
https://support.github.com/contact?form[subject]=Re:%20GitHub%20Packages

Thanks!

Hi y’all, I am faced with probably the same issue, are there any news about this, any insights to share?

Thank you

@jcansdale and @lwille