Slow Downloads Telekom/Germany

Hi,

hope i found the most suitable place for this question:)

Since i/we changed our Internet-Provider to the Telekom ( ~2 weeks ago, we’re located in Germany too), we have A LOT of performance problems downloading anything from github. We get maximum download speed of ~100kb when downloading from those telekom connections.

There has been a lot of talk and some answers in the support forum, basically coming down to “there are no slow routes/paths to aws”, “the problem is with AWS routing”.

This has probably already been asked/answered somewhere, but i couldn’t find anything related…
Is anyone at github aware of the problems? Working on a solution or in contact with AWS?
Currently github it is completely unusable for us.

regards,
strowi

4 Likes

Hi,

I have the same problem. I did not change my provider, but I’m also using Telekom.
My actual download speed is ~450 Bytes. Totally unusable in that form.

Totally forgot, here ist the link to the (german) support forum: https://telekomhilft.telekom.de/t5/Telefonie-Internet/Amazon-AWS-S3-Github-downloads-sehr-langsam-nicht-nutzbar/td-p/4910937/page/56

So far it seems to be with the route back from AWS S3 East via cogent to telekom.

Hey folks,

Thanks for the reports and apologies for the slow download speed! I work for GitHub from my home office in Luebeck, Germany, and am on a DTAG line as well. I’m not seeing any issues on my connection and get a steady 6 MB/s which is within the realm of what’s expected for a 50 MBit VDSL line.

If you could go to https://github-debug.com/, execute the steps outlined on that page, and create a ticket with us, we can bring this up with our infrastructure and network engineers to take a look.

Cheers!
Mike

1 Like

Hey @fooforge,

thx for joining in.:wink: Not sure where the repos are hosted, but while cloning i also get a steady fast 6MB in Bonn/next to the TelekomHQ.
This seem to only relate to the release-downloads from S3 (is git-data also hosted on s3?)

Have you tried downloading a release file like “github com/aquasecurity/trivy/releases/download/v0.15.0/trivy_0.15.0_Linux-64bit deb” (sorry, can only put in 2 links per post as new user :)) ? Here i get a steady ISDN rate of 40kb/s. When i turn on VPN and route traffic through our office (which has a different provider) i get the max again.

I created a ticket, although probably not in the right place since none of the choices seemed to fit.

@strowi Happy to! :slight_smile:

I have tested this with both a (https) clone of GitHub - atom/atom: The hackable text editor and a download of the latest (deb) release at Release 1.54.0 · atom/atom · GitHub. The clone terminates on the nearest POP and then gets routed to our main data center in Virginia, the latter is hosted on S3. Both were maxing out at around 6 MB/s.

The good news is that I have found your ticket. I’ll make sure this goes to the right team. Thanks for creating that one!

1 Like

I am having a very similar issue, but this is specific to releases.

When downloading https://github.com/bell-sw/Liberica/releases/download/11.0.9.1+1/bellsoft-jre11.0.9.1+1-linux-amd64.tar.gz:

Resolving github.com (github.com)... 140.82.121.4
Connecting to github.com (github.com)|140.82.121.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/115621629/caa66000-1f09-11eb-963e-bc36b577b070?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210114%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210114T183050Z&X-Amz-Expires=300&X-Amz-Signature=1d8efbdc739cfc6a1cb534975d5567d84c4f6b85fbd5cbc6c48c67aa8be1e295&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=115621629&response-content-disposition=attachment%3B%20filename%3Dbellsoft-jre11.0.9.1%2B1-linux-amd64.tar.gz&response-content-type=application%2Foctet-stream [following]
--2021-01-14 19:30:50--  https://github-production-release-asset-2e65be.s3.amazonaws.com/115621629/caa66000-1f09-11eb-963e-bc36b577b070?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210114%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210114T183050Z&X-Amz-Expires=300&X-Amz-Signature=1d8efbdc739cfc6a1cb534975d5567d84c4f6b85fbd5cbc6c48c67aa8be1e295&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=115621629&response-content-disposition=attachment%3B%20filename%3Dbellsoft-jre11.0.9.1%2B1-linux-amd64.tar.gz&response-content-type=application%2Foctet-stream
Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.217.13.12
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.217.13.12|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 48526287 (46M) [application/octet-stream]
Saving to: ‘bellsoft-jre11.0.9.1+1-linux-amd64.tar.gz.2’

bellsoft-jre11.0.9.1+1-linux-amd64.tar.gz.2   0%[                                                                                           ]  33.55K  4.99KB/s    eta 2h 38m

While downloading archives (https://github.com/traefik/traefik/archive/v1.7.28.zip) is as fast as expected:

--2021-01-14 19:33:49--  https://github.com/traefik/traefik/archive/v1.7.28.zip
Resolving github.com (github.com)... 140.82.121.4
Connecting to github.com (github.com)|140.82.121.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/traefik/traefik/zip/v1.7.28 [following]
--2021-01-14 19:33:49--  https://codeload.github.com/traefik/traefik/zip/v1.7.28
Resolving codeload.github.com (codeload.github.com)... 140.82.121.10
Connecting to codeload.github.com (codeload.github.com)|140.82.121.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/zip]
Saving to: ‘v1.7.28.zip’

v1.7.28.zip                                     [          <=>                                                                              ]  11.92M  5.96MB/s               ^C

Speed results from github-debug:

2507998 bytes downloaded from github.com at 26.54 Mbps

                
2507998 bytes downloaded from cloud.githubusercontent.com at 12.23 Mbps

                
2507998 bytes downloaded from iad.github-debug.com at 8.29 Mbps

                
2507998 bytes downloaded from sea.github-debug.com at 11.40 Mbps

I can confirm this problem from my side. I’ve run in a similar issue using CircleCI artifact downloads hosted on Amazon S3 as well. Investigations shows if you’re on a Deutsche Telekom (AS3320) connection they have capacity issues handing over traffic to Amazon US. You can test this using a looking glass app. In my case, the only way to get to S3 services is through AS3320, and you are doomed.

I’ve solved the problem using a VPN and tunneling my internet traffic to another ISP with a different AS path to the US in my case IONOS which was smart about, handing traffic off through Telianet which doesn’t have any issues and you get reasonable speedy downloads from S3.

2 Likes

Hi!

We too had problems with downloads from https://github-production-release-asset-2e65be.s3.amazonaws.com since days. Today we could solve those by activating IPv6 in our Fritzbox.
What are your settings?

Edit: possibly only luck with a new IP at reconnect.

Any Updates? Downloading AUR-updates with under 30 kbps: packages with above only 50 mb are not downloadable to the end…

e.g.:

@fooforge

Seems like the problems are addressed and routing is adjusted, this path works for me

.

When I reported the problem the routing path looked like this and was unusable for me.

20-40% packet loss, depending on the IP.

For me this has been “temporarily” solved.
Since yesterday downloads are fast again, but we’ll see if this reappears with an ip-change.

2 Likes

Hi folks,

We have had multiple engineers from various teams look into the matter, and we have also been in touch with DTAG and AWS about the issue.

There seem to be capacity issues between AWS (US East) and DTAG which is to be resolved by those two parties.

The only advice I can give at this point is to continue to raise the issue with your service provider – which hurts as sending folks from one side to the other feels never great. We are also looking at some alternatives, but none of those are ready for prime time yet and will likely take a good amount of time to implement.

I’m sorry I don’t have better news for you at this time, but I hope this clears things up a bit at least.

Cheers,
Mike

2 Likes

Some annotations:

  • the issue only occurs on Telekom end-user-connections (“Privatkundenanschluss”)
  • the Telekom business connections appear to be unaffected (“Geschaeftskundenanschluss”)
  • the issue is worse from around 8am-23pm german time (office hours + evening), during night times the downloads are at an acceptable rate

@tobox agree on that, that’s also what I’ve noticed.
It’s maybe also worth noting that only some of the IP subnets of DTAG are affected by this. There are some subnets that are being peered directly from AWS to DTAG. The “slow” subnets are being routed from AWS over Cogent to DTAG which appears to be a very slow route.

Thx for the info @fooforge .

Reading up on the issue it seems to come down to a price-fight between DTAG and other carrieres that send much more traffic towards DTAG than they receive. DTAG wants those carriers to pay for the increased infrastructural costs.

1 Like

It’s such a shame, I though the DTAG is a “premium” provider.
And now I can’t download big files from github.
50…70kb per seconds with decreasing tendency the longer it takes! WTF.