Raw.githubusercontent.com rate limit

I need to query a file across multiple repositories regularly. raw.githubusercontent.com seems to have rate limit rules, but not covered by docs. Can I just pass Authorization: Bearer headers to it like the api.github.com?

2 Likes

Unauthenticated requests are limited to 60 requests per hour. Authenticated requests are limited at 5000 requests an hour.

If your OAuth application needs to make unauthenticated calls with a higher rate limit, you can pass your app’s client ID and secret before the endpoint route.

I think GitHub “Pro” users get 15000, but I cannot verify that.

The problem is that I don’t know if the raw endpoint is an API call or not. As I said it’s not covered by the documentation, unlike the api endpoint or the GQL.

# check the resource.core
curl -H "Authorization: Bearer $personal_token" https://api.github.com/rate_limit

# download a file via raw endpoint
curl -H "Authorization: Bearer $personal_token" https://raw.githubusercontent.com/favoyang/unity-addressable-importer/master/Media%7E/icon-128.png -o icon.png

# check the resource.core again, it's not changed
curl -H "Authorization: Bearer $personal_token" https://api.github.com/rate_limit

The test shows the resource.core.used field is not changed. I guess the raw endpoint (could be a CDN/Edge server) may have different rate limit rules. I tested with a personal token.

BTW, the GitLab says it defaults to 300reqs per minute for the raw endpoint. https://docs.gitlab.com/ee/user/admin_area/settings/rate_limits_on_raw_endpoints.html

Even if it wasn’t an “endpoint”, it still has ratelimits. Otherwise, I could send tons of requests and cause problems.

Maybe 300 requests is for GitHub too, but GitHub and GitLab are unrelated.

Sure, just want to know the rules here. Anyway as a workaround I can also use the content API of the rest API endpoint.

I think this falls under user-to-server requests, which are contained within the 5000/hour rate limit per authenticated user.

Relevant documentation:

Normal user-to-server rate limits

User-to-server requests are rate limited at 5,000 requests per hour and per authenticated user. All OAuth applications authorized by that user, personal access tokens owned by that user, and requests authenticated with that user’s username and password share the same quota of 5,000 requests per hour for that user.

1 Like

Thanks for the link.

Then based on my test above (using a personal token), I guess your CDN/Edge server just doesn’t talk to the authenticated rate-limiting service. It just actually limited by IP.

1 Like

Hmm this week I will try find some time to dig a bit deeper and confirm a few things I am not 100% sure on, if not hope this helps.

Here is what I do know about github api rate limits though there might be differences with the endpoint that I am aware of:

  • there are authenticated rate limits, this is the 5k that was mentioned. This is not per IP, you can have multiple users behind a NAT without issue. I imagine its likely associated with a session or user table on their backend, though I am not sure they would ever tell me. We could do some inspection while creating a new session (by logging out and in) and make the same automated tests comparing the results. I have had to do some large github automation before and have had to load balance my requests through multiple users.
  • there are unauthenticated rate limits and this at the ip level. If you are an authenticated user you can still make your normal requests as long as you stay in budget.
1 Like