Retry-After header missing?

If you hit the GitHub API with a lot of requests at once to make PRs etc, you may run in to the API Abuse rate limits.

The documentation tells us that:

When you have been limited, use the Retry-After response header to slow down. The value of the Retry-After header will always be an integer, representing the number of seconds you should wait before making requests again. For example, Retry-After: 30 means you should wait 30 seconds before sending more requests.

However, when I run in to these problems I never seem to have that header set:

{
	"message": "You have triggered an abuse detection mechanism and have been temporarily blocked from content creation. Please retry your request again later.",
	"documentation_url": "https://docs.github.com/overview/resources-in-the-rest-api#abuse-rate-limits"
}
{
	'Server': 'GitHub.com',
	'Date': 'Tue, 11 May 2021 11:10:25 GMT',
	'Content-Type': 'application/json; charset=utf-8',
	'Transfer-Encoding': 'chunked',
	'X-OAuth-Scopes': 'repo, workflow',
	'X-Accepted-OAuth-Scopes': '',
	'X-GitHub-Media-Type': 'github.v3; format=json',
	'X-RateLimit-Limit': '5000',
	'X-RateLimit-Remaining': '4628',
	'X-RateLimit-Reset': '1620734914',
	'X-RateLimit-Used': '372',
	'X-RateLimit-Resource': 'core',
	'Access-Control-Expose-Headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset',
	'Access-Control-Allow-Origin': '*',
	'Strict-Transport-Security': 'max-age=31536000; includeSubdomains; preload',
	'X-Frame-Options': 'deny',
	'X-Content-Type-Options': 'nosniff',
	'X-XSS-Protection': '0',
	'Referrer-Policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
	'Content-Security-Policy': "default-src 'none'",
	'Vary': 'Accept-Encoding, Accept, X-Requested-With',
	'Content-Encoding': 'gzip',
	'X-GitHub-Request-Id': '0780:11AC:3AA0BD0:6B7151F:609A6620'
}

The Access-Control-Expose-Headers lists Retry-After as an allowed header, but the Retry-After header itself is not there.

I’m calling the API using the requests library in Python, code here.

Am I missing something at my end, or is this a problem with the GitHub API?

Phil

Hi there @ewels :wave:

Thanks so much for joining us and for sharing your experience. There is additional documentation that speaks to the lack of Retry-After header found here:

Specifically:

  • Requests that create content which triggers notifications, such as issues, comments and pull requests, may be further limited and will not include a Retry-After header in the response. Please create this content at a reasonable pace to avoid further limiting.

The behavior you described in your OP:

If you hit the GitHub API with a lot of requests at once to make PRs etc

…is precisely what we try to protect against. As this is an abuse protection measure, it is based on multiple input variables, and there isn’t an X value of seconds to suggest.

The process to resolve this would be to decrease the frequency of the scripts calls dramatically, and increase as needed. It looks like the code already has logic to identify an updated frequency value, when that header is present; awesome!

Just gotta slow down to start, and everything should behave as you’re hoping.

1 Like

Amazing, thank you for the comprehensive answer @nethgato! I’m not quite sure how I missed that, seeing as the bullet is the one below the one I pasted in my original post :man_facepalming: I think I must have missed the critical word not (include…).

Anyway, that makes sense and as you say I added a backup approach if the header isn’t there. The context is a sync that runs once every few months but does various repos in parallel so submits 40 odd PRs all at once. How synchronous that is depends on how fast GitHub Actions runs the jobs so the abuse mechanism varies a lot. In the last run it was just one PR, in the one before it was about half. Anyway, my random delay seemed to work on the most recent run so I’m happy :+1:

The main thing was that I thought I was going crazy / doing something wrong! So thank you for putting my mind at rest :slight_smile:

1 Like

So glad to help! You’re definitely not going crazy and I’m happy I was able to help verify sanity.

:bow:

2 Likes