How to handle GitHub search API rate limits

Question

I'm building an automated script to do some recurring search using GH search API. However I hit the "Secondary" rate limit sometimes. For referece I'm using github3.py library.

So my workaround so far is to check the rate limit before each search request like this:

for term in SEARCH_TERMS:
    self.check_rate_limit()
    result = gh.search_code(term)

    while True:
        try:
            self.check_rate_limit()  # it checks gh.rate_limit() if ["resources"]["search"]["remaining"] is lower (but it seems it's not being hit)
            item = next(result)

            # process...

            time.sleep(2)  # This helps a bit but I still hit the secondary rate limit
        except ForbiddenError:
            logging.error("(Secondary) Rate limit reached.")
            time.sleep(120)  # sleep 2 minutes but it continues to next iteration
        except StopIteration:
            break

But it never reaches the sleep condition. Instead, after the search query and while iterating the generator results, sometimes I get a github3.exceptions.ForbiddenError because of the secondary rate limit. The probability of raising that error decreases when I force a time.sleep(2) on each result iteration, but that doesn't seem right to me. I tried to understand more about the secondary rate limit but I couldn't find a clear explanation in the documentation. Am I missing something?

Hi, github3.py author here and familiar with rate limit issues with GitHub's API. Can you share more code? What you have here isn't enough to help — Ian Stapleton Cordasco, Jul 16 '22 at 16:26

score -1 · Answer 1 · answered Jul 18 '22 at 14:26

In GitHub docs it is defined as "Rate Limit" and is stated there:

"The Search API has a custom rate limit. For requests using Basic Authentication, OAuth, or client ID and secret, you can make up to 30 requests per minute. For unauthenticated requests, the rate limit allows you to make up to 10 requests per minute." (https://docs.github.com/en/rest/search)

So having some sleep time between the searches does seem reasonable here.

How to handle GitHub search API rate limits

1 Answers1