How to get more than 1000 search results with API Github

Question

I want to list the most star Github repo that were created in the last 30 days, but to display more than 1000 search, I have this error message

{ "message": "Only the first 1000 search results are available", "documentation_url": "https://developer.github.com/v3/search/" }

do you have a solution for this problem?

thank you in advance

score 2 · Answer 1 · edited Mar 18 '23 at 00:14

zifan is right. You can create a query per day for the last 30 days; or two queries per day (one each 12 hours); and so forth. The lower the interval, the more the query calls. At the same time, the more the repositories you catch.

Below an example in Python. It runs a curl call, so you can easily translate it to different languages.

import requests
from datetime import datetime, timedelta

URL = 'https://api.github.com/search/repositories?q=is:public created:{}..{}'
HEADERS = {'Authorization': 'token <PASTE_HERE_GITHUB_ACCESS_TOKEN>'}

since = datetime.today() - timedelta(days=30)  # Since 30 days ago
until = since + timedelta(days=1)   # Until 29 days ago 

while until < datetime.today():
    day_url = URL.format(since.strftime('%Y-%m-%d'), until.strftime('%Y-%m-%d'))
    r = requests.get(day_url, headers=HEADERS)
    print(f'Repositories created between {since} and {until}: {r.json().get("total_count")}')

    # Update dates for the next search
    since = until
    until = since + timedelta(days=1)

Of course, the number of repositories might still be too large. In that case, try

to use pagination;
to reduce the interval SINCE..UNTIL, as well as the timedelta;
to add further filters in the query, for example: exclude archived and forked repositories, get repositories with a minimum number of stars only, and so forth.

Take a look here for an example. Here is a Python tool to collect repositories from Github: https://github.com/radon-h2020/radon-repositories-collector

I tried running your snippet but it returns None for each day. — Sarvesh Bhatnagar, Sep 10 '21 at 13:49
Thank you for mentioning this. Indeed, the api endpoint was wrong (search/repository instead of search/repositories). I edited the script and tested it. — s.dallapalma, Sep 13 '21 at 11:59
@SarveshBhatnagar The code works as of writing. Perhaps you had a problem because you were unauthenticated? — rjpj1998, Mar 15 '23 at 18:03

score 0 · Answer 2 · answered Jul 04 '20 at 19:25

0

as per git hub API v3 documentation, the GitHub Search API provides up to 1,000 results for each search. https://developer.github.com/v3/search/

answered Jul 04 '20 at 19:25

abRam

67
9

score 0 · Answer 3 · answered Aug 07 '20 at 03:18

0

Run into the same issue. I think the only way work around is to split your query into small query which returns less than 1k result. (refer to github search limit results)

answered Aug 07 '20 at 03:18

zifan yan

125
2
11

How to get more than 1000 search results with API Github

3 Answers3