9

I am building an app that fetches the issues and pull requests of over 1K github repos, like this. 

$ curl -i "https://api.github.com/repos/user/repo/issues?state=closed"

My problem is that, after the initial 60 iterations I get a rate limit error: 

{
    "message": "API rate limit exceeded for xxx.xxx.xxx.xxx. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)",
    "documentation_url": "https://developer.github.com/v3/#rate-limiting"
}

The document says I can make upto 5000 requests using Authentication Which I registered an oauth for and obtained Client ID and Client Secret tokens

https://api.github.com/repos/{repo.name}/issues?client_id=...&client_secret=...

Still the rate limit shows up only after about 60 requests.

arbi-g11324115
  • 193
  • 1
  • 2
  • 13
  • 1
    You will probably have to cache whatever data you have and be a bit more selective about when / how you fetch it. What possible reason could you have to hit this API endpoint for over 1K repositories in a loop like this? – ChrisGPT was on strike Nov 11 '15 at 18:30
  • I have updated the question and added clarity. I am saving the data to database actually if that is what you are referring to "*caching*". – arbi-g11324115 Nov 11 '15 at 18:38

1 Answers1

16

The public GitHub API requests are limited to 60 / hour / ip, like you observed. That's why you need authentication.

There are multiple ways to get authenticated when you use the GitHub APIs.

Basic authentication

Basically, you provide the username and the password.

curl -u your-username "https://api.github.com/repos/user/repo/issues?state=closed"

This will prompt you for entering the password.

If you dont want to use the password, you can use a personal token:

curl -u username:token "https://api.github.com/repos/user/repo/issues?state=closed"

Using personal access tokens

This is my favorite, but make sure you don't share the token code with others. To generate a new token, open this page, and you will create the token.

Then you can use it like this:

curl "https://api.github.com/repos/user/repo/issues?state=closed&access_token=token"

(replace the token snippet at the end of the url with your token code)

OAuth

If you want to implement authentication for other users, you should use OAuth. The docs are good in this direction.

Ionică Bizău
  • 109,027
  • 88
  • 289
  • 474