1

So I'm trying to get data from an API which has a max call limit of 60/min. If I go over this it will return a response [429] too many requests.

I thought maybe a better way was to keep requesting the API until I get a response [200] but I am unsure how to do this.

import requests

r = requests.get("https://api.steampowered.com/IDOTA2Match_570/GetTopLiveGame/v1/?key=" + key + "&partner=0")

livematches = json.loads(r.text)['game_list']

So usually it runs but if it returns anything other than a response [200], my program will fail and anyone who uses the website won't see anything.

Nazim Kerimbekov
  • 4,712
  • 8
  • 34
  • 58
  • You might find the solution already answered in the link below. [Link](https://stackoverflow.com/questions/2659952/maximum-length-of-http-get-request) – Eshwer Aaditya Jul 18 '19 at 05:55

1 Answers1

0

Generally with cases like this, I would suggest adding a layer of cache indirection here. You definitely don't want (and can't) try solving this using frontend like you suggested since that's not what the frontend of your app is meant for. Sure, you can add the "wait" ability, but all someone has to do is pull up Chrome Developer tools to grab your API key and then call that as many times as they want. Think of it like this: say you have a chef that can only cook 10 things per hour. Someone can easily come into the restaurant, order 10 things, and then nobody else can order anything for the next hour, which isn't fair. Instead, adding a "cache" layer, which means that you only call the steam API every couple seconds. If 5 people request your site within, say, 5 seconds, you only go to the steam API on the first call, then you save that response. To the other 4 (and anyone who comes within those next few seconds), you return a "cached" version.

The two main reasons for adding a cache API layer are the following:

  1. You would stop exposing key from the frontend. You never want to expose your API key to the frontend directly like this since anyone could just take your key and run many requests and then bam, your site is now down (denial of service attack is trivial). You would instead have the users hit your custom mysite.io/getLatestData endpoint which doesn't need the key since that would be securely stored in your backend.

  2. You won't run into the rate limiting issue. Essentially if your cache only hits the API once every minute, you'll not run into any time where users can't access your site due to an API limit, since it'll return cached results to the user.

This may be a bit tricky to visualize, so here's how this works:

You write a little API in your favorite server-side language. Let's say NodeJS. There are lots of resources for learning the basics of ExpressJS. You'd write an endpoint like /getTopGame that is attached to a cache like redis. If there's a cached entry in the redis cache, return that to the user. If not, go to the steam API and get the latest data. Store that with an expiration of, say, 5 seconds. Boom, you're done!

It seems a bit daunting at first but as you said being a college student, this is a great way to learn a lot about how backend development works.

rb612
  • 5,280
  • 3
  • 30
  • 68
  • I see. So since this API only allows 60 calls / min and if each person using my site at the moment would draw between 2 and 10 calls, at the best case if 31 people used my site, it would crash. So what you suggest I do is like show an image/copy of my site instead of my actual site to the user. Let me know if my analogy is the same as yours because thats what I think you said. So the way I have it now, someone can see my API key? How can they see it. Its in my views in my Django app. –  Jul 18 '19 at 09:29