0

I am relying on the steam API in order for my website to work. In the start everything was working perfectly fine, however now that the site is gaining popularity I keep getting 429 error codes as the API reaches ratelimit. I am hosting the site on a EC2 instance using pm2 and nginx. Is there a way to avoid reaching these ratelimits?

I have done changes so it dosent fetch from API unless needed but the problem is still happening. Is there something I can do with nginx in order to avoid this issue? I am hosting the api on port 3005 I was thinking maybe I could set up several apis on different ports but this seems really tidious. An alternative would be to host the API on different EC2 but I was having some session problems doing it that way.

server {


        root /home/ubuntu/apps/norskins-app/client/build;

        index index.html index.htm index.nginx-debian.html;

        server_name mywebsite.com;

        location / {
                try_files $uri /index.html;
        }
        location /api {
                proxy_pass http://localhost:3005;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection 'upgrade';
                proxy_set_header Host $host;
                proxy_cache_bypass $http_upgrade;
        }

    Under here is a bunch of other stuff setup byCertbot
}
Ole Dybedokken
  • 343
  • 2
  • 15
  • What rate limit are you hitting? Who's responsible for enforcing that limit? Is this an AWS thing? If so, you probably need to adjust your hosting plan and pay for whatever limit you need. The point is that someone is enforcing that limit on you and that's where you need to go to get it raised. – jfriend00 Oct 28 '22 at 23:43
  • It is the steam API that I rely on that is blocking me. It is a external API. – Ole Dybedokken Oct 29 '22 at 14:41
  • What are the terms of service when using that API? You first need to know what you are allowed to do with it and then you can possibly develop a means of staying within those limits. Showing us the code and logic you're using also might help because there may be more efficient ways to accomplish what you're doing that don't require as many API calls. With what you have in your question so far, there's nothing we can do. As best I can tell, this doesn't have anything to do with a proxy. You're just calling the API more than you're allowed to. – jfriend00 Oct 29 '22 at 17:33
  • @jfriend00 So I have figured out that the rate limits are 10request per 30min. Now the problem is that I sometimes have more than 10 people on my site every 30min. So this will be reached regardless of how my code is setup. I have now made it so the API will not be contacted if a user have contacted it in the last 30min. But lets say 11 users use my site the IP I am hosting it on AWS will end up getting banned – Ole Dybedokken Nov 01 '22 at 18:20
  • That sounds like you need a different service plan for more access to that API. Is that something you can pay for? Or, can you get ALL the data you need in fewer requests and cache it on your server so when each new user comes in, you don't need a separate request to the API? – jfriend00 Nov 01 '22 at 19:49
  • @jfriend00 I need a seperate request for each user and it needs to update everytime. http://steamcommunity.com/inventory/76561198027016127/730/2?l=english&count=5000 here is the link that I am fetching. The 76561198027016127 is the steamid of the user. Alot of sites does this with millions of user however I don't understand how they are not being limited as I have read places that they are using proxy's however I cant quite figure out what they mean about that. – Ole Dybedokken Nov 01 '22 at 21:45
  • It appears to me [here](https://steamcommunity.com/dev/apiterms) that if you're using an APIKey and follow the various rules, you get 100,000 API calls per day. – jfriend00 Nov 01 '22 at 22:57
  • @jfriend00 They recently did a update I believe but as you can see from the link you sent it was Last updated July 2010 – Ole Dybedokken Nov 01 '22 at 23:04
  • Well, there are lots of reports of people being able to do 100,000 requests per day if you follow the rules. And, there is lots of discussion of using an APIKey which you do not appear to be doing. I'm just giving you things to look at. – jfriend00 Nov 01 '22 at 23:05
  • I appreciate the suggestions but when I found the end points for finding someone inventories(https://stackoverflow.com/questions/17393099/getting-someones-steam-inventory) they never mentioned anything about a API key. As I believe this is not necessary for the steam-inventory part. – Ole Dybedokken Nov 01 '22 at 23:12
  • You should at least consider that there may be higher rate limiting limits if you're using an APIKey. I'm leaving this discussion now and hopefully someone who knows more about the steam API can come by and help. – jfriend00 Nov 01 '22 at 23:26

1 Answers1

0

Joining this guy's question, I cannot load more than 3 inventories in a minute.

Kezzington
  • 1
  • It beacuse they have added a rate-limit. You need to setup proxies and use them. I have not been able to figure the rate-limits of the API yet. But heard some rumors it might be 10r/30mins but not sure if this is correct – Ole Dybedokken Nov 09 '22 at 14:54
  • I think it is correct. I get error 429 after fetching 3 inventories, and then after giving it a little break of like a few minutes I can fetch another 3 and so forth. If you try to go over the limit by making more requests then you will not be able to load any inventories for at least 12 hours from what I've experienced so far. Not sure if it has something to do with their new mobile update or what, but I don't really see how this restriction makes sense. There are so many trading sites etc that rely solely on other people's inventories to function. – Kezzington Nov 10 '22 at 02:14
  • These other sites are relying on proxies. So that means if one IP gets ratelimited they just use another. – Ole Dybedokken Nov 10 '22 at 08:27