4

The Amazon API limit is apparently 1 req per second or 3600 per hour. So I implemented it like so:

while True:
    #sql stuff
    time.sleep(1)
    result = api.item_lookup(row[0], ResponseGroup='Images,ItemAttributes,Offers,OfferSummary', IdType='EAN', SearchIndex='All')
    #sql stuff

Error:

amazonproduct.errors.TooManyRequests: RequestThrottled: AWS Access Key ID: ACCESS_KEY_REDACTED. You are submitting requests too quickly. Please retry your requests at a slower rate.

Any ideas why?

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
User
  • 23,729
  • 38
  • 124
  • 207
  • Amazon appears to have about a hundred billion different limits, of which the 1/sec variant is just one. Are you sure you're looking at the right limit? – paxdiablo Sep 24 '16 at 13:58
  • Are you certain that this is the only request you are making? Or is there a chance you are making a request further down the line? – Nick is tired Oct 22 '16 at 14:58
  • Where does this code live? In a lambda? On an EC2 instance? Somewhere outside of AWS? – kevin628 Nov 21 '16 at 02:12
  • I'd start looking at your usage report in EC2 see what it says is happening. (I'd also implement a smarter rate-limiting approach -- token bucketing or such -- but in this particular scenario that would prevent you from underutilizing your limit due to delay outside the sleep, not overutilizing it). – Charles Duffy Nov 21 '16 at 02:28

2 Answers2

1

This code looks correct, and it looks like 1 request/second limit is still actual: http://docs.aws.amazon.com/AWSECommerceService/latest/DG/TroubleshootingApplications.html#efficiency-guidelines

You want to make sure that no other process is using the same associate account. Depending on where and how you run the code, there may be an old version of the VM, or another instance of your application running, or maybe there is a version on the cloud and other one on your laptop, or if you are using a threaded web server, there may be multiple threads all running the same code.

If you still hit the query limit, you just want to retry, possibly with the TCP-like "additive increase/multiplicative decrease" back-off. You start by setting extra_delay = 0. When request fails, you set extra_delay += 1 and sleep(1 + extra_delay), then retry. When it finally succeeds, set extra_delay = extra_delay * 0.9.

theamk
  • 1,420
  • 7
  • 14
0

Computer time is funny

This post is correct in saying "it varies in a non-deterministic manner" (https://stackoverflow.com/a/1133888/5044893). Depending on a whole host of factors, the time measured by a processor can be quite unreliable.

This is compounded by the fact that Amazon's API has a different clock than your program does. They are certainly not in-sync, and there's likely some overlap between their "1 second" time measurement and your program's. It's likely that Amazon tries to average out this inconsistency, and they probably also allow a small bit of error, maybe +/- 5%. Even so, the discrepancy between your clock and theirs is probably triggering the ACCESS_KEY_REDACTED signal.

Give yourself some buffer

Here are some thoughts to consider.

Do you really need to hit the Amazon API every single second? Would your program work with a 5 second interval? Even a 2-second interval is 200% less likely to trigger a lockout. Also, Amazon may be charging you for every service call, so spacing them out could save you money.

This is really a question of "optimization" now. If you use a constant variable to control your API call rate (say, SLEEP = 2), then you can adjust that rate easily. Fiddle with it, increase and decrease it, and see how your program performs.

Push, not pull

Sometimes, hitting an API every second means that you're polling for new data. Polling is notoriously wasteful, which is why Amazon API has a rate-limit.

Instead, could you switch to a queue-based approach? Amazon SQS can fire off events to your programs. This is especially easy if you host them with Amazon Lambda.

Community
  • 1
  • 1
John Anderson
  • 484
  • 4
  • 7