How to make only "n" API calls until processed by the server

Question

I have two large lists of same size. I am making subprocess calls with ith element of both lists as arguments:

for index in range(len(skuList)):
    sku = skuList[index]
    qty = qtyList[index]
    cmd1 = 'php -f ReviseItem.php %s %s' % (sku,qty)
    print cmd1
    args1 = shlex.split(cmd1)
    p=subprocess.Popen(args1)
    p.wait()

I am actually making an API call in the ReviseItem script. But, the API server is such that I can make a maximum of 15 requests and throttled for next 2 minutes. How can I modify my code to satisfy this. Is there a way to stop the execution for 2 minutes after 15 subprocesses are spawned?

EDIT: I am still getting throttled after waiting for 2 minutes after 15 request. It seems that I'm throttled until 15 requests are processed by the server, as mentioned here and not after making the requests.

EDIT2: if I get throttled and the request does not get processed, I get the following message (among others) in the standard output:

<ErrorResponse xmlns="http://mws.amazonaws.com/doc/2009-01-01/">
  <Error>
    <Type></Type>
    <Code>RequestThrottled</Code>
    <Message>Request is throttled</Message>
  </Error>
  <RequestID>3ca74b8b-cdc2-47df-b48b-043eb5411cda</RequestID>
</ErrorResponse>

Also, I can make another API call to find out the request ID of the last processed request. Again, Request ID is in the standard output of the calls(both the original and this call) among other messages

is it a sliding windows of the last 2 minutes or is it at 2 minute intervals it dumps any record of you making requests. — cmd, Feb 27 '14 at 20:20
@cmd it dumps my request for 2 minutes if the server has more than 15 requests pending for processing — nish, Feb 27 '14 at 20:23
I can tell by looking at the standard output. If I get a ` RequestThrottled Request is throttled ` in the output, it means it has not been processed — nish, Feb 27 '14 at 20:26
@cmd Although, there are lot of other messages in the output as well, apart from this error message — nish, Feb 27 '14 at 20:27
so you dont know when a previously sent message is done processing? — cmd, Feb 27 '14 at 20:27
@cmd: I can make another API call to find out the request ID of the last processed request. Again, Request ID is in the standard output of the calls(both the original and this call) among other messages — nish, Feb 27 '14 at 20:33
Have you read [this](http://stackoverflow.com/questions/7584507/requestthrottling-issue-in-amazon-mws-api?lq=1)? There is a link in the answer that might save you a lot of grief. — Bi Rico, Feb 27 '14 at 20:53
@BiRico Saw it, but did not pay attention to the link. Thanks for the help — nish, Feb 27 '14 at 21:08

score 1 · Answer 1 · answered Feb 27 '14 at 19:48

Sure -- just keep a list of timestamps from the last 15 times you spawned a subprocess. Before spawning a new subprocess, if there's 15 timestamps in the list, check whether the oldest timestamp is at least two minutes old; if it isn't, wait for the appropriate amount of time until it is. Then add the new timestamp, and pop off the oldest one if there's more than 15.

score 1 · Answer 2 · answered Feb 27 '14 at 19:48

Use a global variable which increments and checks if it reached the count of 15. use os.sleep or sys.sleep (whichever library is applicable in python). A pseudo code would be:

myCount = 0
for index <loop>
  myCounter++
  if myCounter % 15 == 0:
    os.sleep( 2 * 60 ) # or 2000 * 60

Finding the libraries, I see that it is time.sleep(). The code would become:

import time
myCounter = 0
for index in range(len(skuList)):
    myCounter += 1
    if myCounter % 15 == 0:  time.sleep( 2 * 60 )
    sku = skuList[index]
    qty = qtyList[index]
    cmd1 = 'php -f ReviseItem.php %s %s' % (sku,qty)
    print cmd1
    args1 = shlex.split(cmd1)
    p=subprocess.Popen(args1)
    p.wait()

That'll satisfy the requirements, but it'll wait longer than necessary if some of the calls take a nontrivial amount of time to complete. — Sneftel, Feb 27 '14 at 20:00
the script stops. But for a lot lesser time than 2 minutes. more like 10-15 seconds — nish, Feb 27 '14 at 20:03

cmd · Answer 3 · 2014-02-27T20:43:33.667

0

Assuming they process them in order. You can keep a list of outstanding requests ids, then when you are just short of full, stop and make calls to get the last processed id until you can remove any. Remove those ids and continue.

edited Feb 27 '14 at 20:43

answered Feb 27 '14 at 20:06

cmd

5,754
16
30

How to make only "n" API calls until processed by the server

3 Answers3