27

In my utility.py I have,

@contextmanager
def rate_limit_protection(max_tries=3, wait=300):
    tries = 0
    while max_tries > tries:
        try:
            yield
            break
        except FacebookRequestError as e:
            pprint.pprint(e)
            if e._body['error']['message'] == '(#17) User request limit reached':
                print("waiting...")
                time.sleep(wait)
                tries += 1

In my task.py I call:

for date in interval:
   with utility.rate_limit_protection():
      stats = account.get_insights(params=params)

After runing the task for a given date range, once Facebook rate limit kicks in, the program waits for 300 seconds after which it fails with the error.

File "/Users/kamal/.pyenv/versions/3.4.0/lib/python3.4/contextlib.py", line 78, in __exit__
    raise RuntimeError("generator didn't stop")
RuntimeError: generator didn't stop
kamal patwa
  • 429
  • 1
  • 4
  • 9
  • Are you expecting this `rate_limit_protection` context manager to run the body of the `with` statement repeatedly? `with` doesn't do that. The body runs once. – user2357112 Jan 13 '16 at 19:16
  • 1
    No no.. There is a for loop surrounding the with statement. the with statement runs once for the entire body of the for loop. Do you mean I can't have a while loop inside my rate_limit_protection() function – kamal patwa Jan 14 '16 at 05:14
  • Why would you expect the with statement to run once for the entire loop, if it's inside the loop body? – user2357112 Jan 14 '16 at 05:55
  • `for date in interval: with utility.rate_limit_protection(): stats = account.get_insights(params=params)` I have edited the question. Now you might get some clarity on what I am trying to do – kamal patwa Jan 14 '16 at 06:12
  • This is pretty much the same as http://stackoverflow.com/q/29708445/245173 – jpkotta Mar 17 '16 at 22:36

1 Answers1

43

The with statement is not a looping construct. It cannot be used to execute code repeatedly. A context manager created with @contextmanager should only yield once.

A context manager does (basically) three things:

  1. It runs some code before a code block.
  2. It runs some code after a code block.
  3. Optionally, it suppresses exceptions raised within a code block.

If you want to do something like this, you need to rewrite it so that the loop is moved outside the context manager, or so that there is no context manager at all.

One option would be to write a function that accepts a callback as an argument, and then calls the callback in a loop like the one you currently have in your context manager:

def do_rate_protection(callback, max_tries=3):
    tries = 0
    while max_tries > tries:
        try:
            callback()
            break
        except FacebookRequestError as e:
            # etc.

You can then call it like this:

for date in interval:
    def callback():
        # code
    do_rate_protection(callback)

If the callback doesn't need the date variable, you can move it outside the loop to avoid repeatedly recreating the same function (which is wasteful of resources). You could also make date a parameter of the callback() function and pass it using functools.partial.

Kevin
  • 28,963
  • 9
  • 62
  • 81
  • If the `#code` here is replaced by `stats = account.get_insights(params=params)`, `stats` will be defined inside `callback( )`, so it won't make bring the change to the outer scope unless you return or make it global? – Y. Zhou Dec 11 '21 at 12:09
  • To add to this, I had this error raised because I was `yield`ing within the `try` and _also_ yielding inside a `finally` statement -> when no exception was raised, we would `yield` twice from one context manager. – Jake Flynn Jun 13 '23 at 15:30