Elegant way for breaking a Generator Loop: GeneratorExit error

Question

I have a code looping on a generator. I have to break that loop after the second iteration if it reaches it. To do so I use break, which raises a GeneratorExit when it calls the Generator.close() method.

for page in limit_handled(tweepy.Cursor(..., ..., ...):
    while len(the_list) < 400:
        for status in page:

            def process_status(tweet):
                ...
                ...

            the_list.append(process_status(status))

    break

Would there be a more elegant way which would avoid such an error?

Exception ignored in: <generator object limit_handled at 0x000000003AB300A0>
RuntimeError: generator ignored GeneratorExit

I have seen answers to these two questions : How to take the first N... How to get the n next... but this is not the same issue. In my case, the Generator uses a Cursor. Hence, at each iteration it processes a query. I want to stop it querying once at least 400 statuses have been reached, which can happen after the second or the third iteration (a query generally return 200 rows, but it can also be less). Slicing the generator is not an option here. Avoiding to process all the queries (16 total, for approximately 16*200=3200 statuses) is exactly what I want to avoid by Breaking the code after 400 statuses are returned.

Edit: For a better understanding, here is the code for my generator:

def limit_handled(cursor):
    global user_timeline_remaining
    while True:
        if user_timeline_remaining>1:
            try:
                yield cursor.next()
            except BaseException as e:
                print('failed_on_CURSOR_NEXT', str(e))
        else:
            time.sleep(5*60)
                try:
                    data = api.rate_limit_status()
                except BaseException as f:
                    print('failed_on_LIMIT_STATUS', str(f))
                user_timeline_remaining = data['remaining_queries']

Why not just handle the exception in the generator then? `try: ... except GeneratorExit: pass`. — Martijn Pieters, Oct 03 '17 at 10:18
And the `close()` is not (directly) called by the `for` loop, it is called when there are no more references left to the generator object (given that the `for` loop is the only reference, when the loop ends the reference is dropped and the generator is deleted). — Martijn Pieters, Oct 03 '17 at 10:19
Sorry I thought that was clear that it was raised when the break was called (implying that there was generator iterations left) — ylnor, Oct 03 '17 at 10:21
Yes, the `break` ends the `for` loop. The `for` loop ending decrements the reference count to the result of `limit_handled(...)`. Because that was the only reference, the generator `__del__` method is called to clean it up, which calls `generator.close()`. — Martijn Pieters, Oct 03 '17 at 10:26
Gotcha. Anyways, I am not sure what you mean by your first answer (using `try:... except GeneratorExit : pass `), could you explain it a bit more please? — ylnor, Oct 03 '17 at 10:50
You haven't shared your `limit_handled()` generator code, but the exception is raised *in the generator function*. Put the `try: ... except` handler in that function. — Martijn Pieters, Oct 03 '17 at 10:52
I have updated my questions with the generator code, thank you Martjin. — ylnor, Oct 03 '17 at 12:16
Ah! You are swallowing `GeneratorExit` exceptions, so your generator **can't be closed**! Why are you catching `BaseException`? — Martijn Pieters, Oct 03 '17 at 12:27

score 4 · Accepted Answer · answered Oct 03 '17 at 12:31

Your generator ignores the GeneratorExit exception that generator.close() raises. By catching BaseException, you effectively made closing impossible, so your generator will instead yield another value (as the code continues after the exception handler back to the top of the loop until yield is reached again). This is why you see your exception:

If the generator yields a value, a RuntimeError is raised.

You really should not catch BaseException in your code. Catch specific exceptions or, at best, Exception:

except Exception a e:
    # ...

That way you don't catch GeneratorExit (a subclass of BaseException), SystemExit or KeyboardInterrupt.

If you feel your code must catch SystemExit and KeyboardInterrupt at this point, at least add a except GeneratorExit: return before your except BaseException as e: handler.

Elegant way for breaking a Generator Loop: GeneratorExit error

1 Answers1