Timeout (and skip) a block of code inside a python iteration

Question

How can I skip and continue when a step of a loop takes too long?

Below are a few desirable interfaces I'd like:

for x in some_collection:
    with Timeout(2.5):  # time out after 2.5 seconds
        do_something(x)

Would be nice (and even better -- but might be harder):

with TimeoutLoopStep(2.5):  # time out after 2.5 seconds
    for x in some_collection:
        do_something(x)

Possibly an equivalent timeout_loop_step decorator to consume iterators in that manner:

timeout_loop_step(2.5)(map(do_something, some_collection))

--- Addendum (edit) to dispel some ambiguity ---

What I'm looking for is a context manager and/or decorator that will interrupt the processing of a step of an iteration if it lasts too long.

Most APIs that talk to a remote system (e.g. DBs) provide some sort of timeout parameter that means "do this, but if you exceed timeout trying, just phorgetaboutit and move on to the next item..."

But if I'm not offered such control, I'd like to achieve it, externally, with a reusable context manager or decorator.

Check out this: https://stackoverflow.com/questions/16148735/how-to-implement-a-watchdog-timer-in-python and https://stackoverflow.com/questions/13293269/how-would-i-stop-a-while-loop-after-n-amount-of-time — Tom Myddeltyn, Dec 06 '20 at 21:12
Does this answer your question? [How to implement a watchdog timer in Python?](https://stackoverflow.com/questions/16148735/how-to-implement-a-watchdog-timer-in-python) — CryptoFool, Dec 06 '20 at 21:23
@TomMyddeltyn, the second is not what I'm looking for. The first might, but I'd need to try to put it in one of the forms I mentioned in my question to see if it actually works. I'll Add more details to my question to make it clear what I'm looking for. — thorwhalen, Dec 07 '20 at 17:27
No the `Timer` solution (WatchDog) is not a solution. Timer docs say "Call a function after a specified number of seconds". In contrast what I want is a means to "interrupt the call of a function after a specified number of seconds". — thorwhalen, Dec 07 '20 at 19:40
@thorwhalenn - yes. the system put that there automatically when I marked this question as a duplicate. If I'd thought about it, I would have deleted it, knowing that Tom already listed it. Obviously not a big deal either way. — CryptoFool, Dec 07 '20 at 20:09

score 1 · Answer 1 · edited Apr 28 '21 at 02:05

I needed something similar long back. My solution was a thread based iterator decorator kind of thing. I had put it on pypi by the name iterators. For your case it would look like this.

from iterators import TimeoutIterator

m = map(do_something, some_collection)
it = TimeoutIterator(m, timeout=2.5)

for x in it:
  if x == None: # gives None if there is a timeout
    it.interrupt() # stop the underlying thread if breaking the loop
    break

Alternatively if the result of your function itself can be None then you could use something like this:

from iterators import TimeoutIterator

m = map(do_something, some_collection)
it = TimeoutIterator(m, timeout=2.5,  sentinel=object())

for x in it:
  if x == it.get_sentinel(): # gives the sentinal object if there is a timeout
    it.interrupt()  # stop the underlying thread
    break

The code may look verbose as unfortunately the idea of using a context handler to stop underlying thread never came to my mind. That would have removed the need of calling interrupt. I'll add it in a new release. For my case I needed to continue iterating till end of actual iterator, but still be able to identify whether I get a result or a timeout. So calling interrupt manually was never needed.

Checkout other options and examples in the source.

score 0 · Answer 2 · answered Dec 07 '20 at 07:36

0

Use time module:

import time

time_out = time.time()+2.5
for x in some_collection:
    if time.time() == time_out:
        break
    do_something(x)

answered Dec 07 '20 at 07:36

Huy Nguyễn

36
4

1

Sorry, I'm asking about a "lean" way to interrupt a process. What trying to do above is a pause, which you'd better do using `time.sleep(2.5)`. Also, your code will often never terminate. What you meant to say was `if time.time() >= time_out` I think. – thorwhalen Dec 07 '20 at 17:20

Timeout (and skip) a block of code inside a python iteration

2 Answers2