Nestable timeout decorator? (i.e. timeout decorated func calling timeout decorated func)

Question

I need a decorator (or something that is functionally equivalent) that allows the code below to work as expected:

@timeout(1)
def outer():
    inner()

@timeout(5)
def inner():
    time.sleep(3)
    print("Should never be printed if you call outer()")

outer()
# The outer timeout is ignored and "property" finishes

The code seems pointless, but in reality, outer calls multiple functions that take uncertain amount of time, some of which have their own timeout.

I tried timeout-decorator and two SO answers here, but none works.

Everything I listed has the same outcome: the outer timeout is ignored. — akai, Mar 12 '16 at 04:48
There seems to be half a dozen timeout implementations on [pypi](http://pypi.python.org). One challenge is that signal based timeouts don't work outside of python's main thread. If timeout-decorator doesn't work for you, that may be the problem. — tdelaney, Mar 12 '16 at 04:49

Mikhail Gerasimov · Accepted Answer · 2016-03-20T08:42:28.520

4

Something like this:

def timeout(timeout, raise_exc=True):
    """
    raise_exc - if exception should be raised on timeout 
                or exception inside decorated func.
                Otherwise None will be returned.
    """
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            res = None
            exc = None
            def _run():
                nonlocal res
                nonlocal exc
                try:
                    res = func(*args, **kwargs)
                except Exception as e:
                    exc = e
            t = threading.Thread(target=_run)
            t.daemon = True
            t.start()
            t.join(timeout=timeout)
            if raise_exc and t.is_alive():
                raise TimeoutError()
            elif raise_exc and (exc is not None):
                raise exc
            else:
                return res
        return wrapper
    return decorator

Examples:

@timeout(0.5, raise_exc=False)
def outer():
    return inner()

@timeout(2)
def inner():
    time.sleep(1)
    return "Shouldn't be printed"

print(outer())  # None

and

@timeout(2, raise_exc=False)
def outer():
    return inner()

@timeout(2)
def inner():
    time.sleep(1)
    return "Should be printed"

print(outer())  # Should be printed

Note, that your task can be solved only with threads or processes, but this may lead to some non-obvious problems. I recommend you to think if your task can be solved without it. In most cases you can split your code to parts and check for timeout after each. Something like this:

def outer(arg, timeout=None):
    t = Timeout(timeout)
    # some operation:
    time.sleep(1)
    if t.is_timeout: return None
    # use time left as subfunction's timeout:
    return inner(arg, timeout=t.time_left)

edited Mar 20 '16 at 08:42

answered Mar 17 '16 at 05:50

Mikhail Gerasimov

36,989
16
116
159

`if raise_exc and t.is_alive()` is buggy. `t.is_alive()` won't be evaluated if `raise_exec` is `False` and you `return res` without confirming thtat thread has ended. what happens to the started non-main thread when you `raise TimeoutError()` in the main thread? It seems the started thread is not ending, keep running? Actually, how it works? – Nizam Mohamed Mar 19 '16 at 14:22
`t.is_alive()` shouldn't be checked if `raise_exc == False`: in that case we just return `None`, see timer's docstring. When we raise `TimeoutError()` started thread finished: it provided by `t.join(timeout=timeout)` line. – Mikhail Gerasimov Mar 19 '16 at 18:53
`t.join(timeout=timeout)` can return before the thread finishes. That's why we have to check with `t.is_alive()`. Anyhow the started thread keeps running after `raise TimeoutError()` if it's a long running task. – Nizam Mohamed Mar 19 '16 at 19:09
@NizamMohamed, that's true, thread keeps running, but we shouldn't worry about it as soon as our main main thread is not blocked. That's how this timeout decorator works. Trying to kill started thread manually is bad idea: what if it holds critical resources? Only way I see to finish it is to allow finish properly. – Mikhail Gerasimov Mar 19 '16 at 19:38
1

what happens if the non-main threads raises exception inside the thread? How can we make sure main thread is not blocked? – Nizam Mohamed Mar 20 '16 at 04:09
@NizamMohamed main thread shouldn't be blocked because in that case `join` just finish, doesn't it? But thanks for asking about exceptions: I updated function to handle exception inside non-main thread and pass it to main thread if raise_exc. – Mikhail Gerasimov Mar 20 '16 at 08:47
you can't know if timeout has occurred or thread finished without checking `is_alive`. The OP wanted to abort the operation if not finished within some time constraint. But here the operation keeps going. – Nizam Mohamed Mar 20 '16 at 09:32
hmm... you guys' discussions are _a bit_ beyond my level... I'll search and check, and get back... – akai Mar 20 '16 at 14:11
@akai, he is right that task "finished" by timeout still running in background thread. But I don't see any alternative to it. Besides it's hard to send arbitrary exception to another thread, it can easily break code (see answer here http://stackoverflow.com/a/325528/1113207 ). I still recommend you to leave idea of timeout decorator, but instead of this to check time manually after some operations (see the end of answer). – Mikhail Gerasimov Mar 20 '16 at 20:34
All right, I think I understand basically. Thanks for your discussions, they were very helpful. I will use one of the two (three) suggested solutions depending on my cases. – akai Mar 21 '16 at 14:57

score 2 · Answer 2 · answered Mar 15 '16 at 12:24

The timeout function uses threading.Timer to set the timer and thread.interrupt_main to interrupt the main thread.

from thread import interrupt_main
from threading import Timer
from time import time, sleep

def timeout(secs):
    def wrapper(func):
        timer = Timer(secs, interrupt_main)
        def decorated(*args, **kwargs):
            timer.start()
            return func(*args, **kwargs)
        return decorated
    return wrapper            

@timeout(1)
def outer():
    inner()

@timeout(5)
def inner():
    sleep(3)
    print("Should never be printed if you call outer()")

try:
    outer()
except:
    print('timed out')

Nestable timeout decorator? (i.e. timeout decorated func calling timeout decorated func)

2 Answers2