37

I'm using python framework unittest. Is it possible to specify by framework's abilities a timeout for test? If no, is it possible to specify gracefully a timeout for all tests and for some separated tests a private value for each one?
I want to define a global timeout for all tests (they will use it by default) and a timeout for some test that can take a long time.

Jury
  • 1,227
  • 3
  • 17
  • 30

2 Answers2

35

As far as I know unittest does not contain any support for tests timeout.

You can try timeout-decorator library from PyPI. Apply the decorator on individual tests to make them terminate if they take too long:

import timeout_decorator

class TestCaseWithTimeouts(unittest.TestCase):

    # ... whatever ...

    @timeout_decorator.timeout(LOCAL_TIMEOUT)
    def test_that_can_take_too_long(self):
        sleep(float('inf'))

    # ... whatever else ...

To create a global timeout, you can replace call

unittest.main()

with

timeout_decorator.timeout(GLOBAL_TIMEOUT)(unittest.main)()
Lav
  • 2,204
  • 12
  • 23
  • 1
    Interesting. I'm not using `unittest.main()`, but I hope I can adopt `decorator` for my case. But my tests are not going in single thread... – Jury Jan 12 '16 at 13:50
  • 1
    @Jury Check the "Multithreading" section in [`timeout-decorator` reference](https://pypi.python.org/pypi/timeout-decorator) - you just need to use `timeout_decorator.timeout(TIMEOUT, use_signals=False)` in multi-threaded environment. – Lav Jan 12 '16 at 14:24
  • Yes, I've seen it. I'll try. – Jury Jan 13 '16 at 12:48
  • 1
    I don't know what is going on, but `use_signals=False` doesn't work for me, but with signals it looks working. As I found, this module makes hook to call `_Timeout.__call__` instead of testmethod directly. The problem is in fact that `self` of testmethod is lost (replaced) by `self` of `_Timeout`. When called, `testmethod` doesn't have any `self` and it fails. I don't know what is going on and how to fix it. Trick with global timeout doesn't work too. – Jury Jan 17 '16 at 09:47
  • Unfortunately, neither it works in my case with the async websockets tests in Python 3. – Kostanos Sep 23 '17 at 13:04
  • 1
    @Jury I actually make it works. Only by using decorator with exception: `@timeout_decorator.timeout(TIMEOUT, timeout_exception=StopIteration)`. You'll need to put this decorator on all potential stacked tests. In my case it is in tests related to async/websocket – Kostanos Sep 23 '17 at 13:13
  • This is a lovely approach - thanks for sharing! Decorating individual classes works for me, but wrapping unittest.main does not. I'm using Python 3.7.7 on MacOS 15 (Catalina.) – Adam Wildavsky Jul 17 '20 at 21:33
  • This works for my tests that hang in Python code, but not when they get lost during a call to an external C++ library. I tried with and without *use_signals*. Any suggestions? – Adam Wildavsky Jul 18 '20 at 01:48
  • 1
    @AdamWildavsky I'm afraid there's too much that can happen in C++ code. You could try running your entire test suite in a subprocess with parent process in charge of timeout - that's kinda a sledgehammer approach, but it's the best I can offer off the top of my head. Perhaps better to frame it as a separate question as it deserves to be treated as such IMHO. – Lav Dec 08 '20 at 07:11
14

I built a unittest timeout solution using context managers (the with keyowrd), based on this answer.

This approach also uses signal, so it might only be valid on *nix systems (I've only run it in my Ubuntu 16.04 environment).

  1. Import signal, add a TestTimeout exception:
import signal

...

class TestTimeout(Exception):
    pass
  1. Define class test_timeout, which will handle the with blocks:
class test_timeout:
  def __init__(self, seconds, error_message=None):
    if error_message is None:
      error_message = 'test timed out after {}s.'.format(seconds)
    self.seconds = seconds
    self.error_message = error_message

  def handle_timeout(self, signum, frame):
    raise TestTimeout(self.error_message)

  def __enter__(self):
    signal.signal(signal.SIGALRM, self.handle_timeout)
    signal.alarm(self.seconds)

  def __exit__(self, exc_type, exc_val, exc_tb):
    signal.alarm(0)
  1. Embed with test_timeout() blocks in your unit tests:
def test_foo(self):
  with test_timeout(5):  # test has 5 seconds to complete
    ... foo unit test code ...

With this approach, tests that time out will result in an error due to the raise TestTimeout exception.

Optionally, you could wrap the with test_timeout() block in a try: except TestTimeout: block, and handle the exception with more granularity (skip a test instead of error for example).

caffreyd
  • 1,151
  • 1
  • 17
  • 25