How can you safeguard yourself from a flaky library call that might hang indefinitely?

Question

Suppose you find yourself in the unfortunate position of having a dependency on a poorly behaved library. Your code needs to call FlakyClient.call(), but sometimes that function ends up hanging for an unacceptable amount of time.

As shown below, one way around this is to wrap the call in its own Process, and use the timeout parameter in the join method to define a maximum amount of time that you're willing to wait on the FlakyClient. This provides a good safeguard, but it also prevents the main body of code from reacting to the result of calling FlakyClient.call(). The only way that I know of addressing this other problem (getting the result into the main body of code) is by using some cumbersome IPC technique.

What is a clean and pythonic way of dealing with these two problems? I want to protect myself if the library call hangs, and be able to use the result if the call completes.

Thanks!

from multiprocessing import Process
from flaky.library import FlakyClient


TIMEOUT_IN_SECS = 10

def make_flaky_call():
    result = FlakyClient.call()

proc = Process(target=make_flaky_call)
proc.start()
proc.join(TIMEOUT_IN_SECS)
if proc.is_alive():
    proc.terminate()
    raise Exception("Timeout during call to FlakyClient.call().")

You can use threads, or lightweight constructs like greenlets. — Robert Moskal, Apr 28 '16 at 02:04
Instead of waiting for it could you call it a bit earlier then you will need it and see if it has finished processing by the time it is needed? — Tadhg McDonald-Jensen, Apr 28 '16 at 02:05
I like to think this question can best be solved by using `asyncio`'s event loops and running your functions asynchronously (see [example](http://stackabuse.com/python-async-await-tutorial/)), but `asyncio` is not available before Python 3.4. — Akshat Mahajan, Apr 28 '16 at 02:09
@TopherFischer: Why do you need to integrate your task back into the main body of code? — Akshat Mahajan, Apr 28 '16 at 02:17
What do you want to do if the library call hangs? Move on, but let the call finish naturally, or terminate the call? If the latter then you need to know if this flaky call is implemented in pure python or in C. If the latter you should call the function from another process (if the function doesn't release the GIL, then you must). — Dunes, Apr 28 '16 at 08:14

Akshat Mahajan · Answer 1 · 2016-04-28T23:15:57.623

I cannot speak to Python 2.7, but in Python 3, the correct way to handle this is to make use of asyncio and the concept of futures.

import concurrent

def make_flaky_call():
    return FlakyClient.call()

timeout = 10

with concurrent.futures.ThreadPoolExecutor(max_workers=1) as executor:
    future = executor.submit(make_flaky_call) # get a future object
    try:
        result = await future.result(timeout = timeout)
    except concurrent.futures.TimeOutError:
        # if a timeout occurs on the call, do something
        result = None # default value

This is fairly Pythonic. You can integrate this with the main body of the code. It correctly uses try-except for error handling. It comes with an inbuilt timeout. It only works in Python 3.5 (thanks to await - but changing to yield from makes it compatible with Python 3.4).

For Python 2.7, unfortunately, the right way to handle that is to do what you are currently doing.

Thanks for the Python 3 explanation. This approach is what I'm used to seeing in Java-land. Unfortunately I'm still working on Python 2.7 code. — Topher Fischer, Apr 28 '16 at 23:00

score 2 · Accepted Answer · answered Apr 28 '16 at 07:10

If you are using Process I would suggest you use a Queue to handle result transfer and indirectly also manage function timeout.

from multiprocessing import Process, Queue
from flaky.library import FlakyClient
import time

TIMEOUT_IN_SECS = 10

def make_flaky_call(queue):
    result = FlakyClient.call()
    queue.put(result)
    queue.put('END')

q = Queue()
proc = Process(target=make_flaky_call, args=(q,))
proc.start()
content = 0
result = None
while content != 'END':
    try:
        content = q.get(timeout=TIMEOUT_IN_SECS)
        if content != 'END':
            result = content
    except Empty:
        proc.terminate()
        raise Exception("Timeout during call to FlakyClient.call().")

How can you safeguard yourself from a flaky library call that might hang indefinitely?

2 Answers2

Linked