4

I have built a tool using django to automate script execution. The tool is working fine but sometimes the scripts take too long to execute. I want to limit the time for which my tool can execute each script. I have found 2 approaches and implemented them but I am not sure which is the right one to use.

1.) Using the signal module
2.) Using multiprocessing

Here is the sample code for both approaches

1.) Using the signal module

import signal
from contextlib import contextmanager

class TimeoutException(Exception): pass

@contextmanager
def time_limit(seconds):
    def signal_handler(signum, frame):
        raise TimeoutException("Timed out!")
    signal.signal(signal.SIGALRM, signal_handler)
    signal.alarm(seconds)
    try:
        yield
    finally:
        signal.alarm(0)

try:
    with time_limit(10):
        long_function_call()
except TimeoutException as e:
    print("Timed out!")

2.) Using multiprocessing

from multiprocessing import Process
from time import sleep

def f(time):
    sleep(time)

def run_with_limited_time(func, args, kwargs, time):
    p = Process(target=func, args=args, kwargs=kwargs)
    p.start()
    p.join(time)
    if p.is_alive():
        p.terminate()
        return False

    return True

if __name__ == '__main__':
    print run_with_limited_time(f, (1.5, ), {}, 2.5) # True
    print run_with_limited_time(f, (3.5, ), {}, 2.5) # False

The problem am facing with signal module is that signal only works in main thread. Want to know which is the better approach and why? Also if there is any approach I can use to alter the behaviour of signal module.

Krupa Ghetia
  • 41
  • 5
  • did you read the comments for the answers to selected of the question? https://stackoverflow.com/questions/366682/how-to-limit-execution-time-of-a-function-call-in-python – Brown Bear Apr 17 '19 at 07:08
  • Yes I went through it. But not very clear. – Krupa Ghetia Apr 17 '19 at 07:16
  • i think this one is main https://stackoverflow.com/questions/366682/how-to-limit-execution-time-of-a-function-call-in-python#comment62223603_26664130 to make your choose. Because you never know where your code will be run tomorrow. – Brown Bear Apr 17 '19 at 07:19
  • The problem am facing with signal module it that signal only works on the main thread. Is there any approach using which I could alter this. – Krupa Ghetia Apr 17 '19 at 07:30
  • i think you should add the your last comment to the question. – Brown Bear Apr 17 '19 at 07:31
  • I think the first solution uses less system resources. – kantal Apr 17 '19 at 07:41

1 Answers1

2

The signal based approach comes with several corner cases and limitations. It is not portable, signals can be handled only on the main thread and if your application is busy in a low level loop (because it's calling some C api for example) your application will become unresponsive.

I would recommend the multiprocessing based approach as it overcomes all the above limitations and has one major benefit: it protects your service from crashes, timeouts and instabilities deriving from the logic you run in your functions.

There are few libraries built for helping in that, pebble and billiard are the ones which come to my mind.

noxdafox
  • 14,439
  • 4
  • 33
  • 45
  • And what do you think about a multithreaded solution instead of the multiprocessing one? – kantal Apr 17 '19 at 08:11
  • A thread cannot stop another thread in Python (longer answer [here](https://stackoverflow.com/questions/48540668/where-is-the-memory-leak-how-to-timeout-threads-during-multiprocessing-in-pytho/48545357#48545357)). Only feasible way is to let the threat poll a shared flag to check if it has to stop. This becomes difficult to do properly. Especially if you are dealing with third party code. – noxdafox Apr 17 '19 at 08:15
  • what if what you want is a database connection? dbapi compliant. what is the nature of what’s being returned by a connect() and can it “survive” the process change implied by multiprocessing? reason i’m asking is I kinda got this working with thread to connect which is given up on by the main thread. it does avoid 30 second hangs if db is offline. but it’s hacky. – JL Peyret Apr 18 '19 at 01:23