Creating a timeout function in Python with multiprocessing

Question

I'm trying to create a timeout function in Python 2.7.11 (on Windows) with the multiprocessing library.

My basic goal is to return one value if the function times out and the actual value if it doesn't timeout.

My approach is the following:

from multiprocessing import Process, Manager

def timeoutFunction(puzzleFileName, timeLimit):
  manager = Manager()
  returnVal = manager.list()

  # Create worker function
  def solveProblem(return_val):
    return_val[:] = doSomeWork(puzzleFileName) # doSomeWork() returns list

  p = Process(target=solveProblem, args=[returnVal])
  p.start()

  p.join(timeLimit)
  if p.is_alive():
    p.terminate()
    returnVal = ['Timeout']

  return returnVal

And I call the function like this:

if __name__ == '__main__':
  print timeoutFunction('example.txt', 600)

Unfortunately this doesn't work and I receive some sort of EOF error in pickle.py

Can anyone see what I'm doing wrong?

Thanks in advance,
Alexander

Edit: doSomeWork() is not an actual function. Just a filler for some other work I do. That work is not done in parallel and does not use any shared variables. I'm only trying to run a single function and have it possibly timeout.

Just some filler function. My actual solveProblem() doesn't look like that. Was just trying to simplify my problem, seeing as the content of the function works fine. — Alexander Hartvig Nielsen, May 08 '16 at 09:40

noxdafox · Accepted Answer · 2017-11-09T08:20:23.610

You can use the Pebble library for this.

from pebble import concurrent
from concurrent.futures import TimeoutError

TIMEOUT_IN_SECONDS = 10

@concurrent.process(timeout=TIMEOUT_IN_SECONDS)
def function(foo, bar=0):
    return foo + bar

future = function(1, bar=2)

try:
    result = future.result()  # blocks until results are ready or timeout
except TimeoutError as error:
    print "Function took longer than %d seconds" % error.args[1]
    result = 'timeout'

The documentation has more complete examples.

The library will terminate the function if it timeouts so you don't need to worry about IO or CPU being wasted.

EDIT:

If you're doing an assignment, you can still look at its implementation.

Short example:

from multiprocessing import Pipe, Process

def worker(pipe, function, args, kwargs):
    try:
        results = function(*args, **kwargs)
    except Exception as error:
        results = error

    pipe.send(results)

pipe = Pipe(duplex=False)
process = Process(target=worker, args=(pipe, function, args, kwargs))

if pipe.poll(timeout=5):
    process.terminate()
    process.join()
    results = 'timeout'
else:
    results = pipe.recv()

Pebble provides a neat API, takes care of corner cases and uses more robust mechanisms. Yet this is more or less what it does under the hood.

Very neat! Unfortunately this is for an assignment, so I'd prefer to avoid using external libraries. But I'll definitively keep this in mind for other projects. Thanks! — Alexander Hartvig Nielsen, May 08 '16 at 10:43

score 0 · Answer 2 · answered May 08 '16 at 11:03

The problem seems to have been that the function solveProblem was defined inside my outer function. Python doesn't seem to like that. Once I moved it outside it worked fine.

I'll mark noxdafox answer as an answer as I implementing the pebble solution led me to this answer.

Thanks all!

Creating a timeout function in Python with multiprocessing

2 Answers2

Linked