I'm using the Multiprocessing module within joblib. I'm coding an iterative algorithm where I need to call Parallel on every iteration. I suspect I'm suffering from a lot of overhead here, because I'm creating and destroying a pool of workers on every call.
The joblib documentation has a reference to this case. However, the solution doesn't appear to work on Python 2.7:
with Parallel(n_jobs=2) as parallel:
accumulator = 0.
n_iter = 0
while accumulator < 1000:
results = parallel(delayed(sqrt)(accumulator + i ** 2) for i in range(5))
accumulator += sum(results) # synchronization barrier
n_iter += 1
Which yields the error message:
AttributeError: __exit__
Another stackoverflow post (Python Multiprocessing Lib Error (AttributeError: __exit__)) had a response that mentioned that the with statement in Python 2.7 requires a context manager. They suggested defining a context manager that wraps around the Parallel call. I tried the following:
from contextlib import contextmanager, closing
@contextmanager
def terminating(fn):
try:
yield fn
finally:
fn.terminate()
with terminating(Parallel(n_jobs=2)) as parallel:
accumulator = 0.
n_iter = 0
while accumulator < 1000:
results = parallel(delayed(sqrt)(accumulator + i ** 2) for i in range(5))
accumulator += sum(results) # synchronization barrier
n_iter += 1
However, I get the following error message:
AttributeError: 'Parallel' object has no attribute 'terminate'
Does anybody know how I can properly build a context manager for Parallel in Python 2.7?
