Multiprocesses will not run in Parallel on Windows on Jupyter Notebook

Question

I'm currently working on Windows on jupyter notebook and have been struggling to get multiprocessing to work. It does not run all my async's in parallel it runs them singularly one at a time please provide some guidance where am I going wrong. I need to put the results into a variable for future use. What am I not understanding?

import multiprocessing as mp
import cylib
Pool = mp.Pool(processes=4)
result1 = Pool.apply_async(cylib.f, [v]) # evaluate asynchronously 
result2 = Pool.apply_async(cylib.f, [x]) # evaluate asynchronously
result3 = Pool.apply_async(cylib.f, [y]) # evaluate asynchronously 
result4 = Pool.apply_async(cylib.f, [z]) # evaluate asynchronously

vr = result1.get(timeout=420) 
xr = result2.get(timeout=420)
yr = result3.get(timeout=420) 
zr = result4.get(timeout=420)

[look here](https://stackoverflow.com/questions/52693216/multiprocessing-pool-example-does-not-work-and-freeze-the-kernel/52693952#52693952). Why do you think you need `pool.apply_async()` here? Why don't you just use `pool.map()`? — Darkonaut, Feb 02 '19 at 03:20

score 1 · Answer 1 · answered Feb 02 '19 at 06:43

The tasks are executing in parallel.

However, this is fetching the results synchronously i.e. "wait until result1 is ready, then wait until result2 is ready, .." and so on.

vr = result1.get(timeout=420) 
xr = result2.get(timeout=420)
yr = result3.get(timeout=420) 
zr = result4.get(timeout=420)

Consider the following example code, where each task is polled asynchronously

from time import sleep
import multiprocessing as mp
pool = mp.Pool(processes=4)

# Create tasks with longer wait first
tasks = {i: pool.apply_async(sleep, [t]) for i, t in enumerate(reversed(range(3)))}
done = set()


# Keep polling until all tasks complete
while len(done) < len(tasks):

    for i, t in tasks.items():

        # Skip completed tasks
        if i in done:
            continue

        result = None

        try:
            result = t.get(timeout=0)
        except mp.TimeoutError:
            pass
        else:
            print("Task #:{} complete".format(i))
            done.add(i)

You can replicate something like the above or use the callback argument on apply_async to perform some handling automatically as tasks complete.

Multiprocesses will not run in Parallel on Windows on Jupyter Notebook

1 Answers1