Python Multiprocessing - Parrallel processes

Question

I'm new to Python multiprocessing, and I'm trying to implement some parallel calculations. I've got the info that this:

#M is an integer, contains the number of processes I'd like to launch.
results = []
for i in range(0, M):
        p = Process(target=processchild, args=(data[i],q))
        p.start()
        result.append(q.get())
        p.join()

is still sequential, because .join() causes the loop to wait until p is finished before starting the next one. I've read here in an answer, that

You'll either want to join your processes individually outside of your for loop (e.g., by storing them in a list and then iterating over it)...

So if I'd modify my code to

results = []
for i in range(0, M):
        processes[i] = Process(target=processchild, args=(data[i],q))
        processes[i].start()
        result.append(q.get())

for i in range(0, M):
        processes[i].join()

Would it actually run in parallel now? If not, how can I modify my code to work that way? I've read the solution using numpy.Pool and apply_async posted as an answer to the question I previously linked, so I'm mostly interested in a solution that doesn't use these.

sounds correct. You should check with system tools that `M` processes are actually running concurrently. — Jean-François Fabre, Nov 06 '16 at 09:20

score 0 · Answer 1 · answered Nov 06 '16 at 09:20

0

Yes, this will run in parallel.

All processes are started before you try joining one, so this will not block after the first process.

answered Nov 06 '16 at 09:20

Martin Richtarsky

619
7
18

@robyschek Can I add `q.get()` to the second loop after the `join()`? Will that mean correct parallel running? – lte__ Nov 06 '16 at 09:26
@robyschek Thank you! – lte__ Nov 06 '16 at 09:29
@robyschek Since your answer was the correct one, I'll accept it if you add it as an answer! :) – lte__ Nov 19 '16 at 10:24

Python Multiprocessing - Parrallel processes

1 Answers1