0

I have an embarrassingly parallel code that creates a list of arrays and I want to parallelize it (It's my first time working with python's multiprocessing so I'm still grasping the concepts). runNramps(*args) returns five arrays, t1,c1,e1,m1 and d1. I want to run this in multiple threads and join the answers from all jobs together.

Here's the code I'm trying to parallelize by running it in multiple processes. My jobs appear to execute, and close properly, however the queue is empty and queue.get() doesn't return anything. I think I'm missing the way to merge all the outputs together but I can't find any good example on SO/google.

Here is the threaded version of my code (I want to make minimal changes to the previous function and just call it multiple times, joining the results into one array).

def runPramps(numramps,rmpslope,nframes,cosmag=0,method='2pt',thresh=1.0):

    output = mp.Queue()
    processes = [mp.Process(target=runNramps, args=(numramps,rmpslope,nframes,cosmag,method,thresh)) for x in range(4)] #4 processes for example

    for p in processes:
            p.start()
            print "started", p

    for p in processes:
            p.join()
            print 'ended',p


    results = output.get(False) #it hangs here
    return results

I feel I have a small error/misunderstanding as I've made sure the processes are running and ending.

Many thanks for your help!

TZHX
  • 5,291
  • 15
  • 47
  • 56
apmechev
  • 75
  • 1
  • 8
  • 3
    We need to see `runNramps` to answer, I think. But it looks like you're not actually passing the `output` queue to the child processes, so I'm not sure how you expect any values to end up getting added to it... – dano Apr 17 '15 at 15:15
  • 1
    Have you considered using a `multiprocessing.Pool` or a `concurrent.futures.ProcessPoolExecutor` instead of trying to build a pool manually? Of course you _can_ do everything they do yourself, but there's a lot of places you can make a mistake, especially your first time, and it's a lot more code to write and to read, so unless you need to do something they can'tâ€Ĥ – abarnert Apr 17 '15 at 15:18
  • pools and futures, like abarnert says, is the way to structure this kind of code. i just elaborated on that here (https://stackoverflow.com/a/69095442/31024) – Peter S Magnusson Sep 09 '21 at 02:03

1 Answers1

0

The problem is due to the fact that you're joining the processes before draining the queue.

You can take a look at the multiprocessing guidelines under the "Joining processes that use queues" for a better explanation of the issue.

Just swap the join and the Queue.get logic and everything should work.

Nevertheless, I'd rather use a Pool of workers to solve your problem.

noxdafox
  • 14,439
  • 4
  • 33
  • 45