37

I've implemented some simple parallelism in a Monte Carlo code using the Python multiprocessing module. I have code that looks like:

montecarlos = [MonteCarlo(f,fargs) for fargs in farglist]
jobs = [multiprocessing.Process(mc) for mc in montecarlos]
for job in jobs: job.start()
for job in jobs: job.join()
results = [mc.results for mc in montecarlos]

However, when I look at the results list, it looks like the monte carlo iterators haven't even started. I know that they have, because I can have the processes print out information during the monte carlo steps. So I'm doing something dumb. I had thought the job.join() would keep the results list from being constructed until everything had run, and thus the mc.results field would be updated.

I realize I haven't told you the details of my monte carlo routine, and hope that it doesn't matter, and that the mistake I'm making is in my interpretation of what multiprocessing does. Thanks in advance for any help you can offer.

atp
  • 30,132
  • 47
  • 125
  • 187
Rick
  • 1,784
  • 3
  • 15
  • 25

1 Answers1

46

The MonteCarlo objects have been pickled and sent to child processes to be run - the .results attribute in this process isn't populated because the local mc has never been run.

If you create a multiprocessing.Queue, you can pass that into each MonteCarlo job, and when it finishes it should put the result in there. Then the top-level can wait for values from the queue. (Under the hood this will pickle and unpickle the result object.) 

result_queue = multiprocessing.Queue()
montecarlos = [MonteCarlo(result_queue, f,fargs) for fargs in farglist]
jobs = [multiprocessing.Process(mc) for mc in montecarlos]
for job in jobs: job.start()
for job in jobs: job.join()
results = [result_queue.get() for mc in montecarlos]
ajon
  • 7,868
  • 11
  • 48
  • 86
babbageclunk
  • 8,523
  • 1
  • 33
  • 37
  • This worked great! Thanks very much! Thanks also for such a clear explanation as to what was going on. – Rick Nov 30 '11 at 18:01
  • This doesn't work with Python3. Can you update please! – Trect Oct 29 '18 at 17:00
  • 1
    'the f in the second line montecarlos = [MonteCarlo(result_queue, f,fargs) for fargs in farglist] ? – pelos Dec 13 '18 at 19:02
  • 1
    `f` is whatever it is in the original code in the question - some function that is being analysed using monte carlo techniques. – babbageclunk Dec 17 '18 at 22:53