Simple parallel scientific programming

Question

I quite often write simple optimization routines that look something like this:

def createinstance(n):
    while(True):
        #create some instance called instance
        yield instance



loopno = 100000
n= 100
min = 0
for i in xrange(loopno):
    for inst in createinstance(n):
        value = foo(inst)
        if (value < min):
            min = value
            print min

I would like to be able to use all of the cores on my machine to do this.

A very simple method would just split the range into parts and farm them out to the cores and collect the results at the end.

A better method would have the cores request a batch of instances when they are idle.

What's a nice way to solve this problem for maximum efficiency in python? Maybe this is standard enough to be a community question?

This question seems to be an even simpler version of Solving embarassingly parallel problems using Python multiprocessing .

Are you using the values of `i` and `n` somewhere? `createinstance` is now an infinite loop. — Janne Karila, Feb 14 '13 at 12:37
@JanneKarila `n` is used in createinstance. `i` is not used anywhere except as a counter. — Majid, Feb 14 '13 at 13:35

Samizdis · Answer 1 · 2013-02-14T12:06:09.410

0

With regards to the standard Python install, the multiprocessing module seems to be what you'd want, the example case in http://docs.python.org/2/library/multiprocessing.html looks like it would be easily adapted to your case. If memory isn't an issue, simply splitting the range over N processors would be the most efficient option, as you have no communication between processes.

There seem to be many ways of parallelising python, either using MPI or SMP. There's a list of potential candidates on the python wiki http://wiki.python.org/moin/ParallelProcessing

Parallel Python http://www.parallelpython.com supports SMP, which is probably what you want in this instance (single machine, multiple processors, rather than a cluster of machines).

edited Feb 14 '13 at 12:06

answered Feb 14 '13 at 11:57

Samizdis

1,591
1
17
33

Thanks. The problem with just splitting the range is that you risk having idle processors. It is indeed for a simple machine with multiple cores and shared memory. – Majid Feb 14 '13 at 13:36
Why would you necessarily have idles processors? Say you had 4 processors, if you have a Pool of 4 processes and run result=myPool.apply_async(min,myList) and then result.get() you should be automatically sharing over all processors. Pool(None) even sets the number of processors to cpu_count(). With an iterator you might have trouble, I suppose, if you didn't want to evaluate it all at once. – Samizdis Feb 14 '13 at 15:04
The problem is that some work may take less time than others. The ones that finish their work early then set idle. Or did I misunderstand the model? Is there some way for the processors to request more work? – Majid Feb 14 '13 at 16:01

Simple parallel scientific programming

1 Answers1