1

I have a genetic algorithm which I would like to speed up. I'm thinking the easiest way to achieve this is by pythons multiprocessing module. After running cProfile on my GA, I found out that most of the computational time takes place in the evaluation function.

def evaluation():
    scores = []
    for chromosome in population:
        scores.append(costly_function(chromosome))

How would I go about to parallelize this method? It is important that all the scores append in the same order as they would if the program would run sequentially.

I'm using python 2.7

Chicony
  • 393
  • 3
  • 17

1 Answers1

2

Use pool (I show both imap and map because of some results on google say map may not be OK for ordering though I have yet to see proof):

from multiprocessing import Pool
def evaluation(population):
    return list(Pool(processes=nprocs).imap(costly_function,population))

or (what I use):

 return Pool(processes=nprocs).map(costly_function,population)

Define nprocs to the number of parallel process you want.

From: https://docs.python.org/dev/library/multiprocessing.html#multiprocessing.pool.Pool

kabanus
  • 24,623
  • 6
  • 41
  • 74
  • I don't think that like this the scores will append in the same order as they would if the program would run sequentially – MMF Nov 03 '16 at 13:38
  • I suggest you try it out instead of guessing, it will (if you don't believe use a short list at first). Note I edited a syntax error. – kabanus Nov 03 '16 at 13:47
  • I confirmed that `Pool.map` keeps the original order. I created a function that takes an index, sleeps a random amount of time, and returns the index. No matter the number of process or the chunksize `Pool.map` maintained order over hundreds items. This makes sense because `Pool.map` is modeled after the built-in `map()` function. – Steven Rumbalski Nov 03 '16 at 15:49
  • I have also confirmed that although the results are kept in order, evaluation is not in order. This means that `Pool.map` keeps track of the original order internally and it's not just exceeding good luck or idle processes blocking that maintains the order. – Steven Rumbalski Nov 03 '16 at 16:18
  • @kabanus I have a slightly relevant question: I want to parallelize two calls to the same function, each with different arguments, in a for loop but I do not know how to do that in practice. Could you please take a look at my question [here](https://stackoverflow.com/questions/49145136/how-to-use-multiprocessing-to-parallelize-two-calls-to-the-same-function-with-d) and see if you can offer a solution? – Amir Mar 07 '18 at 06:06