Python genetic optimisation multiprocessing with a global constant variable, how to speed up?

Question

I am writing a genetic optimization algorithm based on the deap package in python 2.7 (goal is to migrate to python 3 soon). As it is a pretty heavy process, some parts of the optimisation are processed using the multiprocessing package. Here is a summary outline of my program:

1. Configurations are read in and saved in a config object
2. Some additional pre-computations are made and saved as well in the config object
3. The optimisation starts (population is initialized randomly and mutations, crossover is applied to find a better solution) and some parts of it (evaluation function) are executed in multiprocessing
4. The results are saved

For the evaluation function, we need to have access to some parts of the config object (which after phase 2 stays a constant). Therefore we make it accessible to the different cores using a global (constant) variable:

from deap import base
import multiprocessing

toolbox = base.Toolbox()

def evaluate(ind):
    # compute evaluation using config object
    return(obj1,obj2)

toolbox.register('evaluate',evaluate)

def init_pool_global_vars(self, _config):
    global config
    config = _config

...
# setting up multiprocessing
pool = multiprocessing.Pool(processes=72, initializer=self.init_pool_global_vars,
                                        initargs=[config])
toolbox.register('map', pool.map_async)
...
while tic < max_time:
    # creating new individuals
    # computing in optimisation the objective function on the different individuals
    jobs = toolbox.map(toolbox.evaluate, ind)
    fits = jobs.get()
    # keeping best individuals

We basically make different iterations (big for loop) until a maximum time is reached. I have noticed that if I make the config object bigger (i.e. add big attributes to it, like a big numpy array) even if the code is still same it runs much slower (fewer iterations for the same timespan). So I thought I would make a specific config_multiprocessing object that contains only the attributes needed in the multiprocessing part and pass that as a global variable, but when I run it on 3 cores it is slower than with the big config object and on 72 cores, it is slightly faster, but not much.

What should I do in order to make sure my loops don't suffer in speed from the config object or from any other data manipulations I make before launching the multiprocessing loops?

Running in a Linux docker image on a linux VM in the cloud.

Answer 1

The joblib package is designed to handle cases where you have large numpy arrays to distribute to workers with shared memory. This is especially useful if you are treating the data in shared memory as "read-only" like what you describe in your scenario. You can also create writable shared memory as described in the docs.

Your code might look something like:

import os

import numpy as np
from joblib import Parallel, delayed
from joblib import dump, load

folder = './joblib_memmap'
try:
    os.mkdir(folder)
except FileExistsError:
    pass

def evaluate(ind, data):
    # compute evaluation using shared memory data
    return(obj1, obj2)

# just used to initialize memory mapped data
def init_memmap_data(original_data):
    data_filename_memmap = os.path.join(folder, 'data_memmap')
    dump(original_data, data_filename_memmap)
    shared_data = load(data_filename_memmap, mmap_mode='r')
    return shared_data

...
# however you set up indices needs to be changed here
indexes = range(10)  

# however you load your numpy data needs to be done here
shared_data = init_memmap_data(numpy_array_to_share)  

# change n_jobs as appropriate
results = Parallel(n_jobs=2)(delayed(evaluate)(ind, shared_data) for ind in indexes)  

# get index of the maximum as the "best" individual
best_fit_individual = indexes[results.argmax()]

Additionally, joblib supports a threading backend that may be faster than the process based one. It will be easy to test both with joblib.