6

I would like to use multiprocess package for example in this code. I've tried to call the function create_new_population and distribute the data to 8 processors, but when I do I get pickle error.

Normally the function would run like this: self.create_new_population(self.pop_size)

I try to distribute the work like this:

f= self.create_new_population
pop = self.pop_size/8
self.current_generation = [pool.apply_async(f, pop) for _ in range(8)]

I get Can't pickle local object 'exhaust.__init__.<locals>.tour_select'
or PermissionError: [WinError 5] Access is denied

I've read this thread carefully and also tried to bypass the error using an approach from Steven Bethard to allow method pickling/unpickling via copyreg:

def _pickle_method(method)
def _unpickle_method(func_name, obj, cls)

I also tried to use pathos package without any luck.
I've know that the code should be called under
if __name__ == '__main__': block, but I would like to know if this can be done with minimum possible changes in the code.

Panos Kalatzantonakis
  • 12,525
  • 8
  • 64
  • 85
  • 2
    Hi I'm the `pathos` author. You don't need to follow the advice about picking instance methods, `dill` can pickle instance methods no problem... and `pathos` uses `dill`. You also don't need to run your code under `__main__`, however it typically does help, especially if you are on windows (and if you are, then use `freeze_support`). So your question is... can your code be modified to run under `__main__` with little changes -- the answer is yes. Just (a) add a wrapper function called from `__main__` that calls your code, and (b) if possible, (but not necessary) build `Pool` in `__main__`. – Mike McKerns Apr 09 '18 at 12:24
  • 2
    It looks like you are trying to build a genetic algorithm that leverages parallel computing. There is one in `mystic`, if you are interested. It uses `pathos`, and has over 10 years of support for massively parallel computations. https://github.com/uqfoundation/mystic -- Note: I'm also the `mystic` author. – Mike McKerns Apr 09 '18 at 12:30
  • @MikeMcKerns .Thank you very much Mike, I will implement (a) and (b) suggestions in my code. Meanwhile I've updated python in my pc to 3.6.5 and pathos run! But the fact that is 5 times slower than the serial version puzzles me. – Panos Kalatzantonakis Apr 10 '18 at 08:59
  • 1
    There's overhead in setting up the parallel computation. With `multiprocessing`, it has to spin up N instances of python and also serialize the objects. Parallel doesn't always mean faster... it gets faster when your code is more computationally expensive than the overhead of going to parallel. You might want to try `pathos.pools.ThreadPool`. – Mike McKerns Apr 10 '18 at 11:04

0 Answers0