how to use the partial function with the python multiprocessing module?

Question

I am looking for a generic way to use the multiprocessing module for a function whose parameters have been defined by the positional-OR-keyword type (https://docs.python.org/2/glossary.html#term-parameter).

Below, is a simple example of how I approached the problem

from functools import partial
from multiprocessing import Pool

def VariadicLifter(func, args):
  return func(*args)

def func(x,y,z,a):
  return x+2*y+3*z+4*a


if __name__ == '__main__':
  func_  = partial( func, 500, 1007)
  lfunc_ = partial( VariadicLifter, func_)


  RANGE = zip( range(10,31),range(10,31) )
  pool = Pool(processes=6)
  result_array = pool.map( lfunc_,  RANGE )

  pool.close()
  pool.join()

This works: the result of each call of lfunc_ is available in result_array.

Now, I'm trying to apply this pattern in another context and I get the error message.

File "c:\Python27\lib\multiprocessing\pool.py", line 251, in map
  return self.map_async(func, iterable, chunksize).get()
File "c:\Python27\lib\multiprocessing\pool.py", line 567, in get
  raise self._value
cPickle.PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

but if I replace the multiprocessing map with a regular map, the program has no issue running. Are there any limitation for the function argument for the multiprocessing pool? (my understanding is that, at least, lambda function cannot be used with the multiprocessing pool)

Thanks

Answer 1

"What are the limitations" is a wide topic, but the one you are facing now is discussed more widely here: Why can I pass an instance method to multiprocessing.Process, but not a multiprocessing.Pool?

Your problem is the use of partial. It creates a functools.partial object as a result, not a function. You cannot pass instance methods to Pool.map, they need to be plain functions. The default pickler is unable to pickle them, hence the error.

This is not a solution to your problem, just an explanation why it does not work. There is a workaround in the linked article with a different pickler, but I have never tested it.