Why is multiprocessing.Process able to pickle decorated functions?

Question

So I read here that decorated functions cannot be pickled. Indeed:

import multiprocessing as mp

def deco(f):
    def wrapper(*args, **kwargs):
        try:
            f(*args, **kwargs)
        except:
            print 'Exception caught!'
    return wrapper

@deco
def f(x):
    print x
    raise OverflowError 

if __name__ == '__main__':
    pool = mp.Pool(processes=1)
    for _ in pool.imap_unordered(f, range(10)):
        pass
    pool.close()
    pool.join()
    print 'All done'

Out:

Traceback (most recent call last):
  File "deco0.py", line 19, in <module>
    for _ in pool.imap_unordered(f, range(10)):
  File "/Users/usualme/anaconda/lib/python2.7/multiprocessing/pool.py", line 659, in next
    raise value
cPickle.PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

But now if I replace map by a Process:

import multiprocessing as mp

def deco(f):
    def wrapper(*args, **kwargs):
        try:
            f(*args, **kwargs)
        except:
            print 'Exception caught!'
    return wrapper

@deco
def f(x):
    print x
    raise OverflowError 

if __name__ == '__main__':
    p = mp.Process(target=f, args=(1,))
    p.start()
    p.join()
    print 'All done'

Out:

1
Exception caught!
All done

Why is it working? Doesn't process need to pickle the decorated function as well?

`multiprocessing` uses `cPickle` in python 2.x and `pickle` in 3.x. There is a fork of `multiprocessing` that uses the `dill` serializer… which can pickle decorated functions. The fork, called `pathos.multiprocessing`, only replaces the serializer… and thus can accept decorated functions in both `Pool` and `Process`, and works on windows as well as linux -- and also works in the interpreter, also where `Pool` or `Process` even if not called from `__main__`. See: https://github.com/uqfoundation. — Mike McKerns, Oct 09 '14 at 17:54

dano · Accepted Answer · 2014-10-09T14:22:41.933

It's working because you're running on Linux, which doesn't need to pickle f to call it in a child process via Process.__init__. This is because f gets inherited by the child via os.fork. If you run the same code on Windows (which lacks fork), or try to pass f to Pool.apply/Pool/map (both of which would need to pickle f to call it in a subprocess), you'll get an error.

This example will fail no matter what platform you use:

import multiprocessing as mp

def deco(f):
    def wrapper(*args, **kwargs):
        try:
            f(*args, **kwargs)
        except:
            print 'Exception caught!'
    return wrapper

@deco
def f(x):
    print x
    raise OverflowError 

if __name__ == '__main__':
    p = mp.Pool()
    p.apply(f, args=(1,))  # f needs to be pickled here.
    print 'All done'

Output:

1
Exception caught!
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 319, in _handle_tasks
    put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

Why is multiprocessing.Process able to pickle decorated functions?

1 Answers1