10

I've researched this question multiple times, but haven't found a workaround that either works in my case, or one that I understand, so please bear with me.

Basically, I have a hierarchical organization of functions, and that is preventing me from multiprocessing in the top-level. Unfortunately, I don't believe I can change the layout of the program - because I need all the variables that I create after the initial inputs.

For example, say I have this:

import multiprocessing

  def calculate(x):
    # here is where I would take this input x (and maybe a couple more inputs)
    # and build a larger library of variables that I use further down the line

    def domath(y):
      return x * y

    pool = multiprocessing.Pool(3)
    final= pool.map(domath, range(3))

calculate(2)

This yields the following error:

Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

I was thinking of globals, but I'm afraid that I'd have to define too many and that may slow my program down quite a bit. Is there any workaround without having to restructure the whole program?

boardrider
  • 5,882
  • 7
  • 49
  • 86
Tim
  • 189
  • 1
  • 2
  • 9
  • You may want to read the answer I selected when trying to solve [pretty much the same error](http://stackoverflow.com/questions/6234586/we-need-to-pickle-any-sort-of-callable). – motoku Mar 03 '15 at 00:36
  • Hey Sean, after looking through your question, I'm afraid I may be out of my depth with the solution. Is there anyway you could give me a more conceptual rundown of what these functions are doing when packing and unpacking? – Tim Mar 03 '15 at 00:46
  • Of course. Let me get some sample code together. – motoku Mar 03 '15 at 00:47

3 Answers3

13

You could use pathos.multiprocessing, which is a fork of multiprocessing that uses the dill serializer instead of pickle. dill can serialize pretty much anything in python. Then, no need to edit your code.

>>> from pathos.multiprocessing import ProcessingPool as Pool
>>> 
>>> def calculate(x):
...   def domath(y):
...     return x*y
...   return Pool().map(domath, range(3))
... 
>>> calculate(2)
[0, 2, 4]

You can even go nuts with it… as most things are pickled. No need for the odd non-pythonic solutions you have to cook up with pure multiprocessing.

>>> class Foo(object):
...   def __init__(self, x):
...     self.x = x
...   def doit(self, y):
...     return ProcessingPool().map(self.squared, calculate(y+self.x))
...   def squared(self, z):
...     return z*z
... 
>>> def thing(obj, y):
...   return getattr(obj, 'doit')(y)
... 
>>> ProcessingPool().map(thing, ProcessingPool().map(Foo, range(3)), range(3))
[[0, 0, 0], [0, 4, 16], [0, 16, 64]]

Get pathos here: https://github.com/uqfoundation

Mike McKerns
  • 33,715
  • 8
  • 119
  • 139
  • Thanks Mike! That worked out great. Quick note about the file that available on http://trac.mystic.cacr.caltech.edu/project/pathos/wiki/Installation This version (at least when I downloaded it) does not have the multiprocessing module included. I think this might be why a couple users have had issues importing multiprocessing through pathos. After I downloaded files from github instead of the pathos page, I had everything I needed. – Tim Mar 03 '15 at 18:36
  • Cool. Yeah, I know.. thanks for the reminder. The version on the `pathos` wiki link you mention is very old (timestamp: 06/28/10 17:50). The github code is up to date, and a new release is "imminent". I'll update all links with a new stable release at that time. – Mike McKerns Mar 03 '15 at 20:31
  • Also note that [JobLib](https://joblib.readthedocs.io/en/latest/parallel.html) has an exceptional serialization pattern that bypasses most of the standard problems with built in pickle. – deepelement May 20 '19 at 13:00
  • 2
    @deepelement: most any python package, like `joblib`, that does advanced serialization relies on `dill` or `cloudpickle`. – Mike McKerns May 20 '19 at 15:22
  • @MikeMcKerns Your exactly right. Looks like `joblib` is on `cloudpickle` currently, with R/D towards `dill` – deepelement May 20 '19 at 15:49
6

The problem you encountered is actually a feature. The pickle source is actually designed to prevent this sort of behavior in order to prevent malicious code from being executed. Please consider that when addressing any applicable security implementation.

First off we have some imports.

import marshal
import pickle
import types

Here we have a function which takes in a function as an argument, pickles the parts of the object, then returns a tuple containing all the parts:

def pack(fn):
    code = marshal.dumps(fn.__code__)
    name = pickle.dumps(fn.__name__)
    defs = pickle.dumps(fn.__defaults__)
    clos = pickle.dumps(fn.__closure__)
    return (code, name, defs, clos)

Next we have a function which takes the four parts of our converted function. It translates those four parts, and creates then returns a function out of those parts. You should take note that globals are re-introduced into here because our process does not handle those:

def unpack(code, name, defs, clos):
    code = marshal.loads(code)
    glob = globals()
    name = pickle.loads(name)
    defs = pickle.loads(defs)
    clos = pickle.loads(clos)
    return types.FunctionType(code, glob, name, defs, clos)

Here we have a test function. Notice I put an import within the scope of the function. Globals are not handled through our pickling process:

def test_function(a, b):
    from random import randint
    return randint(a, b)

Finally we pack our test object and print the result to make sure everything is working:

packed = pack(test_function)
print((packed))

Lastly, we unpack our function, assign it to a variable, call it, and print its output:

unpacked = unpack(*packed)
print((unpacked(2, 20)))

Comment if you have any questions.

motoku
  • 1,571
  • 1
  • 21
  • 49
  • Thank you for taking the time to explain in much greater detail! I'm trying to apply this to the example I posted above, but I'm still running into issues. When it unpacks, does it not return the same formatted function as before? I'm still getting the pickling error when I run through this method (which probably means I'm using it incorrectly). EDIT: what do you think of @mike answer? – Tim Mar 03 '15 at 05:51
  • @Tim Yes, the "unpacked" function should be the same as the original. If you want to store the original source you'll need to be handling strings instead. As for Mike's answer, I think that is probably a better option. – motoku Mar 03 '15 at 15:29
  • @SeanPedersen: this is essentially what `dill` does for you, except it creates the same callable type as was stored… so if you pickle a lambda, you get a lambda back, or if you pickle a bound method, you get a bound method back, and so on. It also handles globals for you in most cases, and also enables you to treat `__main__` like a module. – Mike McKerns Mar 03 '15 at 17:59
1

How about just taking the embedded function out?

This seems to me the clearest solution (since you didn't give your expected output, I had to guess):

$ cat /tmp/tmp.py
import multiprocessing

def calculate(x):
    # here is where I would take this input x (and maybe a couple more inputs)
    # and build a larger library of variables that I use further down the line

    pool = multiprocessing.Pool(3)
    _lst = [(x, y) for x in (x,) for y in range(3)]
    final= pool.map(domath, _lst)
    print(final)

def domath(l):
    return l[0] * l[1]

calculate(2)

$ python /tmp/tmp.py
[0, 2, 4]

$
boardrider
  • 5,882
  • 7
  • 49
  • 86