0

Example:

import pickle

def function():
    def closure():
        pass
    return closure

pickle.dumps(function())

Which yields:

AttributeError: Can't pickle local object 'function.<locals>.closure'

There is this question How to refactor method returning closure to be pickleable? which has an answer that suggests import dill as pickle - but in my case that would require monkey patching.

Since I also found some examples Can Python pickle lambda functions? on how to pickle lambda functions (which isn't support by pickle out of the box either), I have some hope that there is a way to actually pickle function() with stdlib pickle - only by refactoring above code example.

finefoot
  • 9,914
  • 7
  • 59
  • 102

1 Answers1

3

I'm the dill author. Here's the code you gave, working with dill:

>>> import dill as pickle
>>> def function():
...   def closure():
...     pass
...   return closure
... 
>>> pickle.dumps(function())
b'\x80\x03cdill._dill\n_create_function\nq\x00(cdill._dill\n_create_code\nq\x01(K\x00K\x00K\x00K\x01KSC\x04d\x00S\x00q\x02N\x85q\x03))X\x07\x00\x00\x00<stdin>q\x04X\x07\x00\x00\x00closureq\x05K\x02C\x02\x00\x01q\x06))tq\x07Rq\x08c__builtin__\n__main__\nh\x05NNtq\tRq\n}q\x0b}q\x0c(X\x0f\x00\x00\x00__annotations__q\r}q\x0eX\x0c\x00\x00\x00__qualname__q\x0fX\x19\x00\x00\x00function.<locals>.closureq\x10u\x86q\x11b.'
>>> _closure = pickle.loads(_)
>>>

I'm not sure how this requires a monkey patch... but if this is your use case, then dill will work for you. There are some special cases, such as you are using a package that is using pickle specifically, and you can't substitute dill for pickle. Your options there are: (1) find a code like multiprocess that forks multiprocessing by replacing pickle with dill, or (2) leverage the appropriate functions in dill to register the needed types in pickle using copyreg and the Pickler.dispatch table.

Mike McKerns
  • 33,715
  • 8
  • 119
  • 139
  • Hey thanks :) I apologize for the misunderstanding. My code above is only an example. The monkey patch would be required because it's exactly like you already said: The pickling happens in a package that uses `pickle` specifically. In the example, `import pickle` and `pickle.dumps` are only in the same file for simplicity - and cannot be changed. I'm sorry, I don't understand your (2). Would it be possible to apply that to the example code above? – finefoot Jul 15 '22 at 12:29
  • What `dill` is doing, essentially, is extending the types of objects that `pickle` knows how to serialize, as one would do with `copyreg`. So, you should be able, in most cases, to copy the registered serialization function from the `dill.Pickler` dispatch table to `pickle` by using `copyreg` (or otherwise). Where serialization of a closure is not possible, it's best to substitute the closure with a class that has a `__call__` method -- then the outer function is equivalent to the `__init__` method, and the inner function is equaled to the `__call__` method. – Mike McKerns Jul 15 '22 at 23:39