Can't pickle defaultdict

Question

I have a defaultdict that looks like this:

dict1 = defaultdict(lambda: defaultdict(int))

The problem is, I can't pickle it using cPickle. One of the solution that I found here is to use module-level function instead of a lambda. My question is, what is module-level function? How can I use the dictionary with cPickle?

Answer 1

In addition to Martijn's explanation:

A module-level function is a function which is defined at module level, that means it is not an instance method of a class, it's not nested within another function, and it is a "real" function with a name, not a lambda function.

So, to pickle your defaultdict, create it with module-level function instead of a lambda function:

def dd():
    return defaultdict(int)

dict1 = defaultdict(dd) # dd is a module-level function

than you can pickle it

tmp = pickle.dumps(dict1) # no exception
new = pickle.loads(tmp)

Answer 2

Pickle wants to store all the instance attributes, and defaultdict instances store a reference to the default callable. Pickle recurses over each instance attribute.

Pickle cannot handle lambdas; pickle only ever handles data, not code, and lambdas contain code. Functions can be pickled, but just like class definitions only if the function can be imported. A function defined at the module level can be imported. Pickle just stores a string in that case, the full 'path' of the function to be imported and referenced when unpickling again.

Answer 3

You can however use partial to accomplish this:

>>> from collections import defaultdict
>>> from functools import partial
>>> pickle.loads(pickle.dumps(defaultdict(partial(defaultdict, int))))
defaultdict(<functools.partial object at 0x94dd16c>, {})

Answer 4

To do this, just write the code you wanted to write. I'd use dill, which can serialize lambdas and defaultdicts. Dill can serialize almost anything in python.

>>> import dill
>>> from collections import defaultdict
>>>
>>> dict1 = defaultdict(lambda: defaultdict(int))
>>> pdict1 = dill.dumps(dict1)
>>> _dict1 = dill.loads(pdict1)
>>> _dict1
defaultdict(<function <lambda> at 0x10b31b398>, {})

Answer 5

Solution that still works as a one-liner for this case, and is actually more efficient than the lambda (or an equivalent def-ed) function to boot:

dict1 = defaultdict(defaultdict(int).copy)

That just makes a template defaultdict(int), and binds its copy method as the default factory for the outer defaultdict. Everything in there is picklable, and on CPython (where defaultdict is a built-in type implemented in C) it's more efficient than invoking any user-defined function to do the same job. No need for extra imports, wrapping, etc.

Answer 6

dict1 = defaultdict(lambda: defaultdict(int))
cPickle.dump(dict(dict1), file_handle)

worked for me

Answer 7

Implementing the anonymous lambda function by a normal function worked for me. As pointed out by Mike, Pickle cannot handle lambdas; pickle only handles data. Hence, converting the defaultdict method from:

    dict_ = defaultdict(lambda: default_value)

to:

    def default_():
        return default_value

and then creating the default dict as follows worked for me:

    dict_ = defaultdict(default_)

Answer 8

If you don't care about preserving the defaultdict type, convert it:

fname = "file.pkl"

for value in nested_default_dict:
    nested_default_dict[value] = dict(nested_default_dict[value])
my_dict = dict(nested_default_dict)

with open(fname, "wb") as f:
    pickle.dump(my_dict, f)  # Now this will work

I think this is a great alternative since when you are pickling, the object is probably in it's final form... AND, if really do need the defaultdict type again, you can simply convert is back after you unpickle:

for value in my_dict:
    my_dict[value] = defaultdict(type, my_dict[value])
nested_default_dict = defaultdict(type, my_dict)

Answer 9

I'm currently doing something similar to the question poser, however, I'm using a subclass of defaultdict which has a member function that is used as the default_factory. In order to have my code work properly (I required the function to be defined at runtime), I simply added some code to prepare the object for pickling.

Instead of:

...
pickle.dump(dict, file)
...

I use this:

....
factory = dict.default_factory
dict.default_factory = None
pickle.dump(dict, file)
dict.default_factory = factory
...

This isn't the exact code I used as my tree is an object which creates instances of the same the tree's type as indexes are requested (so I use a recursive member function to do the pre/post pickle operations), but this pattern also answers the question.

Answer 10

Here is a function for an arbitrary base defaultdict for an arbitrary depth of nesting.

def wrap_defaultdict(instance, times):
    """Wrap an instance an arbitrary number of `times` to create nested defaultdict.
    
    Parameters
    ----------
    instance - e.g., list, dict, int, collections.Counter
    times - the number of nested keys above `instance`; if `times=3` dd[one][two][three] = instance
    
    Notes
    -----
    using `x.copy` allows pickling (loading to ipyparallel cluster or pkldump)
        - thanks https://stackoverflow.com/questions/16439301/cant-pickle-defaultdict
    """
    from collections import defaultdict

    def _dd(x):
        return defaultdict(x.copy)

    dd = defaultdict(instance)
    for i in range(times-1):
        dd = _dd(dd)

    return dd