Questions tagged [dill]

Dill is a module which extends python's 'pickle' module for serializing and de-serializing python objects to the majority of the built-in python types. Use this tag in conjunction with the pickle and python tag for questions about object serialization with Dill.

Dill extends python's 'pickle' module for serializing and de-serializing python objects to the majority of the built-in python types.

Major features

Dill can pickle the following standard types:

  • none, type, bool, int, long, float, complex, str, unicode,
  • tuple, list, dict, file, buffer, builtin,
  • both old and new style classes,
  • instances of old and new style classes,
  • set, frozenset, array, functions, exceptions

Dill can also pickle more 'exotic' standard types:

  • functions with yields, nested functions, lambdas
  • cell, method, unboundmethod, module, code, methodwrapper,
  • dictproxy, methoddescriptor, getsetdescriptor, memberdescriptor,
  • wrapperdescriptor, xrange, slice,
  • notimplemented, ellipsis, quit

Dill cannot yet pickle these standard types:

  • frame, generator, traceback

Dill also provides the capability to:

  • save and load python interpreter sessions
  • save and extract the source code from functions and classes
  • interactively diagnose pickling errors

Current Release

The latest released version of dill is available from http://dev.danse.us/trac/pathos and https://github.com/uqfoundation/dill.

Dill is distributed under a 3-clause BSD license.

290 questions
58
votes
4 answers

What can multiprocessing and dill do together?

I would like to use the multiprocessing library in Python. Sadly multiprocessing uses pickle which doesn't support functions with closures, lambdas, or functions in __main__. All three of these are important to me In [1]: import pickle In [2]:…
MRocklin
  • 55,641
  • 23
  • 163
  • 235
32
votes
3 answers

What are the pitfalls of using Dill to serialise scikit-learn/statsmodels models?

I need to serialise scikit-learn/statsmodels models such that all the dependencies (code + data) are packaged in an artefact and this artefact can be used to initialise the model and make predictions. Using the pickle module is not an option because…
Nikhil
  • 2,230
  • 6
  • 33
  • 51
21
votes
1 answer

dill vs cPickle speed difference

I am trying to serialize thousands of objects and some of these objects are lambda objects. Since cPickle doesn't work for lambdas, I tried using dill. However, the drop in computational speed is more than 10 times when unpickleing (or undilling…
Tohiko
  • 1,860
  • 2
  • 18
  • 26
20
votes
1 answer

How to dill (pickle) to file?

The question may seem a little basic, but wasn't able to find anything that I understood in the internet. How do I store something that I pickled with dill? I have come this far for saving my construct (pandas DataFrame, which also contains custom…
Make42
  • 12,236
  • 24
  • 79
  • 155
18
votes
1 answer

How is dill different from Python's pickle module?

I have a large object in my Python3 code which, when tried to be pickled with the pickle module throws the following error: TypeError: cannot serialize '_io.BufferedReader' object However, dill.dump() and dill.load() are able to save and restore…
sherlock
  • 2,397
  • 3
  • 27
  • 44
15
votes
2 answers

How to use dill to serialize a class definition?

In the answer to Python pickle: dealing with updated class definitions, the author of the dill package writes: "Ok, I have added this feature to dill in the latest revision on github. Implemented with far less trickery than I thought... just…
user8765
  • 151
  • 1
  • 1
  • 3
14
votes
2 answers

TypeError: can't pickle dict_items objects

Why does pickle.dumps({}.items()) fail with TypeError: can't pickle dict_items objects in Python 3.5.2 but not in Python 2.7.12? "Pickling" the dictionary with pickle.dumps({}) works in both Python versions (and in Python 2.7.12 gives the same…
fuenfundachtzig
  • 7,952
  • 13
  • 62
  • 87
14
votes
1 answer

Serialize a python function with dependencies

I have tried multiple approaches to pickle a python function with dependencies, following many recommendations on StackOverflow, (such as dill, cloudpickle, etc.) but all seem to run into a fundamental issue that I cannot figure out. I have a main…
Prasanna
  • 381
  • 2
  • 9
10
votes
2 answers

Python 3.5 dill pickling/unpickling on different servers: "KeyError: 'ClassType'"

See updates at the bottom -- A similar question was asked here, but never resolved: pickling and unpickling user-defined class I'm working on a project which necessitates pickling user defined classes, and sending them to a remote server where they…
wheelium
  • 290
  • 2
  • 9
9
votes
3 answers

Using python lime as a udf on spark

I'm looking to use lime's explainer within a udf on pyspark. I've previously trained the tabular explainer, and stored is as a dill model as suggested in link loaded_explainer = dill.load(open('location_to_explainer','rb')) def…
ArunK
  • 1,731
  • 16
  • 35
9
votes
2 answers

Cannot pickle lambda function in python 3

Using dill pickling of lambda functions work fine in Python 2, but not in Python 3, is there any alternative? Python 3: import dill import pickle pickle.dumps(lambda x: x**2) pickle.PicklingError: Can't pickle at 0x104e97840>: attribute lookup…
rajat
  • 3,415
  • 15
  • 56
  • 90
9
votes
1 answer

Pickle/dill cannot handle circular references if __hash__ is overridden

Consider the following MWE: #import dill as pickle # Dill exhibits similar behavior import pickle class B: def __init__(self): self.links = set() class A: def __init__(self, base: B): self.base = base …
Pavel Kirienko
  • 1,162
  • 1
  • 15
  • 31
9
votes
2 answers

Why is dill much faster and more disk-efficient than pickle for numpy arrays

I'm using Python 2.7 and NumPy 1.11.2, as well as the latest versions of dill ( I just did the pip install dill) , on Ubuntu 16.04. When storing a NumPy array using pickle, I find that pickle is very slow, and stores arrays at almost three times…
Bananach
  • 2,016
  • 26
  • 51
8
votes
3 answers

Pickle class definition in module with dill

My module contains a class which should be pickleable, both instance and definition I have the following structure: MyModule |-Submodule |-MyClass In other questions on SO I have already found that dill is able to pickle class definitions and…
vriesdemichael
  • 914
  • 1
  • 7
  • 19
8
votes
1 answer

Serializing custom modules together with object in python

Problem Let's say I have this module named custom_module: class CustomClass: pass And I use this class in a script that serializes an object of the class I defined in custom_module: import cloudpickle import custom_module as cm custom_object =…
gsmafra
  • 2,434
  • 18
  • 26
1
2 3
19 20