Common use-cases for pickle in Python

Question

I've looked at the pickle documentation, but I don't understand where pickle is useful.

What are some common use-cases for pickle?

The part I don't get about pickling is why don't you just save the value to a file? Why pickle it? — whackamadoodle3000, Aug 11 '17 at 04:30

Dave Kirby · Accepted Answer · 2018-09-15T09:01:11.267

69

Some uses that I have come across:

1) saving a program's state data to disk so that it can carry on where it left off when restarted (persistence)

2) sending python data over a TCP connection in a multi-core or distributed system (marshalling)

3) storing python objects in a database

4) converting an arbitrary python object to a string so that it can be used as a dictionary key (e.g. for caching & memoization).

There are some issues with the last one - two identical objects can be pickled and result in different strings - or even the same object pickled twice can have different representations. This is because the pickle can include reference count information.

To emphasise @lunaryorn's comment - you should never unpickle a string from an untrusted source, since a carefully crafted pickle could execute arbitrary code on your system. For example see https://blog.nelhage.com/2011/03/exploiting-pickle/

edited Sep 15 '18 at 09:01

answered Aug 09 '10 at 12:21

Dave Kirby

25,806
5
67
84

31

One should not transfer pickled objects over network or other untrusted channels, unless the pickled data is carefully secured against manipulation. The pickle documentation explicitly warns to *never* unpickle data from untrusted or unauthenticated sources. – Aug 09 '10 at 13:12
4

@lunaryorn: good point. If you are going to transfer pickled data between machines then use a secure channel such as SSL or SSH tunnelling. – Dave Kirby Aug 09 '10 at 14:20
3

Then you are still trusting the endpoint not to exploit you, which may or not be okay, depending on context. – L̲̳o̲̳̳n̲̳̳g̲̳̳p̲̳o̲̳̳k̲̳̳e̲̳̳ Aug 20 '10 at 17:54
@lunaryorn - good point but in that case how can we encrypt data in public domain. do we have to use some other py lib or not to use pickle – Pardeep Sharma Apr 05 '18 at 07:32
1

Is point 4) true? I found this which has some (old) evidence that wouldn't work [here](http://www.aminus.org/blogs/index.php/2007/11/03/pickle_dumps_not_suitable_for_hashing?blog=2). – salotz Sep 14 '18 at 21:46
1

@salotz see the following paragraph where I note that pickling the same datastructure twice may result in different strings. Whether this is an issue or not depends on the context. If you using it for a cache key to improve performance then the occasional cache miss may not be significant. YMMV. – Dave Kirby Sep 15 '18 at 08:55
Are you talking about [this](https://docs.python.org/3.5/library/pickle.html#persistence-of-external-objects)? I am not looking to have the cache you describe I am looking to hash them as identifiers, and I am trying to figure out if and how that can be done. I'm guessing if there are no references then it should be okay. I just don't see this clearly documented anywhere, how did you figure this out? – salotz Sep 17 '18 at 17:06
Using pickle to get an identifier as in 4) is almost always the wrong pattern. As in, folks who do this already know exactly what they are doing with pickle and why. As Dave Kirby mentions, this is fine when you can accept a cache miss, and your pickling time is fast enough that it's worth the tradeoff. *Most* of the time (e.g. novice to middling experience python devs) you want to do something to implement `__hash__()`. This allows it to be used as keys in a dict/set. – DeusXMachina Dec 13 '21 at 21:07
3) is not a great example, IMHO. One should think long and hard about why they would do this, and whether they can't just convert it to a database row, or json-encoded string. It can be a complete pain to deal with pickles stored long-term in a database. Basically willingness to use pickle should be inverse to the expected lifetime of the data. – DeusXMachina Dec 13 '21 at 21:15

Jacob Oscarson · Answer 2 · 2010-08-09T12:08:37.600

Minimal roundtrip example..

>>> import pickle
>>> a = Anon()
>>> a.foo = 'bar'
>>> pickled = pickle.dumps(a)
>>> unpickled = pickle.loads(pickled)
>>> unpickled.foo
'bar'

Edit: but as for the question of real-world examples of pickling, perhaps the most advanced use of pickling (you'd have to dig quite deep into the source) is ZODB: http://svn.zope.org/

Otherwise, PyPI mentions several: http://pypi.python.org/pypi?:action=search&term=pickle&submit=search

I have personally seen several examples of pickled objects being sent over the network as an easy to use network transfer protocol.

score 9 · Answer 3 · answered Apr 14 '17 at 15:51

Pickle is like "Save As.." and "Open.." for your data structures and classes. Let's say I want to save my data structures so that it is persistent between program runs.

Saving:

with open("save.p", "wb") as f:    
    pickle.dump(myStuff, f)

Loading:

try:
    with open("save.p", "rb") as f:
        myStuff = pickle.load(f)
except:
    myStuff = defaultdict(dict)

Now I don't have to build myStuff from scratch all over again, and I can just pick(le) up from where I left off.

score 8 · Answer 4 · answered Oct 14 '13 at 12:31

Pickling is absolutely necessary for distributed and parallel computing.

Say you wanted to do a parallel map-reduce with multiprocessing (or across cluster nodes with pyina), then you need to make sure the function you want to have mapped across the parallel resources will pickle. If it doesn't pickle, you can't send it to the other resources on another process, computer, etc. Also see here for a good example.

To do this, I use dill, which can serialize almost anything in python. Dill also has some good tools for helping you understand what is causing your pickling to fail when your code fails.

And, yes, people use picking to save the state of a calculation, or your ipython session, or whatever.

score 8 · Answer 5 · answered Aug 09 '10 at 09:34

I have used it in one of my projects. If the app was terminated during it's working (it did a lengthy task and processed lots of data), I needed to save the whole data structure and reload it after the app was run again. I used cPickle for this, as speed was a crucial thing and the size of data was really big.

score 4 · Answer 6 · edited May 23 '17 at 11:54

4

For the beginner (as is the case with me) it's really hard to understand why use pickle in the first place when reading the official documentation. It's maybe because the docs imply that you already know the whole purpose of serialization. Only after reading the general description of serialization have I understood the reason for this module and its common use cases. Also broad explanations of serialization disregarding a particular programming language may help: https://stackoverflow.com/a/14482962/4383472, What is serialization?, https://stackoverflow.com/a/3984483/4383472

edited May 23 '17 at 11:54

Community

1
1

answered Mar 10 '15 at 00:11

Bad

4,967
4
34
50

your "answer" is not an answer, it's more of a comment. The OP's question is "What are some common use-cases for pickle?". Do you feel you have answered that question in any way? – Mike McKerns Mar 10 '15 at 01:41
4

well, I feel that I have answered the question because I also had difficulties to understand common uses of pickle when I tried to read about this module [here](https://docs.python.org/3.4/library/pickle.html), [here](http://en.wikipedia.org/wiki/Pickle_(Python)) and [here](http://pymotw.com/2/pickle/index.html). Because mostly they begin to explain what pickle does assuming that you know the motivation behind the whole concept of serialization. After I read simple wiki article on serialization I grasped the general idea as well as "common cases". Maybe it'll help somebody... – Bad Mar 10 '15 at 09:50
and some of those common cases are…? If there are some that are not listed here in other answers… adding them to your answer would be very appropriate. – Mike McKerns Mar 11 '15 at 00:21

score 3 · Answer 7 · answered Jul 31 '12 at 18:28

3

To add a real-world example: The Sphinx documentation tool for Python uses pickle to cache parsed documents and cross-references between documents, to speed up subsequent builds of the documentation.

answered Jul 31 '12 at 18:28

score 1 · Answer 8 · answered May 30 '16 at 17:01

1

I can tell you the uses I use it for and have seen it used for:

Game profile saves
Game data saves like lives and health
Previous records of say numbers inputed to a program

Those are the ones I use it for at least

answered May 30 '16 at 17:01

Austin Gummy

43
8

score 1 · Answer 9 · edited Feb 12 '21 at 23:10

1

I use pickling during web scraping one of website at that time I want to store more than 8000k urls and want to process them as fast as possible so I use pickling because its output quality is very high.

you can easily reach to url and where you stop even job directory key word also fetch url details very fast for resuming the process.

edited Feb 12 '21 at 23:10

DisappointedByUnaccountableMod

6,656
4
18
22

answered Jul 20 '18 at 12:59

Divanshu Tak

71
7

Common use-cases for pickle in Python

9 Answers9

Linked

Related