What difference between pickle and _pickle in python 3?

Question

I am new in python and want implement fast object serialization. I was trying to use json, but it was too slow, also was trying to use marshall module, but the size of the objects serialized by marshall 6-7 times more than pickle, so i decided to use pickle in my project. I read about cPickle module, read that it quite fast, but there is no such module in python 3 and docs says that module named _pickle is written in C. So in my projects i use

import _pickle as pickle

Is any difference between pickle and _pickle? How i can implement faster objects serialization/deserialization?

score 61 · Accepted Answer · edited Feb 10 '18 at 15:58

61

The pickle module already imports _pickle if available. It is the C-optimized version of the pickle module, and is used transparently.

From the pickle.py source code:

# Use the faster _pickle if possible
try:
    from _pickle import *
except ImportError:
    Pickler, Unpickler = _Pickler, _Unpickler

and from the pickle module documentation:

The pickle module has an transparent optimizer (_pickle) written in C. It is used whenever available. Otherwise the pure Python implementation is used.

In Python 2, _pickle was known as cPickle, but has been updated to allow the transparent use as an implementation detail.

edited Feb 10 '18 at 15:58

0 _

10,524
11
77
109

answered Oct 04 '13 at 22:43

Martijn Pieters

1,048,767
296
4,058
3,343

18

So python3 pickle == python2 cPickle? – m9_psy Oct 04 '13 at 22:44
1

note that in addition to it being unnecessary to import _pickle directly in Python 3, it lacks some features such as the .HIGHEST_PROTOCOL constant. – Michael Scott Asato Cuthbert Aug 05 '15 at 16:44
If I do `import pickle` in Python 2.7.6, which implementation will I get? – Moberg Dec 08 '16 at 15:59
4

@Moberg: you get the pure-Python implementation. Use `try: import cPickle as pickle`, `except ImportError: import pickle`, to use the C-accelerated implementation if available. – Martijn Pieters Dec 08 '16 at 16:01
1

@MartijnPieters Quick answer, thanks! So I can do a try/except ImportError to support C implementation for both python2 and python3 during a transition period. – Moberg Dec 08 '16 at 16:03

score 12 · Answer 2 · edited Apr 17 '15 at 16:43

From the Library Changes section of the What's New In Python 3.0 documentation:

A common pattern in Python 2.x is to have one version of a module implemented in pure Python, with an optional accelerated version implemented as a C extension; for example, pickle and cPickle. This places the burden of importing the accelerated version and falling back on the pure Python version on each user of these modules. In Python 3.0, the accelerated versions are considered implementation details of the pure Python versions. Users should always import the standard version, which attempts to import the accelerated version and falls back to the pure Python version. The pickle / cPickle pair received this treatment. The profile module is on the list for 3.1. The StringIO module has been turned into a class in the io module.

Since it is a python convention that implementation details are prepended with an underscore, cPickle became _pickle. Notably this means that if you are importing _pickle, the API has no guaranteed contract and could break backwards-compatibility in future releases of python3, as unlikely as that may be.

What difference between pickle and _pickle in python 3?

2 Answers2

Linked