Most tutorials on compressing a file in Python involve immediately writing that file to disk with no intervening compressed python object. I want to know how to pickle and then compress a python object in memory without ever writing to or reading from disk.
Asked
Active
Viewed 1.3k times
17
-
You can't compress an object, AFAIK. You *can* compress the *representation* of an object (serialized, for instance). The most obvious representation would be a string, so you may want to check [this question](http://stackoverflow.com/questions/8506897/how-do-i-gzip-compress-a-string-in-python/8513981#8513981) – loopbackbee Oct 21 '13 at 17:04
-
Ok, so how do I pickle and then compress the representation in memory? – Michael Oct 21 '13 at 17:07
-
Compression with what compression technique? There are multiple to choose from. – Martijn Pieters Oct 21 '13 at 17:08
-
Just [pickle to a string](http://docs.python.org/2/library/pickle.html#pickle.dumps) then compress the string (see my previous link) – loopbackbee Oct 21 '13 at 17:08
3 Answers
30
I use this to save memory in one place:
import cPickle
import zlib
# Compress:
compressed = zlib.compress(cPickle.dumps(obj))
# Get it back:
obj = cPickle.loads(zlib.decompress(compressed))
If obj has references to a number of small objects, this can reduce the amount of memory used by a lot. A lot of small objects in Python add up because of per-object memory overhead as well as memory fragmentation.
Max
- 3,384
- 2
- 27
- 26
11
Batteries included.
>>> zlib.compress(pickle.dumps([42]))
'x\x9c\xd3\xc8)0\xe0\xf241\xe2J\xd4\x03\x00\x10A\x02\x87'
jhermann
- 2,071
- 13
- 17
