Can Pickle handle files larger than the RAM installed on my machine?

Question

I'm using pickle for saving on disk my NLP classifier built with the TextBlob library.

I'm using pickle after a lot of searches related to this question. At the moment I'm working locally and I have no problem loading the pickle file (which is 1.5Gb) with my i7 and 16gb RAM machine. But the idea is that my program, in the future, has to run on my server which only has 512Mb RAM installed.

Can pickle handle such a large file or will I face memory issues?

On my server I've got Python 3.5 installed and it is a Linux server (not sure which distribution).

I'm asking because at the moment I can't access my server, so I can't just try and find out what happens, but at the same time I'm doubtful if I can keep this approach or I have to find other solutions.

Imho that you cannot even have variables which refer to things larger than RAM, can you? — palsch, Nov 27 '15 at 21:36
@palsch Honestly i've got no idea, i'm not really expert on how pickle works, or if my machine can use virtual memory if ram ends. — Nico, Nov 27 '15 at 21:43
Is your solution here? http://deeplearning.net/software/theano/tutorial/python-memory-management.html#pickle — palsch, Nov 27 '15 at 21:49
Do you want to accept skrrgwasme's answer? Imho it's enought. — palsch, Nov 27 '15 at 22:11

score 10 · Accepted Answer · edited May 23 '17 at 11:51

Unfortunately this is difficult to accurately answer without testing it on your machine.

Here are some initial thoughts:

There is no inherent size limit that the Pickle module enforces, but you're pushing the boundaries of its intended use. It's not designed for individual large objects. However, you since you're using Python 3.5, you will be able to take advantage of PEP 3154 which adds better support for large objects. You should specify pickle.HIGHEST_PROTOCOL when you dump your data.
You will likely have a large performance hit because you're trying to deal with an object that is 3x the size of your memory. Your system will probably start swapping, and possibly even thrashing. RAM is so cheap these days, bumping it up to at least 2GB should help significantly.
To handle the swapping, make sure you have enough swap space available (a large swap partition if you're on Linux, or enough space for the swap file on your primary partition on Windows).
As pal sch's comment shows, Pickle is not very friendly to RAM consumption during the pickling process, so you may have to deal with Python trying to get even more memory from the OS than the 1.5GB we may expect for your object.

Given these considerations, I don't expect it to work out very well for you. I'd strongly suggest upgrading the RAM on your target machine to make this work.

score 2 · Answer 2 · answered Nov 27 '15 at 22:01

2

I don't see how you could load an object into RAM that exceeds the RAM. i.e. bytes(num_bytes_greater_than_ram) will always raise an MemoryError.

answered Nov 27 '15 at 22:01

Jashandeep Sohi

4,903
2
23
25

Maybe pickle variables refer to the drive or so? – palsch Nov 27 '15 at 22:09
By default `pickle` does not [pickle references](https://docs.python.org/3.5/library/pickle.html#what-can-be-pickled-and-unpickled). If the object generated by the TextBlob library is disk backed and implements a custom pickling interface, then *maybe*. – Jashandeep Sohi Nov 27 '15 at 22:27
As an example try ``pickle.loads(pickle.dumps(memoryview(b"abc")))``. – Jashandeep Sohi Nov 27 '15 at 22:28
Ok. So +1 from me. ;-) – palsch Nov 27 '15 at 22:28

Can Pickle handle files larger than the RAM installed on my machine?

2 Answers2