Save large variables on a file in Python (without Pickle)

Asked Mar 30 '18 at 09:49

Active Mar 30 '18 at 09:49

Viewed 844 times

I looked for a working answer everywhere but up to now I haven't found one.

I have to save multiple variables, mostly dictionaries, on a file using Python. I tried to use Pickle but I had a "Memory Error" because of the size of my data, it's really really large, also if i explicit "protocol = 2".

How can I save such a huge data to a file? I have dictionaries with 7 entries, each one with a (1058694, 10, 9) numpy array, just to be clear.

Thanks in advance

asked Mar 30 '18 at 09:49

Gian Mauro Musso

1

Possible duplicate of [MemoryError with Pickle in Python](https://stackoverflow.com/questions/28068872/memoryerror-with-pickle-in-python) – Georgy Mar 30 '18 at 09:58
`np.savez` would be worth a try. Check its docs. – hpaulj Mar 30 '18 at 10:28
@hpaulj I tried it because I saw it on another question, but it gives me the same error unfortunately. Actually it uses pickle. – Gian Mauro Musso Mar 31 '18 at 12:47

Save large variables on a file in Python (without Pickle)

0 Answers0