Pandas memory error when saving DataFrame to file

Question

I finnally managed to join two big DataFrames on a big machine of my school (512G memory). At the moment we re two people using the same machine, the other one is using about 120G of the memory, after I called the garbage collecter we get to 420G.

I want to save the DataFrame to memory so I then I can reuse it easily and move it on another machine, I have tried to export it to a parquet file, but I get a memory error...

So how can I manage to dump that Dataframe on the hard drive in purpose of reusing it without running into memory error when memory is already near full ?

Thank you

You can play with the `chunksize` parameter in `to_csv` or similar methods. Setting a lower `chunksize` may use less memory. — hilberts_drinking_problem, Nov 30 '18 at 20:51

score 0 · Answer 1 · answered Nov 30 '18 at 20:18

0

There are several options. You can pickle the dataframe or you can use hdf5 format. These will occupy less memory. Also when you load it next time, it would be quicker then other formats.

answered Nov 30 '18 at 20:18

Vikika

318
1
9

Parquet use more memory to create than hdf5? Also parquet is quite fast to load! I try hdf5 and see if it solves the problem... – Nicolas Scotto Di Perto Nov 30 '18 at 20:21
You can use gzip as well to reduce the size. But this can reduce the efficiency in loading it the next time. – Vikika Nov 30 '18 at 20:25
Using gzip would cost me more memory no? I am talking about RAM here. I just wan't to dump the DataFrame on the hard drive. – Nicolas Scotto Di Perto Nov 30 '18 at 20:27
using gzip will result in less memory on the disk. You can try that. I have tried once while dumping the pickel file. That saved me a lot of memory and time in saving as well – Vikika Nov 30 '18 at 20:29
A question abut `to_hdf` (I am new to that kind of formats), is the path a path to the file? What should I specify as the key? – Nicolas Scotto Di Perto Nov 30 '18 at 20:29
Yeah sure, but if I get a RAM memory error just by trying to write to a file I think it won't help to add a compression because it would need more memory. – Nicolas Scotto Di Perto Nov 30 '18 at 20:31
https://stackoverflow.com/questions/20928136/input-and-output-numpy-arrays-to-h5py. You can see this link to save hdf file and load as well – Vikika Nov 30 '18 at 20:31
why not using `pd.to_hdf`? – Nicolas Scotto Di Perto Nov 30 '18 at 20:34
Are you using a 32-bit or 64-bit Python distribution? – jeschwar Nov 30 '18 at 20:38
I am using a 64-bit distribution, the Anaconda one. – Nicolas Scotto Di Perto Nov 30 '18 at 20:45
Have you considered converting any `float64` columns to `float32` to reduce memory? – jeschwar Nov 30 '18 at 20:59
https://www.dataquest.io/blog/pandas-big-data/ You can follow this link ton reduce the size of dataframe – Vikika Nov 30 '18 at 21:02

score 0 · Answer 2 · answered Nov 30 '18 at 20:31

I'm not sure how it would perform with a dataset that large but you can use the pandas function to_csv to save the file to the hard drive.

df.to_csv("filename.csv")

If you're going to be working with that much data in the future I might suggest a chunking approach like the one mentioned here: https://stackoverflow.com/a/25962187/4852976

Pandas memory error when saving DataFrame to file

2 Answers2