Memory error in pandas

Question

I have a csv file which has a size of around 800MB which I'm trying to load into a dataframe via pandas but I keep getting a memory error. I need to load it so I can join it to another smaller dataframe.

Why am I getting a memory error even though I'm using 64bit versions of Windows, and Python 3.4 64bit and have over 8GB of RAM and plenty of harddisk? Is this is a bug in Pandas? How can I solve this memory issue?

Possible duplicate of [Memory error when using pandas read\_csv](http://stackoverflow.com/questions/17557074/memory-error-when-using-pandas-read-csv) — hashcode55, Jun 15 '16 at 13:36
You already have two questions about this: [here](https://stackoverflow.com/questions/37834904/merge-csv-files-too-large-for-pandas) and [here](https://stackoverflow.com/questions/37756991/best-way-to-join-two-large-datasets-in-pandas) stop reposting — Noelkd, Jun 16 '16 at 16:00

score 1 · Answer 1 · answered Jun 15 '16 at 20:51

1

reading your CSV in chunks might help:

chunk_size = 10**5
df = pd.concat([chunk for chunk in pd.read_csv(filename, chunksize=chunk_size)],
               ignore_index=False)

answered Jun 15 '16 at 20:51

MaxU - stand with Ukraine

205,989
36
386
419

That might help but won't solve the problem that a merge will kill it. Pandas is incredibly wasteful with memory – Nickpick Jun 15 '16 at 20:53
@nickpick, so what is your problem - reading 800MB CSV file or merging your DF with another smaller one? – MaxU - stand with Ukraine Jun 15 '16 at 20:56
Both is causing problems. The fact that reading in chunks and concatenating it makes a difference at all, points to issues in pandas – Nickpick Jun 15 '16 at 20:57
@nickpick, did you try to read your CSV file in chunks? if yes what shows `df.info()` after it's done? – MaxU - stand with Ukraine Jun 15 '16 at 21:00

Memory error in pandas

1 Answers1

Linked