Pandas join memoryError

Asked Aug 22 '14 at 12:34

Active Aug 22 '14 at 12:35

Viewed 828 times

I'm trying to join 2 dataframes, essentially trying to replicate a vlookup in python. The command I'm running is

df_Eligible_w_Hier = df_Eligible.join(df_ProdHier, on='Sku', how='left',lsuffix='_ProdHier')

df_Eligible has 13949037 rows and df_ProdHier has 31897.

I'm using a Win7 core i5 with 4Gb of memory.

I keep getting a MemoryError, is that normal. Those df don't seem that big to me. How can I solve or work around this?

Thanks

edited Aug 22 '14 at 12:35

EdChum

asked Aug 22 '14 at 12:34

Alex Kinman

Are you running 32 or 64-bit version of python? – EdChum Aug 22 '14 at 12:36
platform.architecture() ('32bit', 'WindowsPE') – Alex Kinman Aug 22 '14 at 13:36
I don't know if it should run out of memory or not but I would try with 64-bit python to see it works. – EdChum Aug 22 '14 at 13:37
I suspect this depends on how many rows share a sku number (as this can explode - i.e. if 2 rows join with two rows then you get 4 rows etc.) – Andy Hayden Aug 23 '14 at 00:07
each sku corresponds to one row in df_ProdHier and several hundred rows in df_Eligible – Alex Kinman Aug 23 '14 at 01:42

0 Answers0