Why would I get a memory error with pandas?

Asked Mar 30 '18 at 01:12

Active Mar 30 '18 at 01:12

Viewed 61 times

I am trying to load a file from https://data.medicare.gov/Physician-Compare/Physician-Compare-National-Downloadable-File/mj5m-pzi6 into a pandas df. My computer has 16gb ram while this file is less than 800mb as a csv.

I am able to load 1 million of the 2.6 million rows, but I cannot load the whole csv. When I try and load the whole csv, I get a MemoryError:. However, when I look at my memory being used in task manager, it doesn't go over 34% (which means I should have about 10 GB available), but nevertheless I continue to get an error.

I am able to run the code below when I add nrows=1000000

df = pd.read_csv('Physician_Compare_National_Downloadable_File.csv',encoding="utf-8",
             engine='python')

I know pandas needs about 10X the ram compared to the size of the file, so I don't get how 16GB is not enough. Is there something I need to do to optimize my PC or python to use the whole 16GB?

asked Mar 30 '18 at 01:12

CandleWax

2,159
2
28
46

2

what does `python -c "import sys; print(sys.version)"` say? – Stephen Rauch Mar 30 '18 at 01:19
3.6.2 (v3.6.2:5fd33b5, Jul 8 2017, 04:14:34) [MSC v.1900 32 bit (Intel)] – CandleWax Mar 30 '18 at 01:21
3

As I suspected, you are using the 32bit version. It can not address as much memeory. Use a 64 bit version. – Stephen Rauch Mar 30 '18 at 01:22
1

You are using a 32 bit Python interpreter, which can only address about 2^32 bits (about 4 gigs) of memory. Most operating systems limit that further, actually – juanpa.arrivillaga Mar 30 '18 at 01:26
wow. clearly I wasn't paying attention when I installed python. Thanks! – CandleWax Mar 30 '18 at 01:41

Why would I get a memory error with pandas?

0 Answers0