1

I have a 320 MB, comma-separated (csv) - file. To read it in, I use

pd.read_csv(loggerfile, header = 2)

I have 8 GB of Ram (5 are free), how can this ever throw an error? 

  File "C:\Users\me\AppData\Local\Continuum\Anaconda\lib\site-packages\pandas\io\parsers.py", line 443, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "C:\Users\me\AppData\Local\Continuum\Anaconda\lib\site-packages\pandas\io\parsers.py", line 235, in _read
    return parser.read()
  File "C:\Users\me\AppData\Local\Continuum\Anaconda\lib\site-packages\pandas\io\parsers.py", line 686, in read
    ret = self._engine.read(nrows)
  File "C:\Users\me\AppData\Local\Continuum\Anaconda\lib\site-packages\pandas\io\parsers.py", line 1130, in read
    data = self._reader.read(nrows)
  File "parser.pyx", line 727, in pandas.parser.TextReader.read (pandas\parser.c:7146)
  File "parser.pyx", line 777, in pandas.parser.TextReader._read_low_memory (pandas\parser.c:7725)
  File "parser.pyx", line 1788, in pandas.parser._concatenate_chunks (pandas\parser.c:21033)
MemoryError

EDIT:

Windows 7 Enterprise 64 Bit

Anaconda 2.0.1 x86 <- perhaps x86_64 would be better?

Still the memory error occurs way before my memory cap is reached (seen in task manager), even on a 3 Gb - 32 Bit - machine.

user2366975
  • 4,350
  • 9
  • 47
  • 87
  • 2
    dupe of this: http://stackoverflow.com/questions/18282867/python-32-bit-memory-limits-on-64bit-windows. Windows can only access about 1gb of memory TOTAL on a 32-bit python, and it MUST be contiguous. read_csv uses 2x final memory (which its not a straightforward mapping of the file size). Its easily possible to run out of memory. 64-bit has not issues like this. – Jeff Jul 22 '14 at 11:58
  • So I definitly need a 64-bit-Anaconda, but not coersively a 64-bit-Windows (though recommended)? – user2366975 Jul 22 '14 at 12:00
  • Well you have 64-bit (Windows Enterprise 64 Bit), so just install 64-bit Anaconda. (which installs 64-bit python). – Jeff Jul 22 '14 at 12:05
  • Yes anyway I'll need a 64-bit-Windows, just wondered from which part the error came from. – user2366975 Jul 22 '14 at 12:14
  • didn't know you could even have a 32-bit windows now adays. – Jeff Jul 22 '14 at 12:32
  • It is an old computer at work that is going to be replaced in some weeks ;) – user2366975 Jul 22 '14 at 12:35
  • ok...sure. Yeah memory usage is kept to a minimum, but can't do much w/o at least 1-2 gb nowadays (free and in 64-bit) – Jeff Jul 22 '14 at 12:36
  • @Jeff The company I'm currently working for still has Windows in 32-bits edition, with modern computers. That's really a pain to be limited by such issues :-/ – Joël Jan 19 '16 at 10:53

0 Answers0