2

I just started a new job and have been given a Dell XPS 13 7390 laptop and it is really struggling with big data files / processing.

I'm currently working with a 1.5gb csv and I get a memory error when I try to open it with Python in a Jupyter Notebook.

Error tokenizing data. C error: out of memory

I was sure I'd opened files like this with ease on my personal laptop, a 10 year old Macbook, so i tested it on the same file and it opened.

Why is my Dell laptop struggling despite having lots of RAM available? Could settings be adjusted to allocate more memory to Jupyter Notebooks? What tests could I run to look into this further?

Hardware details below. The obvious difference is processor speed - does that explain it?

Dell laptop:

Ram- 16gb

Processor- Intel Core i7-10510U CPU @ 1.80GHz

Macbook:

Ram- 4gb

Processor- 2.7 GHz Intel Core i7

Code used to open the file:

import pandas as pd
data = pd.read_csv('data.csv')
data.shape

The shape = (2250493, 218)

Ben
  • 31
  • 5
  • Have you tried watching your ram in task manager while it opens? – AwesomeCronk Apr 07 '20 at 16:50
  • Can you share the code you used to open the file? – AMC Apr 07 '20 at 20:10
  • @AMC code added. – Ben Apr 08 '20 at 08:36
  • Have a look here: https://stackoverflow.com/questions/57948003/jupyter-notebook-memory-limit – ctenar Apr 08 '20 at 08:41
  • @AwesomeCronk memory usage is averaging 10% and peaked at 23% when I ran the code – Ben Apr 08 '20 at 08:59
  • thanks @ctenar this looks useful – Ben Apr 08 '20 at 09:08
  • @ctenar I tried updating the config file and ran command line update but neither have work. > MemoryError: Unable to allocate 32.0 KiB for an array with shape (4096,) and data type int64 – Ben Apr 08 '20 at 09:38
  • Hm, too bad. I presume you have made sure you installed the 64-bit version of Python and also started the Jupyter-kernel from scratch to be sure no other data is lingering around in memory. Have you tried doing the same operation outside of a Jupyter notebook? – ctenar Apr 08 '20 at 12:58
  • @ctenar somehow I had 32bit python installed... I've switched to 64bit and it works fine. Many thanks for your help! – Ben Apr 09 '20 at 07:48

1 Answers1

1

I had 32bit python installed... I've switched to 64bit and it works fine

Ben
  • 31
  • 5