0

I'm building a training and test set on Jupyter Notebook but I always receive the following memory error:

"Unable to allocate 8.12 GiB for an array with shape (22370, 389604) and data type uint8"

I followed the advice from other stack overflow members (which is increasing the jupyter notebook memory limit by typing c.NotebookApp.max_buffer_size = 100000000000 in the jupyter_notebook_config.py file located in the directories when I installed my anaconda navigator package) however, I'm still encountering the same error. I'm not so sure what to do now.

Kristen Wang
  • 1
  • 1
  • 1

1 Answers1

0

For Jupyter you need to consider 2 processes:

  1. The local HTTP server (which is based on Tornado)
  2. Kernel process (normally local but can be distributed and depends on your config).

max_buffer_size is a Tornado Web Server setting, corresponds to the Maximum amount of incoming data to buffer and defaults to 100MB (104857600). (https://www.tornadoweb.org/en/stable/httpserver.html)

Based on this PR, this value seems to have been increased to 500 MB in Notebook.

Tornado HTTP server does not allow to my knowledge to define the max memory, it runs as a Python3 process.

For the kernel, you should look at the command defined kernel spec.

I would try the following:

  1. Increase physical memory
  2. Optimize how you read the data in the Pandas Dataframe?
  3. Try this one
gogasca
  • 9,283
  • 6
  • 80
  • 125