48

I am using jupyter notebook with Python3 on windows 10. My computer has 8GB RAM and at least 4GB of my RAM is free.

But when I want to make a numpy ndArray with size 6000*6000 with this command: np.zeros((6000, 6000), dtype='float64') I got this : Unable to allocate array with shape (6000, 6000) and data type float64

I don't think this could use more then 100MB RAM. I tried to change the number to see what happens. The biggest array I can make is (5000,5000). Did I make a mistake in estimating how much RAM I need?

smci
  • 32,567
  • 20
  • 113
  • 146
Amir Hossein Shahdaei
  • 1,158
  • 1
  • 8
  • 18
  • It seems to be a limitation hardcoded on your OS. I am not sure about the windows environment, but there is solution stated here: https://stackoverflow.com/questions/57507832/unable-to-allocate-array-with-shape-and-data-type that seems to be addressing the same problem on a ubuntu. I suggest you try that one. – nima Sep 16 '19 at 12:31
  • @nima: 6K*6K floats (8 bytes each) will be 36 million * 8 bytes = only **288MB**, which will be fine on most laptops with >= 4GB. But that other question is about a 3D array of total size 283 GB, i.e. 1000x larger and >> physical memory of most laptops. – smci Apr 20 '20 at 09:07
  • 2
    The obvious way to check how much memory the numpy ndarray actually takes is to run under plain Python console (not Jupyter), create the ndarray, and see how much memory was allocated. Moreover if it works under console and fails under Jupyter, then you know it's due to Jupyter or its configuration. – smci Apr 20 '20 at 10:35

2 Answers2

57

Jupyter notebook has a default memory limit size. You can try to increase the memory limit by following the steps:
1) Generate Config file using command:

jupyter notebook --generate-config
2) Open jupyter_notebook_config.py file situated inside 'jupyter' folder and edit the following property:
NotebookApp.max_buffer_size = your desired value
Remember to remove the '#' before the property value.
3) Save and run the jupyter notebook. It should now utilize the set memory value. Also, don't forget to run the notebook from inside the jupyter folder.

Alternatively, you can simply run the Notebook using below command:

 jupyter notebook --NotebookApp.max_buffer_size=your_value
Divya Kaushik
  • 820
  • 1
  • 8
  • 9
  • 6
    How do I check if the notebook started with the right config file loaded? – LuizAngioletti Mar 10 '20 at 17:33
  • 6
    Not sure if something has changed since, but this does not seem to answer does not seem to be correct. As the name suggest, the "max_buffer_size" restricts a buffer, not the overall memory usage. And it appears to be the buffer of the HTTP server (tornado). – de1 May 21 '20 at 13:23
  • 3
    How do we know which value to put ? I mean.. how do you calculate it to be the best and what things should you consider? – Ariadne R. Aug 15 '20 at 21:12
  • 2
    `The Jupyter folder is in your home directory, ~/.jupyter.` – Sabito stands with Ukraine Sep 20 '20 at 05:42
  • I don't understand how to run the notebook inside of the .jupyter folder, I don't find him in the folder list through the Notebook folder view. How do you do it ? – Lemisourd Mar 28 '22 at 07:40
  • That didn't work for me, you'll just add a limit to your configure. – milad bahari javan Jul 10 '22 at 09:52
  • Will this information be in the form of bytes, kilobytes, megabytes, gigabytes, or will it look like 100 MB or say 10 GB? – jamiel22 Aug 01 '23 at 18:17
6

For Jupyter you need to consider 2 processes:

  1. The local HTTP server (which is based on Tornado)
  2. Kernel process (normally local but can be distributed and depends on your config).

max_buffer_size is a Tornado Web Server setting, corresponds to the Maximum amount of incoming data to buffer and defaults to 100MB (104857600). (https://www.tornadoweb.org/en/stable/httpserver.html)

Based on this PR, this value seems to have been increased to 500 MB in Notebook.

Tornado HTTP server does not allow to my knowledge to define the max memory, it runs as a Python3 process.

For the kernel, you should look at the command defined kernel spec.

An option to try would be this one

gogasca
  • 9,283
  • 6
  • 80
  • 125