Why does numpy crash instantly upon an array insertion?

Question

I've come across some weird behaviour in a project of mine. Specifically, when this code is run:

import numpy as np 

coefficientMatrix = np.zeros([12500, 43750])
coefficientMatrix[229, 798] = 1.0942131827

my Python process crashes:

error message

What can be wrong here?

System specs (in case that's relevant here): Windows 7 x64, 8Gb of RAM, Python 2.7 32-bit, numpy 1.9.2.

@arodriguezdonaire By "it crashes" I mean that windows's "python.exe has stopped working" prompt comes up. — Bartvbl, Sep 04 '15 at 10:47
No problem on my system. Python 2.7 (conda-accelerate) 5 GB Ram, LInux 3.16.0-46 kernel. — Moritz, Sep 04 '15 at 10:47
It looks like the array size exceeds the maximum. Take a look at this thread: http://stackoverflow.com/questions/855191/how-big-can-a-python-array-get — Edwin Torres, Sep 04 '15 at 13:17
@ali_m You are entirely correct: I installed the 64-bit version and it ran fine :) — Bartvbl, Sep 04 '15 at 13:52
Could you edit your question to mention the `MemoryError` you must be getting? — ali_m, Sep 04 '15 at 13:55
@ali_m: I'm not, I just get the "Python.exe has stopped working" dialog with nothing being printed to the terminal. I have to assume this is due to the exception itself occurring inside numpy which then doesn't do anything with it, crashing the whole python executable instead. — Bartvbl, Sep 04 '15 at 13:58
@ali_m: From the command line. Here's a screen showing it: http://i.imgur.com/3uLwPXA.png — Bartvbl, Sep 04 '15 at 14:15
That is a little odd. I assume the lack of a traceback must be due to some Windows-specific issue (I get a `ValueError: array is too big.` with a proper traceback in a 32bit Linux VM). At any rate, I'd highly recommend using an interactive Python interpreter (e.g. [IPython](http://ipython.org/)) for debugging. — ali_m, Sep 04 '15 at 14:35

score 2 · Accepted Answer · edited May 23 '17 at 10:30

The reason why you would get a MemoryError when you assign to an element in coefficientMatrix, rather than when you create the array using np.zeros, is that most OSs (including Windows 7) use lazy memory allocation.

When you instantiate the array using np.zeros, Windows only allocates virtual memory address space rather than physical RAM. However, when you actually try to write to that chunk of memory then the OS will need to find enough physical memory to hold the array. If it fails to do so, you will get a MemoryError.

Since your Python process is 32bit, it can address a maximum of 4GB of memory (and possibly even less). A 12500x43750 array of float64 values will take up 4.375GB of memory, so you simply can't have an array that big using 32bit Python.

Why does numpy crash instantly upon an array insertion?

1 Answers1