8

I am very confused by what is reported by numpy.ndarray.nbytes.

I just created an identity matrix of size 1 million (10^6), which therefore has 1 trillion rows (10^12). Numpy reports that this array is 7.28TB, yet the python process only uses 3.98GB of memory, as reported by OSX activity monitor.

  • Is the whole array contained in memory?
  • Does Numpy somehow compress its representation, or is that handled by the OS?
  • If I simply calculate y = 2 * x, which should be the same size as x, the process memory increases to about 30GB, until it gets killed by the OS. Why, and what kind of operations can I conduct on x without the memory usage expanding so much?

This is the code I used:

import numpy as np
x = np.identity(1e6)
x.size
# 1000000000000
x.nbytes / 1024 ** 4
# 7.275957614183426
y = 2 * x
# python console exits and terminal shows: Killed: 9
Rems
  • 4,837
  • 3
  • 27
  • 24
  • On windows neither 32bit nor 64 bit version allowed me to allocate such a big matrix – Tamas Hegedus Feb 13 '16 at 17:01
  • 2
    My guess is that it's related to [this](http://stackoverflow.com/q/27574881/1461210), although I'm a little surprised that it works for identity matrices rather than just zeros. – ali_m Feb 13 '16 at 17:02
  • @TamasHegedus what if you create an identity matrix of size `2e4`? It should take 2.98GB of RAM. Does the process take up that much RAM? – Rems Feb 13 '16 at 17:04

2 Answers2

5

On Linux (and I'm assuming the same thing happens under Mac), when a program allocates memory, the OS doesn't actually allocate it until it uses it.

If the program never uses the memory, then the OS doesn't have to waste RAM on it, but it does put the OS in a spot when the program has requested a ton of memory and actually needs to use it, but the OS doesn't have enough.

When that happens, the OS may either start killing off minor other processes and give their RAM to the requesting process, or just kill off the requesting process (which is what is happening now).

The initial 4GB of memory that Python uses is likely the pages where numpy set the 1 on the identity matrix; the rest of the pages haven't been used yet. Doing a math operation like 2*x starts accessing (and thus alloocating) all the pages until the OS runs out of memory and kills your process.

Colonel Thirty Two
  • 23,953
  • 8
  • 45
  • 85
1

The system allocates memory only virtual, only the first time your write to this memory, it gets actually physically used. For your example, you allocate 1 trillion numbers which corresponds to 2 billion memory pages, but only 1 million (1e6) of these pages are used to write the ones on the diagonal. That are exactly the 4GB of memory you see.

Daniel
  • 42,087
  • 4
  • 55
  • 81