0

How should I understand this: My 16G RAM machine can generate a matrix of size 10^12, which is 1000G, and then do assignment on its elements?

Below, I let N be 10^6 and can generate a matrix of size 10^12. I can then assign values to the matrix element as we usually do. But there will be a memory issue if I want to do an arithmetic operation (shown below).

 In [103]: N=int(10**6)

In [104]: A=np.zeros((N,N))

In [105]: A
Out[105]:
array([[0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.],
       ...,
       [0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.]])

In [106]: A[0,0]=3.14

In [107]: A
Out[107]:
array([[3.14, 0.  , 0.  , ..., 0.  , 0.  , 0.  ],
       [0.  , 0.  , 0.  , ..., 0.  , 0.  , 0.  ],
       [0.  , 0.  , 0.  , ..., 0.  , 0.  , 0.  ],
       ...,
       [0.  , 0.  , 0.  , ..., 0.  , 0.  , 0.  ],
       [0.  , 0.  , 0.  , ..., 0.  , 0.  , 0.  ],
       [0.  , 0.  , 0.  , ..., 0.  , 0.  , 0.  ]])
In [108]: A = A+1
zsh: killed     ipython
zell
  • 9,830
  • 10
  • 62
  • 115
  • 1
    It's called Virtual Memory. – Mark Ransom Aug 19 '22 at 03:56
  • @MarkRansom Could you elaborate a bit? For example, why I could do assignment of elements, but not doing any arithmetic operations with this kind of matrice that are stored in Virtual Memory? – zell Aug 19 '22 at 04:01
  • 1
    Does this answer your question? [Why doesn't numpy.zeros allocate all of its memory on creation? And how can I force it to?](https://stackoverflow.com/questions/51314255/why-doesnt-numpy-zeros-allocate-all-of-its-memory-on-creation-and-how-can-i-fo) – CCXXXI Aug 19 '22 at 05:08
  • 1
    Here is a good explanation https://stackoverflow.com/questions/2688466/why-mallocmemset-is-slower-than-calloc Basically, all memory allocated from the OS initially points to a singular zero-page. Only on first write access to a memory page does the OS allocate a physical page for a virtual page. Arithmetic computation counts as write access, even if it only writes zero values – Homer512 Aug 19 '22 at 06:56
  • I figured once you had a name, Google would tell you all you needed to know. You can do arithmetic operations just fine, but it's going to be slow. – Mark Ransom Aug 19 '22 at 12:33

0 Answers0