2
<ipython-input-52-62a6baf77ac9> in <module>
      1 import numpy as np
----> 2 a = np.zeros((387256, 31894))

MemoryError: Unable to allocate array with shape (387256, 31894) and data type float64

This tiny code snippet works perfectly fine on my laptop with 16GB memory, but when I run this exact code on another machine that has 64GB memory, it gives this memory error. I checked the memory usage of the 64GB memory machine and it says it still has 62GB free memory since I'm not really doing anything else on it. Could this memory error be due to some other reason?

Edit: Numpy version for both machines: numpy.version.version is '1.17.2'

JYCH
  • 61
  • 7
  • `MemoryError: Unable to allocate 92.0 GiB for an array with shape (387256, 31894) and data type float64`. Even with `dtype=int32`, it's showing `MemoryError: Unable to allocate 46.0 GiB for an array with shape (387256, 31894) and data type int32`. I have a hard time believing that this works on an older laptop (and I'm on 16GB, which isn't that old) even if the dtype is left to the OS – roganjosh Feb 19 '20 at 21:53
  • I am using MacBook Pro 2015, which has '16 GB 2133 MHz LPDDR3' of memory. And I just now confirmed that this code snippet works in Jupyter Notebook on this MacBook – JYCH Feb 19 '20 at 21:55
  • This... does not make sense. I wonder what's going on – roganjosh Feb 19 '20 at 21:56
  • The context for running this test line is that in my code one of the lines errors out because of memory issue on my 16GB laptop, so I decided to use a 64GB machine to see if it will work, and that's when I faced this issue. – JYCH Feb 19 '20 at 21:58
  • Please include the numpy version in your question as an [edit]. Something doesn't seem right here – roganjosh Feb 19 '20 at 22:02
  • Added - both are 1.17.2 – JYCH Feb 19 '20 at 22:08
  • 2
    Are both machines running the same OS? – roganjosh Feb 19 '20 at 22:09
  • Do you have different architecture versions of Python between the two PCs (i.e. 32-bit vs 64-bit)? I have 32-bit installed and running your code gives me a `ValueError: array is too big; 'arr.size * arr.dtype.itemsize' is larger than the maximum possible size.` This is using 1.18.1 of `numpy`. – b_c Feb 19 '20 at 22:09
  • 1
    So, depending on a lot of things, `np.zeros` may not be actually initializing the large array, not until you actually try to do something with it. Conisder: https://stackoverflow.com/questions/44487786/performance-of-zeros-function-in-numpy?noredirect=1&lq=1 – juanpa.arrivillaga Feb 19 '20 at 22:10
  • Do either of the machines have a swap partition? – ApproachingDarknessFish Feb 19 '20 at 22:10
  • Do you have access to the full memory? E.g. are you running inside a container or are ulimits in effect? – MisterMiyagi Feb 19 '20 at 22:13
  • From these questions it seems that the issue is MacBook has swap partition but the 64GB machine I have does not, that is probably the reason. – JYCH Feb 19 '20 at 22:16

0 Answers0