1

I have a script which sometimes runs successfully, providing the desired output, but when rerun moments later it provides the following error:

numpy.core._exceptions.MemoryError: Unable to allocate 70.8 MiB for an array with shape (4643100, 2) and data type float64

I realise this question has been answered several times (like here), but so far none of the solutions have worked for me. I was wondering if anyone has any idea how it's possible that sometimes the script runs fine and then moments later it provides an error?

I have lowered my computer's RAM usage, have increased the virtual memory, rebooted my laptop, none of which seemed to help (Windows 10, RAM 8.0GB, python 3.9.2 32 bit).

PS: Unfortunately not possible to share the script/create dummy.

neòinean
  • 39
  • 7

1 Answers1

1

Python is a garbage collected language. Garbage collection is non-deterministic. This means that peak memory usage may be different each time a program is run. So the first time you run the program, its peak memory usage is less than the available memory. But the next time you run the program, its peak memory usage is sufficient to consume all available memory. This assumes that the available memory on the host system is constant, which is an incorrect assumption. So the fluctuation in available memory, i.e. the memory not in use by the other running processes, is another reason that the program may raise a MemoryError one time, but terminate without error another time.

Sidenote: Increase virtual memory as a last resort. It isn't memory, it's disk that is used like memory, and it is much slower than memory.

Michael Ruth
  • 2,938
  • 1
  • 20
  • 27
  • Thanks for the reply, this explains a lot. Any tips on bypassing this issue? – neòinean May 18 '21 at 16:38
  • 1
    There are several ways to avoid `MemoryError`S: add more memory to the host system, reduce the memory consumption of your program, reduce the amount of memory used by the rest of the system, or rewrite the program in a language that allows memory management like C or C++. – Michael Ruth May 18 '21 at 16:58