Python MemoryError on large array

Question

This is the python script that I'm trying to run:

n = 50000000000 ##50 billion 
b = [0]*n
for x in range(0,n):
    b[x] = random.randint(1,899999)

... But the output I'm getting is:

E:\python\> python sort.py
Traceback (most recent call last):
  File "E:\python\sort.py", line 8, in <module>
    b = [0]*n
MemoryError

So, what do I do now?

An `int` takes at least 24 bytes; you have 5,000,000,000 `int`s. That's 111GB right there. — chepner, May 18 '17 at 19:51
Maybe it would be more useful if you explained what you were planning to do with so many random numbers. Almost certainly there's a way to do it without blowing out your memory. — alexis, May 18 '17 at 19:53
@chepner ints in python seems to be 64 bits, so 8 bytes. 24 bytes would be one heck of a number — litelite, May 18 '17 at 19:53
@litelite No, an `int` is a Python object, representing an integer with arbitrary precision, not a machine word. — chepner, May 18 '17 at 19:53
That's right, Python has overhead in order to keep track of types etc. You want an 8-byte int, use C. — alexis, May 18 '17 at 19:54
@chepner Right, not a pro in python I keep forgeting about that — litelite, May 18 '17 at 19:54
Also you counted your zeros wrong, so n is 50 billion, not 5 billion. — user2357112, May 18 '17 at 19:55
Also, ints in Python are not even 64 bits; Python has unlimited-length integers, welcome to the modern world. — alexis, May 18 '17 at 19:55
You mean to limit the memory footprint? Good point. Though the `struct` option isn't actually an `int`, it's just cargo... — alexis, May 18 '17 at 19:56
Even if you *have* a terabyte of free memory, there's no need to initialize the list with zeros first; just use `b = [random.randint(1,899999) for _ in range(n)]`. — chepner, May 18 '17 at 20:04
@chepner well, in CPython at least, initializing with all zeros will be using the cached `0` object. — juanpa.arrivillaga, May 18 '17 at 20:06
if not python3 use `xrange` and listen to the comments above, what is the purpose of initalizing a list with zero? — , May 19 '17 at 17:02

Tenchi2xh · Accepted Answer · 2017-05-18T20:05:36.820

10

The size of the list you are generating (which is 50 billion not 5).

An int object instance takes 24 bytes (sys.getsizeof(int(899999)), the upper limit of your random numbers), so that list would take 50,000,000,000 * 24 bytes, which is about 1.09 TB.

In other words to create such a list you would need at least 1118 GB of RAM in your computer.

I don't know what your use case is, but you should consider a different approach to what you are trying to solve (maybe define a generator, or just don't store your numbers in memory and instead directly use the numbers in the for loop).

edited May 18 '17 at 20:05

answered May 18 '17 at 19:59

Tenchi2xh

190
1
7

That's actually a *lower limit* generally. The smallest `int` takes that much, but `int` objects take variable amounts of memory, e.g. `sys.getsizeof(1000000000000000000) == 32`, whereas `sys.getsizeof(1000) == 28`, – juanpa.arrivillaga May 18 '17 at 20:05
@juanpa.arrivillaga In the author's question, the maximum attainable number is 899999, and int objects from 0 up to that number are all 24 bytes in size. (I updated the answer to show how this upper limit is calculated) – Tenchi2xh May 18 '17 at 20:06
I'm getting 28 on my system... 64bit python 3.5, Mac OSX – juanpa.arrivillaga May 18 '17 at 20:07
Python 3 apparently has 4 more bytes of overhead compared to Python 2 – Tenchi2xh May 18 '17 at 20:09
@juanpa.arrivillaga For zero, you don't need to store any data. For anything with a larger magnitude, you need to store something. – chepner May 18 '17 at 20:40
1

Also, just to pile on, we've only been talking about the memory needed for the `int` objects themselves. The *list* is another 64 bytes plus 50,000,000,000 pointers (4-8 bytes, depending on your architecture) to reference each `int` object. – chepner May 18 '17 at 20:42

score 2 · Answer 2 · answered Aug 26 '21 at 22:33

Since other people already answered your question here's a quick tip when dealing with big numbers: you can use "_" to separate the digits of your numbers as you wish:

n = 50_000_000_000

is the same as

n = 50000000000

but the former is much easier on the eyes

score 0 · Answer 3 · answered Dec 11 '18 at 17:09

0

One other possibility is to increase you computers vitual memory. It helped me in my code. I had a max 3000MB virtual memory, when I increased it to 5000MB the memory error was gone.

answered Dec 11 '18 at 17:09

Sanjay

169
2
9

Python MemoryError on large array

3 Answers3

Linked