0

Numpy is known for optimized arrays and various advantages over python-lists. But when I check for the memory usage python-lists have less space than the numpy arrays. The code I used is entered below. Can anyone explain me why?

import sys
Z = np.zeros((10,10),dtype = int)
A = [[0] * 10] * 10
print(A,'\n',f'{sys.getsizeof(A)} bytes')
print(Z,'\n',f'{Z.size * Z.itemsize} bytes')
Teja Kompella
  • 1
  • 2
  • 1
    Have a look here to compliment Thomas' answer https://stackoverflow.com/questions/5022725/how-do-i-measure-the-memory-usage-of-an-object-in-python. – Reti43 Feb 08 '21 at 15:25

2 Answers2

1

You're not measuring correctly; the native Python list only contains 10 references. You need to add in the collective size of the sub-lists as well:

>>> sys.getsizeof(A) + sum(map(sys.getsizeof, A))
1496

And it might get worse: each element inside the sub-lists could also be a reference (to an int). It's difficult to check whether the Python implementation is optimizing this away and storing the actual numbers inside the list.

You're also under-representing the size of the numpy array, because it includes a header:

>>> Z.size * Z.itemsize
800
>>> sys.getsizeof(Z)
912

In either case it's not an exact science and will depend on your platform and Python implementation.

Thomas
  • 174,939
  • 50
  • 355
  • 478
  • With `[[0] * 10] * 10` he creates 10 references to the same sublist, compared to [[0] * 10 for _ in range(10)]`. which would create 10 different lists, so technically the total size would be different. – Reti43 Feb 08 '21 at 15:25
  • Thanks for your info – Teja Kompella Feb 08 '21 at 15:40
0

According to the spec (https://docs.python.org/3/library/sys.html#sys.getsizeof), "only the memory consumption directly attributed to the object is accounted for, not the memory consumption of objects it refers to." Also "getsizeof calls object’s " sizeof method

So you are given the size of just the container (a list object).

Please check https://code.activestate.com/recipes/577504/ for a complete size computation, which returns 296bytes for your example since only two unique objects are used. A list [0 0 0 0 0 0 0 0 0 0] and int 0.

If you initialize the list with different values, overall size will increase and will become bigger than np.array, which reserves 4bytes for numpy.int32 type elements, plus the size of its own internal data structure.

Find detailed info with examples here: https://jakevdp.github.io/PythonDataScienceHandbook/02.01-understanding-data-types.html

Marcos Jiménez
  • 1