1

I always thought numpy array is more compact and takes less memory size compare to list, however, for a 3-D float64 np array,

print (sys.getsizeof(result2)/1024./1024./1024.)
print (sys.getsizeof(result2.astype('float16'))/1024./1024./1024.)
print (sys.getsizeof(list(result2))/1024./1024./1024.)
print (sys.getsizeof(result2.tolist())/1024./1024./1024.)

The output is,

0.6521792411804199
0.16304489970207214
0.00033977627754211426
0.0003019943833351135

The list took much smaller memory. Is it validate to use sys.getsizeof for list? If it is, can I do anything to improve np array memory use?

######################

Using pympler @J_H, (it seems pympler can't deal with arraies in a list, like list(a 3-D array ). )

print (result2.nbytes/1024./1024./1024.)
print (asizeof.asizeof(result2)/1024./1024./1024.)
print (result2.astype('float16').nbytes/1024./1024./1024.)
print (asizeof.asizeof(list(result2))/1024./1024./1024.)
print (asizeof.asizeof(result2.tolist())/1024./1024./1024.)

0.6521791219711304
0.6521792411804199
0.1630447804927826
0.004566863179206848
4.078836984932423

Thank you all!

user9660558
  • 31
  • 3
  • 2
    `getsizeof` is non-recursive, i.e. it will not account for the conent of a container. – Paul Panzer Jul 26 '20 at 01:16
  • See https://stackoverflow.com/questions/5022725/how-do-i-measure-the-memory-usage-of-an-object-in-python – Tarik Jul 26 '20 at 01:21
  • 1
    The memory use of a ndarray` is easy - `arr.nbytes`, or product of shape and bytes per element. Memory use of a list is hard to estimate. – hpaulj Jul 26 '20 at 01:21

1 Answers1

1

You show that each of your list elements consumes 8 bytes.

But each element is just a pointer to a 24-byte float object.

Additionally, when you start with a 3-D array, you'll be looking at lists within lists. You could recurse through the data structures yourself to accurately add up the allocated bytes. Or you could use pympler.

J_H
  • 17,926
  • 4
  • 24
  • 44