15

What's a good way to estimate the memory footprint of an object?

Conversely, what's a good way to measure the footprint?

For example, say I have a dictionary whose values are lists of integer,float tuples:

d['key'] = [ (1131, 3.11e18), (9813, 2.48e19), (4991, 9.11e18) ]

I have 4G of physical memory and would like to figure out approximately how many rows (key:values) I can store in memory before I spill into swap. This is on linux/ubuntu 8.04 and OS X 10.5.6 .

Also, what's the best way to figure out the actual in-memory footprint of my program? How do I best figure out when it's exhausting physical memory and spilling?

smci
  • 32,567
  • 20
  • 113
  • 146
Parand
  • 102,950
  • 48
  • 151
  • 186
  • When you say 'an object', presumably you want to handle the case where it can be an arbitrarily deeply nested object, with (multiple, possibly aliased) references to other objects/strings/arrays etc. – smci Jun 30 '20 at 01:04

2 Answers2

10

Guppy has a nice memory profiler (Heapy):

>>> from guppy import hpy
>>> hp = hpy()
>>> hp.setrelheap() # ignore all existing objects
>>> d = {}
>>> d['key'] = [ (1131, 3.11e18), (9813, 2.48e19), (4991, 9.11e18) ]
>>> hp.heap()
 Partition of a set of 24 objects. Total size = 1464 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0      2   8      676  46       676  46 types.FrameType
     1      6  25      220  15       896  61 str
     2      6  25      184  13      1080  74 tuple
 ...

Heapy is a little underdocumented, so you might have to dig through the web page or source code a little, but it's very powerful. There are also some articles which might be relevant.

Torsten Marek
  • 83,780
  • 21
  • 91
  • 98
  • 1
    Why are there 6 'str' objects in 'd'? – Dustin Boswell Feb 25 '11 at 06:45
  • When binding a new variable, a new string is created ('d'). Attribute lookup (heap) accounts for another one. I don't know where the rest is coming from. – Torsten Marek Mar 14 '11 at 15:47
  • The actual results vary and you must either check them against the particular interpreter implementation you are using or do some trial-and-error to find the variance. For instance, I just tried the above example and it reported three strings totaling 96 bytes. – stw_dev Mar 20 '12 at 10:56
5

You can do this with a memory profiler, of which there are a couple I'm aware of:

  1. PySizer - poissibly obsolete, as the homepage now recommends:

  2. Heapy.

This is possibly a duplicate of this question.

Community
  • 1
  • 1
Brian
  • 116,865
  • 28
  • 107
  • 112