71

I have a few (almost ten) Gb of memory taken by the ipython kernel. I think this is coming from large objects (matrices, lists, numpy arrays, ...) that I might have produced during some operation and now I do not need anymore.

I would like to list all of the objects I have defined and sort them by their memory footprint. Is there a simple way to do that? For certain types there is nbytes method, but not for all ... so I am looking for a general way to list all objects I have made and their memory occupation.

Rho Phi
  • 1,182
  • 1
  • 12
  • 21
  • 1
    `import sys; sorted([(x, sys.getsizeof(x)) for x in dir()], key=lambda x: x[1], reverse=True)`? – Abdou Dec 06 '16 at 11:01
  • Possible duplicate of [How can I check the memory usage of objects in iPython?](http://stackoverflow.com/questions/563840/how-can-i-check-the-memory-usage-of-objects-in-ipython) – amin Dec 06 '16 at 11:10
  • 3
    Not a duplicate, he's asking how to find objects he doesn't know the name of. @Abdou you should make that an answer I think. – Daniel F Dec 06 '16 at 11:24
  • Thanks! the issue with `getsizeof` is what units does it use? the documentation says bytes, but it cannot possibly be right as for me it shows `[('T2T_cij_lsi200_600histo', 72), ('T2T_cij_lsi20_600histo', 71), ('T2T_cij_lsi50_600histo', 71), ('cluster_single_author', 70), ('SpectralBiclustering', 69), ('SpectralCoclustering', 69), ('begin_of_allocation', 68), ('feature2feature_log', 68,....` and the frist objects are about 1 Gb (I know from numpy .nbytes and from pickling them). Furthermore doing `%whos` these symbols are listed with the correct size around 1 Gb – Rho Phi Dec 06 '16 at 12:47
  • 1
    Actually should be: `import sys; sorted([(x, sys.getsizeof(globals().get(x))) for x in dir()], key=lambda x: x[1], reverse=True)`. The previous one was getting the sizes of the strings. This one actually gets the objects. – Abdou Dec 06 '16 at 13:52
  • 2
    `getsizeof` for a list is just the size of the pointers in the list, not the size of the objects pointed to. http://stackoverflow.com/questions/33052836/getsizeof-returns-the-same-value-for-seemingly-different-lists – hpaulj Dec 06 '16 at 17:12

2 Answers2

110

Assuming that you are using ipython or jupyter, you will need to do a little bit of work to get a list all of the objects you have defined. That means taking everything available in globals() and filtering out objects that are modules, builtins, ipython objects, etc. Once you are sure you have those objects, then you can proceed to grabbing their sizes with sys.getsizeof. This can be summed up as follows:

import sys

# These are the usual ipython objects, including this one you are creating
ipython_vars = ['In', 'Out', 'exit', 'quit', 'get_ipython', 'ipython_vars']

# Get a sorted list of the objects and their sizes
sorted([(x, sys.getsizeof(globals().get(x))) for x in dir() if not x.startswith('_') and x not in sys.modules and x not in ipython_vars], key=lambda x: x[1], reverse=True)

Please keep in mind that for python objects (those created with python's builtin functions), sys.getsizeof will be very accurate. But it can be a bit inaccurate on objects created using third-party libraries. Furthermore, please be mindful that sys.getsizeof adds an additional garbage collector overhead if the object is managed by the garbage collector. So, some things may look a bit heavier than they actually are.

As a side note, numpy's .nbytes method can be somewhat misleading in that it does not include memory consumed by non-element attributes of the array object.

starball
  • 20,030
  • 7
  • 43
  • 238
Abdou
  • 12,931
  • 4
  • 39
  • 42
  • 2
    super working code.. tried lot of other places but the size values were incorrect. thanks a ton!! – rishi jain Jun 11 '20 at 12:19
  • 1
    Didn't seem to work for me — I dont use ipython so modified like so `sorted([(x, sys.getsizeof(globals().get(x))) for x in dir() if not x.startswith('_') and x not in sys.modules], key=lambda x: x[1], reverse=True)` (no `ipython_vars`) but the value of `hi` is `1` and it registered as `28`...? Is this the garbage collector you were talking about, because `i` has a value of `43892978` and it registers as 28 too. – TheTechRobo the Nerd Jun 27 '20 at 14:28
  • fantastic, I had no idea how to list these or what the default objects were, thank you. – jimh Dec 06 '22 at 20:40
2

I like the answer @Abdou provided! I would only add the following suggestion. Instead of a list of tuples, I would convert it to a dictionary.

import sys

# These are the usual ipython objects, including this one you are creating
ipython_vars = ["In", "Out", "exit", "quit", "get_ipython", "ipython_vars"]

# Get a sorted list of the objects and their sizes
mem = {
    key: value
    for key, value in sorted(
        [
            (x, sys.getsizeof(globals().get(x)))
            for x in dir()
            if not x.startswith("_") and x not in sys.modules and x not in ipython_vars
        ],
        key=lambda x: x[1],
        reverse=True,
    )
}

Then if I wanted to get the total amount in MBs, all I'd have to do is:

sum(mem.values()) / 1e6
Daniel Cárdenas
  • 68
  • 1
  • 5