4

I needed to check the memory stats of objects I use in python.
I came across guppy and pysizer, but they are not available for python2.7.
Is there a memory profiler available for python 2.7?
If not is there a way I can do it myself?

simha
  • 869
  • 1
  • 8
  • 12
  • Do you *really* need to profile your code? Just asking! – user225312 Dec 11 '10 at 12:41
  • "check the memory stats of objects" Why? What **specific** problem are you having? Out of memory? Profiling isn't usually necessary, it's usually *obvious* which object is too large. Please explain what problem you are actually observing. – S.Lott Dec 11 '10 at 14:19
  • 2
    possible duplicate of [Python memory profiler](http://stackoverflow.com/questions/110259/python-memory-profiler) – Lennart Regebro Dec 11 '10 at 15:25

4 Answers4

2

Here's one that works for Python 2.7: The Pympler package.

partofthething
  • 1,071
  • 1
  • 14
  • 19
2

You might want to try adapting the following code to your specific situation and support your data types:

import sys


def sizeof(variable):
    def _sizeof(obj, memo):
        address = id(obj)
        if address in memo:
            return 0
        memo.add(address)
        total = sys.getsizeof(obj)
        if obj is None:
            pass
        elif isinstance(obj, (int, float, complex)):
            pass
        elif isinstance(obj, (list, tuple, range)):
            if isinstance(obj, (list, tuple)):
                total += sum(_sizeof(item, memo) for item in obj)
        elif isinstance(obj, str):
            pass
        elif isinstance(obj, (bytes, bytearray, memoryview)):
            if isinstance(obj, memoryview):
                total += _sizeof(obj.obj, memo)
        elif isinstance(obj, (set, frozenset)):
            total += sum(_sizeof(item, memo) for item in obj)
        elif isinstance(obj, dict):
            total += sum(_sizeof(key, memo) + _sizeof(value, memo)
                         for key, value in obj.items())
        elif hasattr(obj, '__slots__'):
            for name in obj.__slots__:
                total += _sizeof(getattr(obj, name, obj), memo)
        elif hasattr(obj, '__dict__'):
            total += _sizeof(obj.__dict__, memo)
        else:
            raise TypeError('could not get size of {!r}'.format(obj))
        return total
    return _sizeof(variable, set())
Noctis Skytower
  • 21,433
  • 16
  • 79
  • 117
  • Looks promising, but I think you should change it to `def getsizeof(obj, memo=None):` and move the initialization of `memo` inside the function with as `if memo is None: memo = set()`. Of course all the recursive calls to `getsizeof()` would then need to have their arguments swapped around, too. – martineau Dec 12 '10 at 14:12
  • 1
    @martineau: A promise which should not be casually tossed in the path of a newbie. Handling instances of classes through all the complexities of old/new, with/without `__slots__`, no/single/multiple inheritance, etc is a nightmare. See for example http://code.activestate.com/recipes/546530-size-of-python-objects-revised/ which took well over 2000 SLOC and multiple versions to get it to a not-known-to-be-wrong state. I do agree with your comment on the `memo` arg. – John Machin Dec 12 '10 at 20:34
  • @John Machin: The activestate recipe is now closer to 3000 lines of source code and is version 5.12, so the general case is apparently a very hard egg to crack -- which is one reason I found @Noctis Skytower's answer interesting because frequently one doesn't need a tool that handles every conceivable case, just a few of their own. Hopefully `sys.getsizeof()` will improve in the future -- I've never actually needed it for anything, so was only aware of it haven been added, and not of all its deficiencies. – martineau Dec 13 '10 at 00:01
  • The code up above was used to find the size of the data structures created by this searching algorithm: http://stackoverflow.com/questions/3242597/what-is-memoization-good-for-and-is-it-really-all-that-helpful/3276775#3276775 – Noctis Skytower Dec 13 '10 at 03:01
1

I'm not aware of any profilers for Python 2.7 -- but check out the following function which has been added to the sys module, it could help you do it yourself.

"A new function, getsizeof(), takes a Python object and returns the amount of memory used by the object, measured in bytes. Built-in objects return correct results; third-party extensions may not, but can define a __sizeof__() method to return the object’s size."

Here's links to places in the online docs with information about it:

    What’s New in Python 2.6
    27.1. sys module — System-specific parameters and functions

martineau
  • 119,623
  • 25
  • 170
  • 301
  • It could but it doesn't. `sys.getsizeof` is "a snare and a delusion". Read this SO question, in particular the answer by Thomas Wouters: http://stackoverflow.com/questions/2117255 – John Machin Dec 11 '10 at 20:44
  • @John Machin: In that case what do you think of @Noctis Skytower's answer? – martineau Dec 12 '10 at 14:14
0

pympler 0.9 is the latest version that supports Python 2.7, see https://github.com/pympler/pympler/tags or just use pip

MonkeyProof Solutions
  • 111
  • 4