Python profiling methods

Question

I would like to profile python code from an object point of view. E.g.:

foo = Foo()
profiled_foo = add_profiling(foo)

# use profiled_foo like foo
...

# later
profiled_foo.print_profile()

And I would like to get calls per method and cumulative time spent per method. I didn't find anything similar, although I think it shouldn't be too hard to write.

Does any library like that exist? Or maybe not because profiling this way would be a bad idea?

Based on Paul McGuire's answer:

import inspect

from time import sleep
from profilehooks import profile

class Foo(object):
    def a(self):
        sleep(0.1)

    def b(self):
        sleep(0.3)

    def c(self):
        sleep(0.5)

def add_profiling(obj):
    for k in dir(obj):
        attr = getattr(obj, k)
        if inspect.ismethod(attr) and k != '__init__':
            setattr(obj, k, profile(attr))

if __name__ == '__main__':
    foo = Foo()
    add_profiling(foo)

    foo.a()
    foo.a()
    foo.b()
    foo.b()
    foo.a()
    foo.c()

.

*** PROFILER RESULTS ***
c (oprof.py:13)
function called 1 times

         3 function calls in 0.501 CPU seconds

   Ordered by: cumulative time, internal time, call count

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.501    0.501 oprof.py:13(c)
        1    0.501    0.501    0.501    0.501 {time.sleep}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        0    0.000             0.000          profile:0(profiler)

...

There's a similar package to `profilehooks`, however, it also supports flame graph generation and measuring arbitrary code regions, not only function calls: https://github.com/metopa/region_profiler — Viacheslav Kroilov, Feb 10 '19 at 20:19

score 6 · Accepted Answer · answered Oct 25 '11 at 16:29

6

I've had pretty good success with these decorators: http://mg.pov.lt/profilehooks/

answered Oct 25 '11 at 16:29

PaulMcG

62,419
16
94
130

score 1 · Answer 2 · answered Oct 25 '11 at 16:29

1

You can look at http://docs.python.org/library/profile.html

In my personal opinion cProfile is the more intuitive and quick profiling tools for python.

To profile single class you can write simple test script and run them with cProfile.

answered Oct 25 '11 at 16:29

Davide Aversa

5,628
6
28
40

score 1 · Answer 3 · edited May 23 '17 at 11:51

You may have heard of this before, but there's a distinction between profiling for its own sake versus profiling with the objective of making code run as fast as possible.

It's good that you want cumulative time per method summed over all calls to the method, because if you divide that by total execution time you get the percent of time the method is responsible for. If there's something one can fix to get better speed, it is among the high-percent methods in one's own code.

Even better than getting the methods of high cumulative percent is getting the code lines of high cumulative percent. That's because if you want to look inside a method for where the problem is, you are looking for specific lines of code that have high percent. So if your profiler tells you where those lines are, you already have (depending on method size) possibly an order of magnitude greater precision of locating the problem.

Even better than knowing the location of lines of your code that have high percent, is knowing the context in which they are likely to be executed.

One thing some think is necessary is to get accurate time measurements. However, once you find a line in your code that clearly represents an opportunity to get good speedup, because removing-replacing-doing it differently would save a substantial percentage of overall time, does it really matter if your estimate of that percentage is off a little bit? Suppose you find that line and you think it costs 50%. If the true actual savings from fixing it were as small as 30% or as large as 70%, would you feel bad that you spent the time to fix it?

That's why I encourage stack-sampling, for which this is a small example.

Stack samples automatically give you line-level percent, because the percentage cost of a line is just the percent of samples containing it. They automatically include blocked time such as I/O, which is something you need to know because it contributes to overall time. If 100s or 1000s of samples are taken, you can get high precision of measurement, but some very simple statistics proves that for all but the smallest problems, a small number of samples is adequate to locate the problem with precision. Finally, the context that tells whether a line is being well-used is in the samples. Many profilers mush the samples together to get numbers, but don't let the user see representative samples. As a result, they tend to assume the "hot" line of code is necessary when in fact the context could tell them it is not.

Here's a more extensive listing of the issues.

Python profiling methods

3 Answers3