Return value while using cProfile

Question

I'm trying to profile an instance method, so I've done something like:

import cProfile

class Test():

    def __init__(self):
        pass

    def method(self):
        cProfile.runctx("self.method_actual()", globals(), locals())

    def method_actual(self):
        print "Run"

if __name__ == "__main__":
    Test().method()

But now problems arise when I want "method" to return a value that is computed by "method_actual". I don't really want to call "method_actual" twice.

Is there another way, something that can be thread safe? (In my application, the cProfile data are saved to datafiles named by one of the args, so they don't get clobbered and I can combine them later.)

score 37 · Accepted Answer · answered Oct 18 '09 at 09:39

37

I discovered that you can do this:

prof = cProfile.Profile()
retval = prof.runcall(self.method_actual, *args, **kwargs)
prof.dump_stats(datafn)

The downside is that it's undocumented.

answered Oct 18 '09 at 09:39

detly

29,332
18
93
152

5

Brilliant! This looks perfect - but what is 'datafn'? – Jonathan Hartley Sep 22 '15 at 10:55
@JonathanHartley - The filename for the data file IIRC. – detly Sep 22 '15 at 11:24
Ah, thanks. I thought 'fn' meant function, not filename. – Jonathan Hartley Sep 22 '15 at 14:12
1

@Ngoral it's just a file name. You can put whatever name you like there. – detly Apr 08 '18 at 21:21
@Ngoral no worries, I should clarify in the answer – detly Apr 08 '18 at 21:26

score 30 · Answer 2 · edited Jan 05 '17 at 21:01

30

An option for any arbitrary code:

import cProfile, pstats, sys
pr = cProfile.Profile()
pr.enable()

my_return_val = my_func(my_arg)

pr.disable()
ps = pstats.Stats(pr, stream=sys.stdout)
ps.print_stats()

Taken from https://docs.python.org/2/library/profile.html#profile.Profile

edited Jan 05 '17 at 21:01

wordsforthewise

13,746
5
87
117

answered Jun 23 '13 at 09:35

Alex Constantin

519
4
8

You could even make a little context manager for that using `contextlib`s `contextmanager` decorator. – detly Jun 23 '13 at 09:46
I get `Random listing order was used` - how can I specify listing order ? – Mr_and_Mrs_D Feb 27 '16 at 20:49
3

ps.sort_stats('cumulative') – marsh Mar 29 '16 at 15:11
how do I generate a valid profile file with this method so that I can use it with SnakeViz – Ishan Srivastava Mar 19 '18 at 19:45
I would only add something like: ps.dump_stats('my_dumpfile.dmp') so one can use snakeviz or similar to view the results, otherwise this worked great for me! I was having a heckuva time getting my return values before this. – Kevin Newman Aug 19 '21 at 14:23

score 8 · Answer 3 · edited Feb 19 '12 at 04:03

I was struggling with the same problem and used a wrapper function to get over direct return values. Instead of

cP.runctx("a=foo()", globals(), locales())

I create a wrapper function

def wrapper(b):
  b.append(foo())

and profile the call to the wrapper function

b = []
cP.runctx("wrapper(b)", globals(), locals())
a = b[0]

extracting the result of foo's computation from the out param (b) afterwards.

Jonathan Hartley · Answer 4 · 2015-09-22T14:15:00.833

I created a decorator:

import cProfile
import functools
import pstats

def profile(func):

    @functools.wraps(func)
    def inner(*args, **kwargs):
        profiler = cProfile.Profile()
        profiler.enable()
        try:
            retval = func(*args, **kwargs)
        finally:
            profiler.disable()
            with open('profile.out', 'w') as profile_file:
                stats = pstats.Stats(profiler, stream=profile_file)
                stats.print_stats()
        return retval

    return inner

Decorate your function or method with it:

@profile
def somefunc(...):
   ...

Now that function will be profiled.

Alternatively, if you'd like the raw, unprocessed profile data (e.g. because you want to run the excellent graphical viewer RunSnakeRun on it), then:

import cProfile
import functools
import pstats

def profile(func):

    @functools.wraps(func)
    def inner(*args, **kwargs):
        profiler = cProfile.Profile()
        profiler.enable()
        try:
            retval = func(*args, **kwargs)
        finally:
            profiler.disable()
            profiler.dump_stats('profile.out')
        return retval

    return inner

This is a minor improvement on several of the other answers on this page.

score 2 · Answer 5 · answered Feb 12 '14 at 17:16

I think @detly the .runcall() is basically the best answer, but for completeness, I just wanted to take @ThomasH 's answer to be function independent:

def wrapper(b, f, *myargs, **mykwargs):
    try:
        b.append(f(*myargs, **mykwargs))
    except TypeError:
        print 'bad args passed to func.'

# Example run
def func(a, n):
    return n*a + 1

b = []
cProfile.runctx("wrapper(b, func, 3, n=1)", globals(), locals())
a = b[0]
print 'a, ', a

Return value while using cProfile

5 Answers5

Linked