Using cProfile results with KCacheGrind

Question

I'm using cProfile to profile my Python program. Based upon this talk I was under the impression that KCacheGrind could parse and display the output from cProfile.

However, when I go to import the file, KCacheGrind just displays an 'Unknown File Format' error in the status bar and sits there displaying nothing.

Is there something special I need to do before my profiling stats are compatible with KCacheGrind?

...
if profile:
    import cProfile

    profileFileName = 'Profiles/pythonray_' + time.strftime('%Y%m%d_%H%M%S') + '.profile'

    profile = cProfile.Profile()
    profile.run('pilImage = camera.render(scene, samplePattern)')

    profile.dump_stats(profileFileName)
    profile.print_stats()
else:            
    pilImage = camera.render(scene, samplePattern)
...

Package Versions

• KCacheGrind 4.3.1
• Python 2.6.2

Answer 1

With cProfile you can also profile existing programs, without making any separate profiling script. Just run program with profiler

python -m cProfile -o profile_data.pyprof script_to_profile.py

and open profile data in kcachegrind with pyprof2calltree, whose -k switch automatically opens data in kcachegrind

pyprof2calltree -i profile_data.pyprof -k

For example profiling whole paster server and webapp would be done like this

python -m cProfile -o pyprof.out `which paster` serve development.ini

pyprof2calltree can be installed with easy_install.

Answer 2

You could use profilestats.profile decorator ($ pip install profilestats) -- a simple wrapper for pyprof2calltree module (rebranding of lsprofcalltree.py):

from profilestats import profile

@profile
def func():
    # do something here

Script can be run as usual. profilestats creates two files: cachegrind.out.profilestats and profilestats.prof in KCachegrind-compatible and cProfile formats correspondingly.

Answer 3

It can be done using an external module called lscallproftree

This article explains how: CherryPy - CacheGrind

With my resulting code looking like so:

...
if profile:
    import cProfile
    import lsprofcalltree

    profileFileName = 'Profiles/pythonray_' + time.strftime('%Y%m%d_%H%M%S') + '.profile'

    profile = cProfile.Profile()
    profile.run('pilImage = camera.render(scene, samplePattern)')

    kProfile = lsprofcalltree.KCacheGrind(profile)

    kFile = open (profileFileName, 'w+')
    kProfile.output(kFile)
    kFile.close()

    profile.print_stats()    
else:            
    pilImage = camera.render(scene, samplePattern)
...

If anyone knows a way to do this that doesn't require an external (ie. not shipped with Python) module, I'd still be very interested to hear about it.

Answer 4

3 differents ways to profile your code and visualizing results in KCachegrind/Qcachegrind:

I - CPROFILE

1 - Profile myfunc() from ipython

import cProfile
filename = 'filename.prof'
cProfile.run('myfunc()', filename)

2 - Convert your file to a usable kcachegrind file in your shell

sudo pip install pyprof2calltree
pyprof2calltree -i filename.prof -o callgrind.filename.prof

3 - Open callgrind.filename.prof in kcachegrind

II - EMBEDDED CPROFILE

1 - Profile few lines in your code.

import cProfile
filename = 'filename.prof'
pr = cProfile.Profile()
pr.enable()
# ... lines to profile ...
pr.disable()
pr.dump_stats(filename)

2 - Convert your file to a usable kcachegrind file in your shell

sudo pip install pyprof2calltree
pyprof2calltree -i filename.prof -o callgrind.filename.prof

3 - Open callgrind.filename.prof in kcachegrind

III - YAPPI

1 - Profile myfunc() from ipython or from your code

import yappi
filename = 'callgrind.filename.prof'
yappi.set_clock_type('cpu')
yappi.start(builtins=True)
myfunc()
stats = yappi.get_func_stats()
stats.save(filename, type='callgrind')

2 - Open callgrind.filename.prof in kcachegrind

Answer 5

If what you're actually trying to do is see what parts of your code could be optimized for speed, and you can randomly pause it in the debugger, this method works. It may be surprising, but you don't need very many stackshots.