29

Brendan Gregg's CPU Flame Graphs are a way of visualising CPU usage over a period of time based on call stacks.

His FlameGraph github project provides a language-independent way to plot these graphs:

For each language, FlameGraph requires a way of providing stack input in the form of lines like this:

grandparent_func;parent_func;func 42

This means that the instrumented program was observed running function func, where that was called from parent_func, in turn called from top-level function grandparent_func. It says that call stack was observed 42 times.

How can I gather stack information from Python programs and provide it to FlameGraph?

For bonus points: How can that be extended so that both the C and Python stack is shown, or even down to the kernel on Linux (in a similar way to some of the Java and node.js flame graphs on Brendan's website)?

enter image description here

Reblochon Masque
  • 35,405
  • 10
  • 55
  • 80
Croad Langshan
  • 2,646
  • 3
  • 24
  • 37
  • 1
    You should be able to do something with dtrace - requires an OS that supports dtrace and a patched version of python. – ErikR Jan 17 '15 at 15:36
  • 4
    For what it's worth, in my opinion, the prettiness of flame-graphs exceeds their usefulness for finding speedups, and there's a way that's less pretty but more effective. [*Explained here.*](http://stackoverflow.com/a/27867426/23771) – Mike Dunlavey Jan 18 '15 at 01:29
  • user5402: last I heard dtrace was not suitable for production use on Linux – Croad Langshan Jan 18 '15 at 17:15
  • 2
    Mike Dunlavey: I don't disagree but I think all's fair with finding performance problems, so I'd like to try flame graphs too – Croad Langshan Jan 18 '15 at 17:17
  • perf script [-s [Python]:script[.py] ], but linux kernel should build with special flag (NO_LIBPYTHON=0) like describe here http://askubuntu.com/questions/577768/how-can-i-make-perf-script-g-python-work – VelikiiNehochuha May 02 '16 at 22:12

2 Answers2

9

Maybe you can try sys.setprofile, which is the core for the standard python profiler profile and cProfile. This method sets a hook to the "call" and "return" events of every function, including those functions of C-API.

The system’s profile function is called similarly to the system’s trace function (see settrace()), but it isn’t called for each executed line of code (only on call and return, but the return event is reported even when an exception has been set).

Below is a working example:

from time import clock 
t0 = clock()

def getFun(frame):
    code = frame.f_code 
    return  code.co_name+' in '+code.co_filename+':'+str(code.co_firstlineno)

def trace_dispatch(frame, event, arg):
    if event in [ "c_call" , 'call', 'return', 'c_return']:
        t = int((clock()-t0)*1000)
        f = frame
        stack=[]
        while(f):
          stack.insert( 0,getFun(f) )
          f = f.f_back
        print event, '\t', '; '.join(stack), '; ', t

import sys
sys.setprofile(trace_dispatch)
try:
    execfile('test.py')
finally:
    sys.setprofile(None)

Test.py

def f(x):
    return x+1
def main(x):
    return f(x)
main(10)

This will print out

c_call    0
call      <module> in test.py:2 ;  1
call      <module> in test.py:2; main in test.py:5 ;  1
call      <module> in test.py:2; main in test.py:5; f in test.py:2 ;  5
return    <module> in test.py:2; main in test.py:5; f in test.py:2 ;  8
return    <module> in test.py:2; main in test.py:5 ;  11
return    <module> in test.py:2 ;  14
c_return  18
c_call    21

See a more comprehensive profiling function here.

C stack in python

You cannot access the C stack within the python interpreter. It is necessary to use a debugger or profiler that supports C/C++. I would recommand gdb python.

gdlmx
  • 6,479
  • 1
  • 21
  • 39
3

Pyflame supports plotting flame graphs in two formats (either the 'traditional' form as in the question, or chrome's 'sideways' flame graphs, using chrome itself).

From https://github.com/uber/pyflame:

# Attach to PID 12345 and profile it for 1 second
pyflame -p 12345

# Attach to PID 768 and profile it for 5 seconds, sampling every 0.01 seconds
pyflame -s 5 -r 0.01 -p 768

# Run py.test against tests/, emitting sample data to prof.txt
pyflame -o prof.txt -t py.test tests/
Croad Langshan
  • 2,646
  • 3
  • 24
  • 37