Show lines where a particular method is called

Question

Let's say you have a particular method (function) from a particular module (of a particular class, optional). Is it possible via introspection of the library source code print all the lines where that method is called (used)? It can be called internally (with self.method_name()) or externally with (object1.method_name() in source file 1, object2.method_name() in source file 2, ... and objectN.method_name() in source file N).

An example could be shown on re module and it's method re.findall.

I tried to print the lines with grep, but this is a problem for methods having the same name (e.g. I tried it with method named connect(), but 24 classes have a method named connect... I'd like to filter this for particular class (and/or module).

Answer 1

I grep for function usage fairly regularly. Fortunately, I have never been interested in something that would be duplicated so heavily.

Here is what I might do, rather than write one-time code, if the false hits for Class.method were too common to filter manually. First grep for class Class to find the module with the class definition and note the range of lines. Then grep that module for self.method and delete or ignore hits outside that range. Then grep all modules of interest for import module and from module to find modules that might use the class and method. Then grep groups of modules depending on the specific form of import.

As others have pointed out, even this will miss calls that use aliases for the method name. But only you can know if this is an issue for your scenario. It is not, that I know of, for what I have done.

An entirely different approach, not depending on names, is to instrument the function with code that logs its calls, after using dynamic introspection to determine the caller. (I believe there are SO Q&As about this.)

Answer 2

I am adding this as another answer because the code is too big to cramp all together in the first one.

This is very simplistic example for finding out which function called which using abstract syntax tree.

To apply this on objects you have to stack when you enter them, then jump to their class and upon encountering a call to a function say that it is called from that particular object.

You can see how complicated this becomes when modules get involved. Each module should be entered and its submodules and all functions mapped so that you can track calls to them and so on.

import ast

def walk (node):
    """ast.walk() skips the order, just walks, so tracing is not possible with it."""
    end = []
    end.append(node)
    for n in ast.iter_child_nodes(node):
        # Consider it a leaf:
        if isinstance(n, ast.Call):
            end.append(n)
            continue
        end += walk(n)
    return end

def calls (tree):
    """Prints out exactly where are the calls and what functions are called."""
    tree = walk(tree) # Arrange it into our list
    # First get all functions in our code:
    functions = {}
    for node in tree:
        if isinstance(node, (ast.FunctionDef, ast.Lambda)):
            functions[node.name] = node
    # Find where are all called functions:
    stack = []
    for node in tree:
        if isinstance(node, (ast.FunctionDef, ast.Lambda)):
            # Entering function
            stack.append(node)
        elif stack and hasattr(node, "col_offset"):
            if node.col_offset<=stack[-1].col_offset:
                # Exit the function
                stack.pop()
        if isinstance(node, ast.Call):
            if isinstance(node.func, ast.Attribute):
                fname = node.func.value.id+"."+node.func.attr+"()"
            else: fname = node.func.id+"()"
            try:
                ln = functions[fname[:-2]].lineno
                ln = "at line %i" % ln
            except: ln = ""
            print "Line", node.lineno, "--> Call to", fname, ln
            if stack:
                print "from within", stack[-1].name+"()", "that starts on line", stack[-1].lineno
            else:
                print "directly from root"

code = """
import os

def f1 ():
    print "I am function 1"
    return "This is for function 2"

def f2 ():
    print f1()
    def f3 ():
        print "I am a function inside a function!"
    f3()
f2()
print "My PID:", os.getpid()
"""

tree = ast.parse(code)

calls(tree)

The output is:

Line 9 --> Call to f1() at line 4
from within f2() that starts on line 8
Line 12 --> Call to f3() at line 10
from within f2() that starts on line 8
Line 13 --> Call to f2() at line 8
directly from root
Line 14 --> Call to os.getpid()
directly from root

Answer 3

You probably know, but I just can't risk that you don't know: Python is not a strongly typed language.

As such something like objectn.connect() doesn't care what objectn is (it could be a module, a class, a function that aquired an attribute, ...). It also doesn't care if connect is a method or if it's a class that happens to be callable or a factory for functions. It would happily accept any objectn that somehow gives back a callable when you try to get the attribute connect.

Not only that, there are lots of ways to call methods, just assume something like:

class Fun(object):
    def connect(self):
        return 100

objectn = Fun()

(lambda x: x())(getattr(objectn, '{0}t'.format('co' + {0:'nnec'}[0])))

There's no way you could reliable search for objectn.connect() and match get a match for (lambda x: x())(getattr(objectn, '{0}t'.format('co' + {0:'nnec'}[0]))), but both do call the method connect of objectn.

So I'm very sorry to say that even with abstract syntax trees, (optional) annotations and static code analysis it will be (nearly?) impossible to find all places where a specific method of a specific class is called.

Answer 4

You can use ast or compiler module to dig through the compiled code and find out places where functions are explicitly called.

You can also just compile code with compile() with ast flag and get it parsed as abstract syntax tree. Then you go see what is called where within it.

But you can track down all what is happening during the codes execution using some tricks from sys, inspect and traceback modules.

For example, you can set your trace function that will snatch each interpreter frame before letting it be executed:

import dis
import sys
def tracefunc (frame, evt, arg):
    print frame.f_code.co_filename, frame.f_lineno, evt
    print frame.f_code.co_name, frame.f_code.co_firstlineno
    #print dis.dis(f.f_code)
sys.settrace(tracefunc)

After this code, every step done will be printed with file that contains the code, line of the step, where the code object begins and it will disassemble it so you can see all that is being done or will be done in background too (if you uncomment it).

If you want to match executed bytecode with Python code, you can use tokenize module. You make a cache of tokenized files as they appear in trace and snatch the Python code out of the corresponding lines whenever needed.

Using all mentioned stuff you can do wanders including writing byte code decompiler, jumping all over your code like with goto in C, forcefully interrupting threads (not recommended if you don't exactly know what you are upto), track which function called your function (nice for streaming servers to recognize clients catching up their parts of stream), and all sorts of crazy stuff.

Advanced crazy stuff I have to say. DON'T GO MESSING code flow in SUCH A MANNER UNLESS IT IS ABSOLUTELY NECESSARY and you don't know EXACTLY what you are doing.

I'll get down voted just because I mentioned such things are even possible.

Example for dynamically detecting which instance of client() tries to get the content:

from thread import get_ident
import sys

class Distributer:
    def read (self):
        # Who called me:
        cf = sys._current_frames()
        tid = get_ident() # Make it thread safe
        frame = cf[tid]
        # Now, I was called in one frame back so
        # go back and find the 'self' variable of a method that called me
        # and self, of course, contains the instance from which I was called
        client = frame.f_back.f_locals["self"]
        print "I was called by", client

class Client:
    def __init__ (self, name):
        self.name = name

    def snatch (self):
        # Now client gets his content:
        content.read()

    def __str__ (self):
        return self.name

content = Distributer()
clients = [Client("First"), Client("Second"), Client("Third"), Client("Fourth"), Client("Etc...")]
for client in clients:
    client.snatch()

Now, you write this within the tracing function instead of fixed method, but cleverly, not relying on variable names but on addresses and stuff and you can track what happens when and where. Big job, but possible.