8

When inside tracing function, debugging a function call, is it possible to somehow retrieve the calling expression?

I can get calling line number from traceback object but if there are several function calls (possibly to the same function) on that line (eg. as subexpression in a bigger expression) then how could I learn where this call came from? I would be happy even with the offset from start of the source line.

traceback.tb_lasti seems to give more granual context (index of last bytecode tried) -- is it somehow possible to connect a bytecode to its exact source range?

EDIT: Just to clarify -- I need to extract specific (sub)expression (the callsite) from the calling source line.

Aivar
  • 6,814
  • 5
  • 46
  • 78
  • 1
    will this help? http://stackoverflow.com/questions/13903751/how-to-see-complete-traceback/13903934#13903934 – Alexey Kachayev Dec 19 '12 at 12:03
  • @Alexey: unfortunately no, the source line is not precise enough for me, I need to extract specific subexpression. – Aivar Dec 19 '12 at 12:46
  • maybe if you paste the source line, associated bytecodes, and the instruction you are debugging, we can help you work out where in the source that instruction is coding for. – SingleNegationElimination Dec 19 '12 at 13:24
  • @TokenMacGuy: I'm not debugging any specific code, I want to create a visual expression debugger on top of Python tracing facilities. – Aivar Dec 19 '12 at 13:34

4 Answers4

6

Traceback frames have a line number too:

lineno = traceback.tb_lineno

You can also reach the code object, which will have a name, and a filename:

name = traceback.tb_frame.f_code.co_name
filename = traceback.tb_frame.f_code.co_filename

You can use the filename and line number, plus the frame globals and the linecache module to efficiently turn that into the correct source code line:

linecache.checkcache(filename)
line = linecache.getline(filename, lineno, traceback.tb_frame.f_globals)

This is what the traceback module uses to turn a traceback into a useful piece of information, in any case.

Since bytecode only has a line number associated with it, you cannot directly lead the bytecode back to the precise part of a source code line; you'd have to parse that line yourself to determine what bytecode each part would emit then match that with the bytecode of the code object.

You could do that with the ast module, but you can't do that on a line-by-line basis as you'd need scope context to generate the correct bytecodes for local versus cell versus global name look-ups, for example.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • The thing is that line number (or source line itself) is not granular enough for me, I need to extract specific callsite from that line. – Aivar Dec 19 '12 at 12:40
  • @Aivar: then you'd have to parse out the bytecode (from the `.f_code` object plus the offset). That's a lot more involved. – Martijn Pieters Dec 19 '12 at 12:41
  • @Martin: I don't doubt it's involved, but do you know, does bytecode contain source locations at line level only or at character level? From Python documentation I got the impression that bytecode instructions have only source line numbers associated with them. – Aivar Dec 19 '12 at 12:45
  • @Aivar: it has line numbers only. – Martijn Pieters Dec 19 '12 at 12:52
3

Unfortunately, compiled bytecode has lost its column offsets; the bytecode index to line number mapping is contained in the co_lnotab line number table. The dis module is a nice way of looking at the bytecode and interpreting co_lnotab:

>>> dis.dis(compile('a, b, c', '', 'eval'))
  1           0 LOAD_NAME                0 (a)
              3 LOAD_NAME                1 (b)
              6 LOAD_NAME                2 (c)
              9 BUILD_TUPLE              3
             12 RETURN_VALUE        
  ^-- line number

However, there's nothing stopping us from messing with the line number:

>>> a = ast.parse('a, b, c', mode='eval')
>>> for n in ast.walk(a):
...     if hasattr(n, 'col_offset'):
...         n.lineno = n.lineno * 1000 + n.col_offset
>>> dis.dis(compile(a, '', 'eval'))
1000           0 LOAD_NAME                0 (a)

1003           3 LOAD_NAME                1 (b)

1006           6 LOAD_NAME                2 (c)
              9 BUILD_TUPLE              3
             12 RETURN_VALUE

Since compiling code directly should be the same as compiling via ast.parse, and since messing with line numbers shouldn't affect the generated bytecode (other than the co_lnotab), you should be able to:

  • locate the source file
  • parse it with ast.parse
  • munge the line numbers in the ast to include the column offsets
  • compile the ast
  • use the tb_lasti to search the munged co_lnotab
  • convert the munged line number back to (line number, column offset)
ecatmur
  • 152,476
  • 27
  • 293
  • 366
1

That's how I finally solved the problem: I instrumented each function call in the original program by wrapping it in a call to a helper function together with information about the source location of the original call. Actually I was interested in controlling the evaluation of each subexpression in the program, so I wrapped each subexpression.

More precisely: when I had an expression e in the original program, it became 

_after(_before(location_info), e)

in the instrumented program. The helpers were defined like this:

def _before(location_info):
    return location_info

def _after(location_info, value):
    return value

When tracer reported the call to _before, I knew that it's about to evaluate the expression at location represented by location_info (tracing system gives me access to local variables/parameters, that's how I got to know the value of location_info). When tracer reported the call to _after, I knew that the expession indicated by location_info was just evaluated and the value is in value.

I could have written the execution "event handling" right into those helper functions and bypass the tracing system altogether, but I needed it for other reasons as well, so I used those helpers only for triggering a "call" event in tracing system.

The result can be seen here: http://thonny.org

Aivar
  • 6,814
  • 5
  • 46
  • 78
1

I know it's necromancy but I posted a similar question yesterday without seeing this one first. So just in case someone is interested, I solved my problem in a different way than the accepted answer by using the inspect and ast modules in Python3. It's still for debugging and educational purpose but it does the trick.

The answer is rather long so here is the link

Community
  • 1
  • 1
polyvertex
  • 749
  • 6
  • 15