2

In my python debugger I have a way of remapping a string to a filename so that when you are stepping through an exec'd function inside the debugger you can list lines pygmentized, or view them along inside an editor like Emacs via realgud.

So I'd like to be able to extract the string in an exec statement when CPython is stopped inside evaluating that.

I already have a mechanism that can look back in the call frame to see if the caller was an EXEC_STMT and I can look back one instruction to see if the previous instruction was say DUP_TOP. So I'd be home free if I could just figure out a way to read the stack entry at the time of the call and that gives the string evaluated. There is probably a way to drop into C to get this, but my knowledge of CPython internals lacking, and would prefer not to do this. If there's a package out there, maybe I could include that optionally.

CPython already provides access to function arguments, and local variables but of course since this is a built-in function this isn't recorded as a function parameter.

If there are other thoughts at how to do the same thing, that'd be okay too. I feel a less good solution would be to somehow try to overload or replace exec since debuggers can be brought in late in the game.

I understand that CPython2 and CPython3 may be a little bit different here, but to start off either would do.

rocky
  • 7,226
  • 3
  • 33
  • 74

3 Answers3

1

I think I've now found a way.

Inside the debugger I go up the call stack one level to get to the exec statement. Then I can use uncompyle6 to get a syntax tree of the source code. (A change may be needed in uncompyle6 to make this easier.)

The tree at the point of call will have something like exec_stmt -> expr .... That expression will have the text of the expression which is not necessarily the value of the expression. The expression could be a constant string value, but it could be something complex like "foo" + var1.

So then the debugger can evaluate that string in the context of the debugger which knows how to evaluate expressions up the call stack.

This still has a problem of the reevaluating the expression may have side effects. But that's bad programming practice, right? ;-)

So instead what I do is just decompile the code from the bytecode if the source isn't there. This has a disadvantage in that the line numbers mentioned in the bytecode don't always line up with those in the bytecode. For that the method of recreating the string above is better.

In closing, I hope to give some idea why writing a really good debugger is hard and why the vast number of debuggers have a number of limitations on even simple things like getting the source text at the point you are currently stopped.

A totally different approach would be to stop early and switch to an sub-interpreter like x-python (or some suitably modified Python C module) which would have access to a stack.

rocky
  • 7,226
  • 3
  • 33
  • 74
  • Does uncompyle6 provide an API to be used from code? – laike9m May 27 '19 at 07:32
  • uncompyle6 has api's and trepan3k uses them. Whether that answers your question, I dunno. Little effort in posing a question -> little thought in providing an answer. Pssst - all of this code is open source. – rocky May 27 '19 at 12:15
  • Thanks. I found the usage from trepan3k's source. Just noticed you're the author of those. Maybe I'm wrong, it seems the API is not documented? – laike9m May 27 '19 at 18:34
  • I've heard that before. I'm not sure what you are expecting. Since this is open source in gratitude how about if you make a stab at documenting this in a way that is pleasing to you at least to the point that it satisfies what you were looking for? Thanks. – rocky May 27 '19 at 20:06
  • This has the assumption that the `exec` is on the current stack. However, when the `exec` created another function which you call later, or registered some callback which is executed later, then I think there is no reference anymore to the code string. – Albert Mar 10 '22 at 13:37
  • The question makes clear that this in the context of a debugger, and that I am looking at the caller for a string. So yes, I assume there will be a stack entry with a string even that string was created by a function and sometimes is even if it is f-string format substitution. If you want this information from say the source code, well then the only alternative is to run or simulate run the code. And for that I also have https://pypi.org/project/x-python/ which can help there. – rocky Mar 10 '22 at 14:03
0

The open-source Thonny IDE has [sub]expression evaluation stepping. See the author's answer to the SO question Tracing Python expression evaluation step by step.

Abraham
  • 2,860
  • 1
  • 20
  • 9
  • I see you haven't given up on this problem. Thanks! I have looked at Thonny and like it. However overcoming the changes I suspect will be harder than using one of the other approaches. Note first that Thonny uses 3.4. or greater while my debuggers support 2.6 on; `exec` as a reserved word appears only in Python 2.x. [continued...] – rocky Jun 18 '17 at 17:52
  • But that aside, how would this work? Well, I'd _first_ have to deparse the code, and then using Thonny it would use the source text to create a Python AST that it can interpret. Again papering over the fact that Python source and ASTs change a bit between Python versions, I'd still have to seed values of variables in that frame. So, simpler I think would just be deparse the expression that gets fed into `eval` or `exec` and then feed that into a separate interpreter that has read access to stack entries. – rocky Jun 18 '17 at 17:54
0

Not exactly an answer but this might be a solution in some cases to this problem.

You can provide your own custom exec function which extends linecache to include this. Many tracebacks will then include the code. Like:

def custom_exec(code_str, _globals=None, _locals=None):
  compile_string_fn = f"<custom code str {hash(code_str)}>"
  c = compile(code_str, compile_string_fn, "exec")
  set_linecache(compile_string_fn, code_str)
  exec(c, _globals, _locals)

With:

def set_linecache(filename, source):
  import linecache
  linecache.cache[filename] = None, None, [line+'\n' for line in source.splitlines()], filename
Albert
  • 65,406
  • 61
  • 242
  • 386