Python how to print full stack, including magic methods (dunder methods) used?

Question

I am trying to debug a Python built-in class. My debugging has brought me into the realm of magic methods (aka dunder methods).

I am trying to figure out which dunder methods are called, if any. Normally I would do something like this:

import sys
import traceback

# This would be located where the I'm currently debugging
traceback.print_stack(file=sys.stdout)

However, traceback.print_stack does not give me the level of detail of printing what dunder methods area used in its vicinity.

Is there some way I can print out, in a very verbose manner, what is actually happening inside a block of code?

Sample Code

#!/usr/bin/env python3.6

import sys
import traceback
from enum import Enum


class TestEnum(Enum):
    """Test enum."""

    A = "A"


def main():
    for enum_member in TestEnum:
        traceback.print_stack(file=sys.stdout)
        print(f"enum member = {enum_member}.")


if __name__ == "__main__":
    main()

I would like the above sample code to print out any dunder methods used (ex: __iter__).

Currently it prints out the path to the call to traceback.print_stack:

/path/to/venv/bin/python /path/to/file.py
  File "/path/to/file.py", line 56, in <module>
    main()
  File "/path/to/file.py", line 51, in main
    traceback.print_stack(file=sys.stdout)
enum member = TestEnum.A.

P.S. I'm not interested in going to the byte code level given by dis.dis.

Lydia van Dyke · Accepted Answer · 2020-04-12T22:20:34.297

I think, with the stacktrace you are looking at the wrong place. When you call print_stack from a place, that is executed only when coming from a dunder method, this method is very well included in the output.

I tried this code to verify:

import sys
import traceback
from enum import Enum


class TestEnum(Enum):
    """Test enum."""

    A = "A"


class MyIter:

    def __init__(self):
        self.i = 0

    def __next__(self):
        self.i += 1
        if self.i <= 1:
            traceback.print_stack(file=sys.stdout)
            return TestEnum.A
        raise StopIteration

    def __iter__(self):
        return self


def main():
    for enum_member in MyIter():
        print(f"enum member = {enum_member}.")


if __name__ == "__main__":
    main()

The last line of the stack trace is printed as

File "/home/lydia/playground/demo.py", line 21, in __next__
traceback.print_stack(file=sys.stdout)

In your original code, you are getting the stack trace at a time when all dunder methods have already returned. Thus they have been removed from the stack.

So I think, you want to have a look at a call graph instead. I know that IntelliJ / PyCharm can do this nicely at least in the paid editions.

There are other tools that you may want to try. How does pycallgraph look to you?

Update:

Python makes it actually pretty easy to dump a plain list of all the function calls.

Basically all you need to do is

import sys
sys.setprofile(tracefunc)

Write the tracefunc depending on your needs. Find a working example at this SO question: How do I print functions as they are called

Warning: I needed to start the script from an external shell. Starting it by using the play button in my IDE meant that the script would never terminate but write more and more lines. I assume it collides with the internal profiling done by my IDE.

The official documentation of sys.setprofile: https://docs.python.org/3/library/sys.html#sys.setprofile

And a random tutorial about tracing in Python: https://pymotw.com/2/sys/tracing.html

Note however, that by my experience you can get the best insights into the questions "who is calling whom?" or "where does this value even come from?" by using a plain-old debugger.

Thank you @LydiaVanDyke for the great answer! I gave `pycallgraph` a try, and I think that's quite useful. I didn't know about the whole world of profiling. Per this answer: https://stackoverflow.com/q/582336/11163122 I see there's other resources. — Intrastellar Explorer, Apr 12 '20 at 21:29
Can you suggest any tools that just print the full stack history in a text-based manner, as opposed to the visual manner of `pycallgraph`? — Intrastellar Explorer, Apr 12 '20 at 21:30
I doubt that I can find any that is not already mentioned in the SO thread that you posted. `python -m cProfile` would have been my next - and that is shown in the second answer (got more up votes than the top one). The official documentation seems not to be linked though: https://docs.python.org/3/library/profile.html — Lydia van Dyke, Apr 12 '20 at 21:40
Thank you so much @LydiaVanDyke!! That is a lot of great information :) your searching skills are much better than mine! — Intrastellar Explorer, Apr 13 '20 at 02:15

Intrastellar Explorer · Answer 2 · 2020-05-09T08:56:49.657

I also did some research on the subject matter, as information in @LydiaVanDyke's answer fueled better searches.

Printing Entire Call Stack

As @LydiaVanDyke points out, an IDE debugger is a really great way. I use PyCharm, and found that was my favorite solution, because one can:

Follow function calls + exact line numbers in the code
Read code around the calls, better understanding typing
Skip over calls one doesn't care to investigate

Another way is Python's standard library's trace. It offers both command line and embeddable methods for printing the entire call stack.

And yet another one is Python's built-in debugger module, pdb. This (invoked via pdb.set_trace()) really changed the game for me.

Visualization of Profiler Output

gprof2dot is another useful profiler visualization tool.

Finding Source Code

One of my other problems was not actually seeing the real source code, due to my IDE's stub files (PyCharm).

How to retrieve source code of Python functions details two methods of actually printing source code

With all this tooling, one feels quite empowered!

Python how to print full stack, including magic methods (dunder methods) used?

2 Answers2