25

Is there a way on Linux to check what a running Python daemon process is doing? That is, without instrumenting the code and without terminating it? Preferably I'd like to get the name of the module and the line number in it that is currently running.

Conventional debugging tools such as strace, pstack and gdb are not very useful for Python code. Most stack frames just contain functions from the interpreter code like PyEval_EvalFrameEx and and PyEval_EvalCodeEx, it doesn't give you any hint on were in the .py-file the execution is.

Nickolay
  • 31,095
  • 13
  • 107
  • 185
Björn Lindqvist
  • 19,221
  • 20
  • 87
  • 122

8 Answers8

11

Some of the answers in Showing the stack trace from a running Python application are applicable in this situation:

Community
  • 1
  • 1
Nickolay
  • 31,095
  • 13
  • 107
  • 185
  • 1
    If you want to use pyrasite and gdb in a Docker container, then you need to run the container with the following flags: `--cap-add=SYS_PTRACE --privileged` and you need to exec into the container as root (even if a dedicated app user runs the process in the container) for which you will need: `docker exec -it --user root -w=/root [container id] bash` – Zoltan Fedor Feb 20 '19 at 17:01
  • What is `dump_stacks.py`? When I run these commands with pyrasite installed nothing happens. Pyrasite just exits silently. – Björn Lindqvist Jul 07 '21 at 18:33
  • @BjörnLindqvist it's [one of the sample payloads](https://github.com/lmacken/pyrasite/blob/master/pyrasite/payloads/dump_stacks.py). I'm not the best person to help you troubleshoot this, so perhaps a separate question is warranted. – Nickolay Jul 07 '21 at 21:01
  • I cannot accept the answer if I cannot get it to work. – Björn Lindqvist Jul 14 '21 at 17:30
  • @BjörnLindqvist sorry, didn't realize you were the OP. Still as this worked for me 6 years ago, the answer might help someone even if not marked as accepted... – Nickolay Jul 14 '21 at 22:49
  • 1
    @BjörnLindqvist probably too late, but: use `--verbose` to see the problem; in my case, I had to install `gdb` – Valentin Golev Mar 16 '22 at 09:22
4

py-spy (https://github.com/benfred/py-spy) has a few useful tools for inspecting running Python processes. In particular, py-spy dump will print a stack trace (including function, file, and line) for every thread.

Sophie Alpert
  • 139,698
  • 36
  • 220
  • 238
2

on POSIX systems like Linux, you can use good old GDB, see

There's also the excellent PyCharm IDE (free community version available) that can attach to a running Python process right from within the IDE, using Pdb 4 under the hood, see this blog entry:

Gregor
  • 1,297
  • 1
  • 19
  • 31
2

winpdb allows you to attach to a running python process, but to do this, you must start the python process this way:

 rpdb2 -d -r script.py

Then, after setting a password:

A password should be set to secure debugger client-server communication.
Please type a password:mypassword

you could launch winpdb to File>Attach to (or File>Detach from) the process.

unutbu
  • 842,883
  • 184
  • 1,785
  • 1,677
1

lptrace does exactly that. It allows you to attach to a running Python process and show currently executing functions, like strace does for system calls. You can call it like this:

vagrant@precise32:/vagrant$ sudo python lptrace -p $YOUR_PID
fileno (/usr/lib/python2.7/SocketServer.py:438)
meth (/usr/lib/python2.7/socket.py:223)

fileno (/usr/lib/python2.7/SocketServer.py:438)
meth (/usr/lib/python2.7/socket.py:223)
...

Note that it requires gdb to run, which isn't available on every server machine.

Karim H
  • 61
  • 2
1

You can use madbg (by me). It is a python debugger that allows you to attach to a running python program and debug it in your current terminal. It is similar to pyrasite and pyringe, but newer, doesn't require gdb, and uses IPython for the debugger (which means colors and autocomplete).

To see the stack trace of a running program, you could run:

madbg attach <pid>

And in the debugger shell, enter: bt

kmaork
  • 5,722
  • 2
  • 23
  • 40
  • Worked for me on the first try, while pyrasite (topvoted solution at the time of writing) didn't. – Sylvain Mar 09 '23 at 14:48
0

It's possible to debug Python with gdb. See Chapter 22: gdb Support in the Python Developer’s Guide.

For example, on Debian with Python 3.7:

# apt-get update -y && apt-get install gdb python3.7-dbg
# gdb

(gdb) source /usr/share/gdb/auto-load/usr/bin/python3.7-gdb.py
(gdb) attach <PID>
(gdb) py-bt
jrc
  • 20,354
  • 10
  • 69
  • 64
  • What is python3.7-dbg? I'm using Arch Linux and I don't think I have a corresponding package. – Björn Lindqvist Jun 24 '21 at 02:09
  • Not sure about Arch Linux but this page (outdated) mentions some other distros… see "Debugging Python with gdb 101": https://www.podoliaka.org/2016/04/10/debugging-cpython-gdb/ – jrc Jun 24 '21 at 09:27
  • You might try simply `python-dbg`: https://aur.archlinux.org/packages/python-dbg/ – jrc Jun 24 '21 at 09:29
  • There is something wrong with that AUR package and I could't install it. – Björn Lindqvist Jul 07 '21 at 18:25
0

You can also use satella to do this. A nice side effect will be that every local variable in every stack frame will be printed out. The code would be:

from satella.instrumentation import Traceback
import sys

for frame_no, frame in sys._current_frames().items():
    sys.stderr.write("For stack frame %s" % (frame_no,))
    tb = Traceback(frame)
    tb.pretty_print()
sys.stderr.write("End of stack frame dump\n")
Piotr Maślanka
  • 62
  • 7