I'm trying to debug a deadlock in a multi-threaded Python application after it has locked up. Is there a way to attach a debugger to inspect the state of the process?
Edit: I'm attempting this on Linux, but it would be great if there were a cross-platform solution. It's Python after all :)
Use Winpdb. It is a platform independent graphical GPL Python debugger with support for remote debugging over a network, multiple threads, namespace modification, embedded debugging, encrypted communication and is up to 20 times faster than pdb.
Features:
(source: winpdb.org)
Yeah, gdb is good for lower level debugging.
You can change threads with the thread command.
e.g
(gdb) thr 2
[Switching to thread 2 (process 6159 thread 0x3f1b)]
(gdb) backtrace
....
You could also check out Python specific debuggers like Winpdb, or pydb. Both platform independent.
You can attach a debugger to a multi-threaded Python process, but you need to do it at the C level. To make sense of what's going on, you need the Python interpreter to be compiled with symbols. If you don't have one, you need to download source from python.org and build it yourself:
./configure --prefix=/usr/local/pydbg
make OPT=-g
sudo make install
sudo ln -s /usr/local/pydbg/bin/python /usr/local/bin/dbgpy
Make sure your workload is running on that version of the interpreter. You can then attach to it with GDB at any time. The Python folks have included a sample ".gdbinit" in their Misc directory, which has some useful macros. However it's broken for multi-threaded debugging (!). You need to replace lines like this
while $pc < Py_Main || $pc > Py_GetArgcArgv
with the following:
while ($pc < Py_Main || $pc > Py_GetArgcArgv) && ($pc < t_bootstrap || $pc > thread_PyThread_start_new_thread)
Otherwise commands like pystack won't terminate on threads other than the main thread. With this stuff in place, you can do stuff like
gdb> attach <PID>
gdb> info threads
gdb> thread <N>
gdb> bt
gdb> pystack
gdb> detach
and see what's going on. Kind of.
You can parse what the objects are with the "pyo" macro. Chris has some examples on his blog.
Good luck.
(Shoutout for Dan's blog for some key information for me, notably the threading fix!)
My experience debugging multi-threaded programs in PyDev (Eclipse on Windows XP) is, threads created using thread.start_new_thread could not be hooked, but thread created using threading.Thread could be hooked. Hope the information is helpful.
If you mean the pydb, there is no way to do it. There was some effort in that direction: see the svn commit, but it was abandoned. Supposedly winpdb supports it.
python3 provides gdb extensions. Using them, gdb can attach to a running program, select a thread and print its python backtrace.
On Debian (since at least Buster) the extensions are part of the
python3.x-dbg package (ex. python3.10-dbg installs
/usr/share/gdb/auto-load/usr/bin/python3.10-gdb.py) and gdb auto-loads them.
Example with a simple threaded python script:
#!/usr/bin/env python3
import signal
import threading
def a():
while True:
pass
def b():
while True:
signal.pause()
threading.Thread(target=a).start()
threading.Thread(target=b).start()
Running gdb:
user@vsid:~$ ps -C python3 -L
PID LWP TTY TIME CMD
1215 1215 pts/0 00:00:00 python3
1215 1216 pts/0 00:00:19 python3
1215 1217 pts/0 00:00:00 python3
user@vsid:~$ gdb -p 1215
GNU gdb (Debian 10.1-2+b1) 10.1.90.20210103-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
[...]
(gdb) info auto-load python-scripts
Loaded Script
Yes /usr/share/gdb/auto-load/usr/bin/python3.10-gdb.py
(gdb) info threads
Id Target Id Frame
* 1 Thread 0x7f2f034b4740 (LWP 1215) "python3" 0x00007f2f036a60fa in __futex_abstimed_wait_common64 (futex_word=futex_word@entry=0x7f2ef4000b60, expected=expected@entry=0,
clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>,
cancel=cancel@entry=true) at ../sysdeps/nptl/futex-internal.c:74
2 Thread 0x7f2f02ea7640 (LWP 1216) "python3" 0x000000000051b858 in _PyEval_EvalFrameDefault
(tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>)
at ../Python/ceval.c:3850
3 Thread 0x7f2f026a6640 (LWP 1217) "python3" 0x00007f2f036a3932 in __libc_pause ()
at ../sysdeps/unix/sysv/linux/pause.c:29
(gdb) thread 2
(gdb) py-bt
Traceback (most recent call first):
File "/root/./threaded.py", line 7, in a
while True:
File "/usr/lib/python3.10/threading.py", line 946, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.10/threading.py", line 1009, in _bootstrap_inner
self.run()
File "/usr/lib/python3.10/threading.py", line 966, in _bootstrap
self._bootstrap_inner()
(gdb)
We can confirm that thread 1216 which used the most cpu time according to
ps is indeed the thread running function a() that is busy-looping.
pdbinject allows you to inject pdb into an already running python process.
The pdbinject executable only works under python2, but can inject into python3 just fine too.
PyCharm IDE allows attaching to a running Python process since version 4.0.
Here is described how to do that.
What platform are you attempting this on? Most debuggers allow you to attach to a running process by using the process id. You can either output the process id via logging or using something like Task Manager. Once that is achieved it will be possible to inspect individual threads and their call stacks.
EDIT: I don't have any experience with GNU Debugger (GDB), which is cross platform, however I found this link and it may start you on the right path. It explains how to add debug symbols (handy for reading stack traces) and how to instruct gdb to attach to a running python process.
This can be used as a dead simple "remote" debugger:
import sys
import socket
import pdb
def remote_trace():
server = socket.socket()
server.bind(('0.0.0.0', 12345))
server.listen()
client, _= server.accept()
stream = client.makefile('rw')
sys.stdin = sys.stdout = sys.stderr = stream
pdb.set_trace()
remote_trace()
# Execute in the shell: `telnet 127.0.0.1 12345`
On Windows it's easier to use Netcat instead of Telnet (which will also work on linux).
