13

I have a django application using a C++ library (imported via swig). The C++ library launches own thread which calls callbacks in Python code.

I cannot setup a breakpoint in python code, neither in PyDev nor PyCharm. Tried also 'gevent compatibility' option too with no luck.

I verified the callbacks are properly called as logging.info dumps what expected. Breakpoints set in other threads work fine. So it seems that python debuggers cannot manage breakpoints in python code called by threads created in non-python code.

Does anyone know a workaround? Maybe there is some 'magic' thread initialization sequence I could use?

Alek Kowalczyk
  • 710
  • 6
  • 21

2 Answers2

24

You have to setup the debugger machinery for it to work on non-python threads (this is done automatically when a Python thread is created, but when you create a thread for which Python doesn't have any creation hook, you have to do it yourself) -- note that for some frameworks -- such as QThread/Gevent -- things are monkey patched so that we know about the initialization and start the debugger, but for other frameworks you have to do it yourself.

To do that, after starting the thread you have to call:

import pydevd
pydevd.settrace(suspend=False, trace_only_current_thread=True)

Note that if you had put suspend=True, it'd simulate a manual breakpoint and would stop at that point of the code.

For a real-world case, if you have a callback function and that only that function in your current script is called by another thread from another program, say C++, then you will have to put the call inside the function instead of calling it right after the import statement. See here about this example in PyTorch.

NeoZoom.lua
  • 2,269
  • 4
  • 30
  • 64
Fabio Zadrozny
  • 24,814
  • 4
  • 66
  • 78
  • Thank you @Fabio Zadrozny ! Works flawlessly. As my callbacks were methods, I just created a metaclass wrapping them with initialization. – Alek Kowalczyk Dec 19 '15 at 13:18
  • 2
    I'm sorry but I'm not sure how I'm supposed to use this code. Does it mean that I have to put a line of pydevd.settrace(suspend=true....) at the place where I want the debugging to break at? If so does it mean that I have to modify the source code(which is not very convenient of a debugging experience) and the features such as conditional breakpoint won't work? – tete Mar 26 '21 at 11:28
  • Note that on Python 3.7 onwards (if you have the pydevd compiled extensions), this is no longer needed. Still if you're on an older version or don't have the compiled extensions, then you need to add that call once per thread so that the thread is traced and breakpoints work on it. – Fabio Zadrozny Mar 26 '21 at 17:53
  • @FabioZadrozny thank you for your reply. I asked because I encountered this problem and it is indeed a py 3.7 environment. So do you know if it should work in this setup(you.7, callback initialized in a c++ thread)? And I can actually make it stop by putting a line of "breakpoint()" But like I mentioned it is not a convenient way to debug. Because I need to modify the source code and I can't make a conditional breakpoint – tete Mar 28 '21 at 01:42
  • I just figured out the cause of my problem: I put the line `import pydevd; pydevd.settrace(suspend=False, trace_only_current_thread=True)` on top of my .py file, whereas I should put it on the callback function (or on the line where I want the program to break. But the second part made me hesitate because I thought I needed to write this line every time I want to make a new breakpoint). So now after I put this line in the beginning of my callbacks, I am able to make it stop by adding a breakpoint in the IDE (VS Code in my case) – tete Mar 29 '21 at 02:24
  • 1
    For anybody who has used debugpy.debug_this_thread(), this is the equivalent you are looking for. – Joseph Bradshaw Sep 15 '21 at 11:03
  • It's not clear where I should add this line of code. Am I correct that I should add this line where I set the breakpoints? – NeoZoom.lua May 22 '23 at 04:49
  • Can confirm that putting the settrace later worked. Not directly in the foreign code after importing, but right in the python function itself one is calling. However, does anyone know why is it the case? Why does settrace right after import not work? – SoptikHa Jun 13 '23 at 09:14
3

This is a follow-up to @fabio-zadrozny answer.

Here is a mixin I've created that my class (which gets callbacks from a C-thread) inherits from.

class TracingMixing(object):
    """The callbacks in the FUSE Filesystem are C threads and breakpoints don't work normally.
       This mixin adds callbacks to every function call so that we can breakpoint them."""

    def __call__(self, op, path, *args):
        pydevd.settrace(suspend=False, trace_only_current_thread=True, patch_multiprocessing=True)
        return getattr(self, op)(path, *args)
Setheron
  • 3,520
  • 3
  • 34
  • 52
  • 1
    I'm writing an FS using `python-fuse`. I'm using PyCharm. My breakpoints are not being hit. I tried to put your `TracingMixing` class in my code, but it doesn't seem to have any effect. Could you please give a more detailed example of how to use it. Thanks. – nonbeing Feb 09 '17 at 10:24