0

I've embarked on a rather ambitious project in my dev environment because I like keeping up to date I guess. So I've been working on a Django project with Eclipse and PyDev for years and it works beautifully.

I've been working with Python 3.7 an Django 2.1 and Eclipse Photon for some while.

I've tried a kind of complete new start of sorts as follows moving to latest version of everything:

  1. Installed latest Eclipse (2019-12 Eclipse IDE for Web and JavaScript Developers (includes Incubating components)
  2. Add PyDev to it (Help > Install New Software, from http://www.pydev.org/updates)
  3. Installed Python 3.8.
  4. Installed Django 3.0.1.
  5. Installed all the packages my project depends on into my local 3.8 site-packages
  6. In a new workspace I imported my old project (File > Import > General > Existing Projects into Workspace, checked Copy projects into workspace
  7. Created a debug configuration with the Python 3.8 interpreter and (because for some reason the default sys.path put my local site-packages after python 3 dist-packages) I added site-packages to my external libraries and forced it to the top of my sys.path.
  8. Started a Debug run ...

This works fine. It starts, I can work with my website pointing at http://127.0.0.1:8000/ and it's mostly functional, in fact can't complain, almost expected more dramas in upgrading. Not perfect had a few tweaks to make to get it Django 3.0.1 compatible but surprisingly few.

I am stuck on page which is crashing though and so I set a breakpoint only to discover it's not working ... Hmmmm. I have been here before with PyDev and so have many others. So I have performed the basic diags already and will share what I know. But I would be most grateful if someone (not least Fabio) had a further gem of wisdom to lend to get them working.

Installed on Eclipse are:

  PyDev for Eclipse 7.4.0.201910251334  org.python.pydev.feature.feature.group  Fabio Zadrozny
  PyDev for Eclipse Developer Resources 7.4.0.201910251334  org.python.pydev.feature.source.feature.group   Fabio Zadrozny
  Pydev Mylyn Integration   0.6.0   org.python.pydev.mylyn.feature.feature.group    Fabio Zadrozny

Key Observations:

  1. If I set a breakpoint in settings.py it breaks fine.
  2. If I set a breakpoint in a view it does not break.
  3. I know I'm in the right file and place because if I add a console print line at the place I want to break, it is printed to the console. So I'm not looking at the wrong source file in any way shape or form.

Fabio left some really nice (dated now) tips here: pydev breakpoints not working and I can say in response to those:

  1. If I put this at top of views.py where Iw ant my breakpoint: 
import sys
print('DEBUG: current trace function', sys.gettrace())

I see this output:

DEBUG: current trace function None

which seems to be the problem. That something is turning off debugging.

Following Fabio's first tip and accounting for changes in PyDev structure since that tip, I found pydevd_constants.py in plugins/org.python.pydev.core_7.4.0.201910251334/pysrc/_pydevd_bundle and in it could set:

DEBUG_TRACE_LEVEL = 3 
DEBUG_TRACE_BREAKPOINTS = 3

As Fabio suggested. This reveals little alas. Essentially I see lots of cool tracing when start the debug run until the server is running. Then I see the standard output saying the sever is now running:

System check identified no issues (0 silenced).
December 26, 2019 - 12:29:54
Django version 3.0.1, using settings 'MyProject.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

It stops on the breakpoint in settings.py fine along the way and I continue with F8.

Now in my browser I load the view I'm debugging. And there and zero PyDev traces on the console alas, in fact I get only my little printed line I put int he view's get_queryset() method to ensure my code (with the breakpoint) is running.

So that didn't help any. I can in my views.py file then look for a needle int he haystack with Fabio's suggestion:

import sys
print('DEBUG: current trace function', sys.gettrace())
def trace_func(frame, event, arg):
    with open('pydev-trace.txt', 'a') as f:
        print('Context: ', frame.f_code.co_name, '\tFile:', frame.f_code.co_filename, '\tLine:', frame.f_lineno, '\tEvent:', event, file=f)
    return trace_func

sys.settrace(trace_func)
print('DEBUG: current trace function', sys.gettrace())

and on the console I now see:

DEBUG: current trace function None
DEBUG: current trace function <function trace_func at 0x7ffaa6f18b80>
System check identified no issues (0 silenced).
December 26, 2019 - 12:42:48
Django version 3.0.1, using settings 'MyProject.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

Which sort of looks good.

But the output trace seems never to stop. It grows and grows and grows and seems stuck in a cycle of autoreloads in an endlessly looping cycle of:

Context:  snapshot_files    File: /home/me/.local/lib/python3.8/site-packages/django/utils/autoreload.py    Line: 370   Event: call
Context:  snapshot_files    File: /home/me/.local/lib/python3.8/site-packages/django/utils/autoreload.py    Line: 361   Event: line
Context:  watched_files     File: /home/me/.local/lib/python3.8/site-packages/django/utils/autoreload.py    Line: 260   Event: call
Context:  watched_files     File: /home/me/.local/lib/python3.8/site-packages/django/utils/autoreload.py    Line: 260   Event: return
Context:  snapshot_files    File: /home/me/.local/lib/python3.8/site-packages/django/utils/autoreload.py    Line: 362   Event: line
Context:  __hash__  File: /usr/lib/python3.8/pathlib.py     Line: 758   Event: call
Context:  __hash__  File: /usr/lib/python3.8/pathlib.py     Line: 759   Event: line
Context:  __hash__  File: /usr/lib/python3.8/pathlib.py     Line: 760   Event: line
Context:  __hash__  File: /usr/lib/python3.8/pathlib.py     Line: 760   Event: return
Context:  snapshot_files    File: /home/me/.local/lib/python3.8/site-packages/django/utils/autoreload.py    Line: 364   Event: line
Context:  snapshot_files    File: /home/me/.local/lib/python3.8/site-packages/django/utils/autoreload.py    Line: 365   Event: line
Context:  stat  File: /usr/lib/python3.8/pathlib.py     Line: 1186  Event: call
Context:  stat  File: /usr/lib/python3.8/pathlib.py     Line: 1191  Event: line
Context:  __fspath__    File: /usr/lib/python3.8/pathlib.py     Line: 721   Event: call
Context:  __fspath__    File: /usr/lib/python3.8/pathlib.py     Line: 722   Event: line
Context:  __str__   File: /usr/lib/python3.8/pathlib.py     Line: 711   Event: call
Context:  __str__   File: /usr/lib/python3.8/pathlib.py     Line: 714   Event: line
Context:  __str__   File: /usr/lib/python3.8/pathlib.py     Line: 715   Event: line
Context:  __str__   File: /usr/lib/python3.8/pathlib.py     Line: 715   Event: return
Context:  __fspath__    File: /usr/lib/python3.8/pathlib.py     Line: 722   Event: return
Context:  stat  File: /usr/lib/python3.8/pathlib.py     Line: 1191  Event: return
Context:  snapshot_files    File: /home/me/.local/lib/python3.8/site-packages/django/utils/autoreload.py    Line: 369   Event: line
Context:  __hash__  File: /usr/lib/python3.8/pathlib.py     Line: 758   Event: call
Context:  __hash__  File: /usr/lib/python3.8/pathlib.py     Line: 759   Event: line
Context:  __hash__  File: /usr/lib/python3.8/pathlib.py     Line: 760   Event: line
Context:  __hash__  File: /usr/lib/python3.8/pathlib.py     Line: 760   Event: return
Context:  snapshot_files    File: /home/me/.local/lib/python3.8/site-packages/django/utils/autoreload.py    Line: 370   Event: line
Context:  snapshot_files    File: /home/me/.local/lib/python3.8/site-packages/django/utils/autoreload.py    Line: 370   Event: return
Context:  tick  File: /home/me/.local/lib/python3.8/site-packages/django/utils/autoreload.py    Line: 346   Event: line
Context:  __hash__  File: /usr/lib/python3.8/pathlib.py     Line: 758   Event: call
Context:  __hash__  File: /usr/lib/python3.8/pathlib.py     Line: 759   Event: line
Context:  __hash__  File: /usr/lib/python3.8/pathlib.py     Line: 760   Event: line
Context:  __hash__  File: /usr/lib/python3.8/pathlib.py     Line: 760   Event: return
Context:  __eq__    File: /usr/lib/python3.8/pathlib.py     Line: 753   Event: call
Context:  __eq__    File: /usr/lib/python3.8/pathlib.py     Line: 754   Event: line
Context:  __eq__    File: /usr/lib/python3.8/pathlib.py     Line: 756   Event: line
Context:  _cparts   File: /usr/lib/python3.8/pathlib.py     Line: 744   Event: call
Context:  _cparts   File: /usr/lib/python3.8/pathlib.py     Line: 747   Event: line
Context:  _cparts   File: /usr/lib/python3.8/pathlib.py     Line: 748   Event: line
Context:  _cparts   File: /usr/lib/python3.8/pathlib.py     Line: 748   Event: return
Context:  _cparts   File: /usr/lib/python3.8/pathlib.py     Line: 744   Event: call
Context:  _cparts   File: /usr/lib/python3.8/pathlib.py     Line: 747   Event: line
Context:  _cparts   File: /usr/lib/python3.8/pathlib.py     Line: 748   Event: line
Context:  _cparts   File: /usr/lib/python3.8/pathlib.py     Line: 748   Event: return
Context:  __eq__    File: /usr/lib/python3.8/pathlib.py     Line: 756   Event: return
Context:  tick  File: /home/me/.local/lib/python3.8/site-packages/django/utils/autoreload.py    Line: 347   Event: line
Context:  __hash__  File: /usr/lib/python3.8/pathlib.py     Line: 758   Event: call
Context:  __hash__  File: /usr/lib/python3.8/pathlib.py     Line: 759   Event: line
Context:  __hash__  File: /usr/lib/python3.8/pathlib.py     Line: 760   Event: line
Context:  __hash__  File: /usr/lib/python3.8/pathlib.py     Line: 760   Event: return
Context:  __eq__    File: /usr/lib/python3.8/pathlib.py     Line: 753   Event: call
Context:  __eq__    File: /usr/lib/python3.8/pathlib.py     Line: 754   Event: line
Context:  __eq__    File: /usr/lib/python3.8/pathlib.py     Line: 756   Event: line
Context:  _cparts   File: /usr/lib/python3.8/pathlib.py     Line: 744   Event: call
Context:  _cparts   File: /usr/lib/python3.8/pathlib.py     Line: 747   Event: line
Context:  _cparts   File: /usr/lib/python3.8/pathlib.py     Line: 748   Event: line
Context:  _cparts   File: /usr/lib/python3.8/pathlib.py     Line: 748   Event: return
Context:  _cparts   File: /usr/lib/python3.8/pathlib.py     Line: 744   Event: call
Context:  _cparts   File: /usr/lib/python3.8/pathlib.py     Line: 747   Event: line
Context:  _cparts   File: /usr/lib/python3.8/pathlib.py     Line: 748   Event: line
Context:  _cparts   File: /usr/lib/python3.8/pathlib.py     Line: 748   Event: return
Context:  __eq__    File: /usr/lib/python3.8/pathlib.py     Line: 756   Event: return
Context:  tick  File: /home/me/.local/lib/python3.8/site-packages/django/utils/autoreload.py    Line: 348   Event: line
Context:  tick  File: /home/me/.local/lib/python3.8/site-packages/django/utils/autoreload.py    Line: 351   Event: line
Context:  tick  File: /home/me/.local/lib/python3.8/site-packages/django/utils/autoreload.py    Line: 345   Event: line

and that is the best clue we have to date. I suspect something here is in a very tight unending loop. Still the webpage loads so it's not draining resources at a breakneck speed and it runs for ages and doesn't overflow any stacks or anything but clearly seems broken somehow.

But it's getting very hard to pin something down here.

Finally in that response Fabio suggested that a brute force fix might be:

import pydevd;
pydevd.settrace()

Alas could not import pydevd, but if I add to my External libraries:

...plugins/org.python.pydev.core_7.4.0.201910251334/pysrc

I can, and then run it but it produces an endless stream of text on my console and my website won't respond while that's happening. So it's brute force in the sense of brining by debug server down ;-). This may relate to endless cycle above.

Other checks:

I checked Fabios list of needs:

http://www.pydev.org/manual_adv_django.html

and on those I'm good.

  1. The project is marked as a Django project.
  2. DJANGO_MANAGE_LOCATION is set.
  3. DJANGO_SETTINGS_MODULE is set.

I can see the latter two on Project Properties > PyDev PYTONPATH > String Substitution Variables. And I cna infer the first from the presence of Django menus on the projects context Menu and absense of the option to Set as Django Project.

At this point I'm hoping for some further pointers. I have invest a good deal of time into diagnosis here and trying to uncover a cause but my efficiency at this point will skyrocket with some injected expertise and advice. Fabio?

I am asking here, in no small part because many otehrs have, and it is a superbly useful well findable resource for anyone facing a similar problem. And there's no reason to believe that unlikely given, the circumstances. So I'd like this solved, and a solution posted basically for my sake and others'.

Bernd Wechner
  • 1,854
  • 1
  • 15
  • 32

2 Answers2

1

Some things to check:

  1. Are those in 2 different processes? i.e.: if you import os;print(os.getpid()) in settings and in views, is it the same pid? (I suspect you're seeing the log for the main process which makes autoreload and not on the secondary process which actually executes your code).

  2. Is the setting to attach to subprocesses turned on in PyDev? (i.e.: preferences > pydev > debug > attach to subprocess automatically while debugging).

  3. If you make a launch with --noreload, does it work?

Fabio Zadrozny
  • 24,814
  • 4
  • 66
  • 78
  • I'll check ASAP on the processes (which won't be until Monday or Tuesday). In the mean time (my money is on two processes) just to clarify, is the --noreload option a command line argument to runserver in the debug configuration? – Bernd Wechner Dec 27 '19 at 20:16
  • 1) I do have two different processes and each one loads the settings file but the breakpoint only stops in the first one. But I can note that there always has been to best of my knowledge as for years I've wondered why prints in my settings file appear twice on every debug run. But it has always worked with two processes in past. 2) I have that option set (i.e. to attach to subprocesses). Indeed. Always have and still do. 3) If I add --noreload option to runserver on my debug configuration I note I have only one process and breakpoints in my views work once again. – Bernd Wechner Dec 30 '19 at 11:45
  • What puzzles me is that when I use `runserver --noreload` in the debug configuration, an edit to a file does still cause a reload it seems. At least it goes live, perhaps that is not what reload means? A number of questions arise from this: 1) Why has the option to attach to subprocess stopped working in this new environment? 2) Why does a debug run produces two processes at all? 3) why, when `--noreload` is used do changes to .py view still take effect? – Bernd Wechner Dec 30 '19 at 11:50
  • Django has an autoreload that will spawn a different process and on file changes it'll kill the process and restart it, but the pydev debugger also has a reload builtin which will do a live reload (which is what you're seeing -- it's not 100% reliable, but covers many cases). I'm still not sure why it's not tracing though... can you create that same tracing profile but adding the process id to the files to know what may be happening in the subprocess? – Fabio Zadrozny Dec 31 '19 at 13:38
  • I.e.: in `with open('pydev-trace.txt', 'a') as f:` do: `with open('pydev-trace_%s.txt' % (os.getpid(),), 'a') as f:` – Fabio Zadrozny Dec 31 '19 at 13:44
  • I tried that just now. Interesting to note that I have that code in my views.py. When I then run a debug session I get only one txt file output with second oft he two PIDs reported on console and that file grew to 51GB before (probably) bringing my system down (it was locked when I came home and I left it running ;-), – Bernd Wechner Jan 01 '20 at 08:08
  • More instrumentation later I can see that without `--noreload`: Eclipse starts `pydev.py` which is given `manage.py runserver` as an argument and this loads the settings file but not my `views.py`. This then runs a subprocess which is just `manage.py runserver` and this subprocess loads both my settings file AND my `views.py`. I have the code to set the trace function in `views.py` which explains why I see a trace output file only from the subprocess. Inspecting that 51GB file I see it is the familiar endless loop of pathlib calls I describe in the question. – Bernd Wechner Jan 01 '20 at 08:58
  • If I move the `trace_func()` definition and call to `settrace()` into my settings file I get a warning from PyDev: PYDEV DEBUGGER WARNING: sys.settrace() should not be used when the debugger is being used. This may cause the debugger to stop working correctly. If this is needed, please check: http://pydev.blogspot.com/2007/06/why-cant-pydev-debugger-work-with.html to see how to restore the debug tracing back correctly. – Bernd Wechner Jan 01 '20 at 09:05
  • That said it works and produces a trace file for both processes if I do it. The parent process file grows to a fixed 343 Mb but the child process file grows and grows and grows seemingly endlessly in this apparent pathlib recursion. The parent process trace file seems to end (unsurprisingly) in subprocess.py ... (where I presume it's started a subprocess (the child) and is waiting on it ... – Bernd Wechner Jan 01 '20 at 10:26
  • The real shift from my old Eclipse/Python/Django stack to the new one is that the debugger isn't attaching to the subprocess which quite possibly relates to this apparent pathlib recursion. To better understand it, rather than a trace a profile might be useful. The trace is so huge and cyclic anyhow. – Bernd Wechner Jan 01 '20 at 10:28
  • Fabio, you're a champ. The latest PyDev 7.5.0 fixed this issue from which (combined with your release notes) leads me to suspect the issue was a Python 3.8 one! – Bernd Wechner Jan 10 '20 at 22:13
0

PyDev release 7.5.0 which appeared on Jan 10 2020, fixed this issue. Breakpoints in my views function properly once more. The release notes I have include:

  • Fixed support for Python 3.8 (which wasn't properly added to 7.4.0).
  • Debugger improvements (updated to pydevd 1.9.0).
    • Improved stepping into coroutines
    • Attach to process no longer needs to match the bitness of the target program with the interpreter.
    • File with a relative path is searched in sys.path folders (i.e.: so, cython builds can find the source).

Leading me to suspect that the whole issue related to Python 3.8 (which would be far from surprising given the very recent release of Python 3.8) though it's possible tweaks in pydevd 1.9.0 played a role. Either way a PyDev update to 7.5.0 fixes the issue!

Bernd Wechner
  • 1,854
  • 1
  • 15
  • 32