5

I would like to debug some of the basic packages that come with the Python install and/or are built-in packages, including pip and venv.

The desire comes from an error message of file permissions (unable to access a file with an "unprintable file name") some of my team is getting running these commands - see this question for details.

Question

How do you debug the Python source code when trying to catch issues in the main python executable, or when directly running a base python module (see following examples for pip and venv)?

$ python -m pip install --upgrade
$ python -m venv .venv

If it matters, my environment is VSCode, where I am happily able to engage the debugger on any custom script I have written, using the built-in debugger that interacts (I assume) with the main Microsoft Python extension.

LightCC
  • 9,804
  • 5
  • 52
  • 92
  • 1
    `pip` is a pre-built binary, so if you want to debug that it's most likely going to involve modifying its [source code](https://github.com/pypa/pip) directly and rebuilding the executable. For `venv`, the source appears to be located in the `__init__.py` file under `\lib\venv`. In there, you could add `import pdb; pdb.set_trace()` to start the Python DeBugger (or just `breakpoint()`, which should work in VSCode). – b_c Jan 20 '20 at 19:17
  • 2
    @b_c: The `pip` 'executable' is just a Python script that uses the [`pkg_resources` library](https://setuptools.readthedocs.io/en/latest/pkg_resources.html) to (indirectly) import the `pip._internal.cli.main` module and runs the `main()` function in it. Open the path that `which pip` points to and you'll see it's just text you can add a debug statement to. – Martijn Pieters Jan 20 '20 at 19:18
  • @MartijnPieters Good to know! I didn't see a pip module when I looked (wasn't expecting it to be in `site-packages`) and I didn't consider `python -m pip` would suggest that it _does_ exist as a module somewhere. – b_c Jan 20 '20 at 20:21

2 Answers2

7

You will need to set "justMyCode": false in your launch.json for the debugger to trace into third-party code.

Brett Cannon
  • 14,438
  • 3
  • 45
  • 40
6

Start by looking at the source code for those modules; the -m switch looks for a package or module to import first. If it's a package, then Python imports the __main__ module in that package and runs it as the main script. If it is a module, the module itself is imported and run as __main__.

Usually the code is structured such that a function is called you can import directly too. You can then just write a bit of code that imports the same function and calls it the same way the __main__ module would. From there on out it is trivial to run this under a debugger.

E.g. pip is a package, so python -m pip will import pip.__main__ and run that as a script. This then triggers:

from pip._internal.cli.main import main as _main  # isort:skip # noqa

if __name__ == '__main__':
    sys.exit(_main())

to be run. You can do the same in VSCode; import pip._internal.cli.main.main and call it.

You can find the source code for these modules by just importing them and printing out the resulting object:

python -c "import pip; print(pip)"

The representation of a module, if loaded from disk, will include it's filename. If the filename ends in /__init__.py it's a package, so you can also double-check that the __main__.py file exists:

python -c "import pip.__main_; print(pip.__main__)"

You can do the same for the venv module. This one is part of the Python standard library, so the documentation actually links directly to the source code, and the venv.__main__ module just imports venv.main() and calls it.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • Thanks! So find and download the source code - do I also need the "debug binaries" from the main Python installation? I recall checkboxes for those during the original Python installs (on Windows at least). – LightCC Jan 20 '20 at 19:18
  • @LightCC: `python -c "import pip; print(pip)"` will give you the path to the source code on your harddisk. – Martijn Pieters Jan 20 '20 at 19:19
  • 1
    @LightCC: you don't need the Python debug build, no. Nothing here is a binary extension, it's all just Python code. – Martijn Pieters Jan 20 '20 at 19:19