4

If I build CPython from source on Windows I encounter problems when I want to pip install a package that contains a C-Extension. It seems like the error happens while linking the libraries.

For example when installing cython (but it also crashes with the same error on other C extension packages):

LINK : fatal error LNK1104: cannot open file 'python38.lib'

error: command 'C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.23.28105\bin\HostX86\x86\link.exe' failed with exit status 1104

The reason why it cannot open the "python38.lib" is because the ".lib" file in debug mode is called "python38_d.lib".

A minimal reproducible example would be (on the command-line) based on the Quick Reference of the CPython developer guide:

git clone --branch v3.8.0 https://github.com/python/cpython.git
cd cpython
git checkout v3.8.0
.\PCbuild\build.bat -e -d
.\PCbuild\win32\python_d.exe -m ensurepip
.\PCbuild\win32\python_d.exe -m pip install pip --upgrade -vv
.\PCbuild\win32\python_d.exe -m pip install setuptools --upgrade -vv
.\PCbuild\win32\python_d.exe -m pip install cython -vv

The resulting distutils.sysconfig.get_config_vars() is:

{'BINDIR': '...\\cpython\\PCbuild\\win32',
 'BINLIBDEST': ...\\cpython\\Lib',
 'EXE': '.exe',
 'EXT_SUFFIX': '_d.cp38-win32.pyd',
 'INCLUDEPY': '...\\cpython\\include;...\\cpython\\PC',
 'LIBDEST': '...\\cpython\\Lib',
 'SO': '_d.cp38-win32.pyd',
 'VERSION': '38',
 'exec_prefix': '...\\cpython',
 'prefix': '...\\cpython',
 'srcdir': '...\\cpython'}

Is there something I'm missing? Is building C-Extensions on Python-debug builds on Windows simply not supported? If it is supported: how would I do it?

Community
  • 1
  • 1
MSeifert
  • 145,886
  • 38
  • 333
  • 352
  • Distutils gets the information from config. Not sure how it looks like for your setup. Here is an example how one could overrule settings in a setup file https://stackoverflow.com/a/57057959/5769463 – ead Dec 03 '19 at 06:12
  • @ead Do you mean the CPython configs or the config from the extension module I'm trying to install? Also shouldn't the CPython build options be picked up by default (and one of the modules I wanted to install contained basically no customizations in the setup) - then why doesn't it realize it has to link against `python38_d.lib`? I tested the commands from the question on my computer and on CI and all of them fail. If you have a workaround or suggestion how to fix this I would appreciate it :) – MSeifert Dec 03 '19 at 07:11
  • 1
    I mean distutils.config. You can inspect the provided values and see what is wrong. Sadly for next days I won’t be able to look into this problem – ead Dec 03 '19 at 07:46
  • @ead Thanks for the clarification. I included the values in the question. – MSeifert Dec 03 '19 at 11:10

2 Answers2

6

This code is a little bit hacky, but works for me on MSVC19, allowing to debug application without building debug python libraries.

#ifdef _DEBUG
#define _DEBUG_WAS_DEFINED
#undef _DEBUG
#endif

#include "Python.h"

#ifdef _DEBUG_WAS_DEFINED
#define _DEBUG
#undef _DEBUG_WAS_DEFINED
#endif
Fairyteller
  • 312
  • 1
  • 10
4

Linking against pythonXY.lib is a little bit sneaky on Windows. When you look at the command line for linking, you will see that no python-library is passed to the linker, i.e. 'link.exe`. Note: This is also the case for Linux, but on Linux one doesn't have to because the needed symbols will be provided by the python-executable.

However, it is easy to check via dumpbin /dependents resulting.pyd, that there is a dependency on pythonXY.dll, also adding extra_link_args = ["/VERBOSE:LIB"] to extension-definition and triggering verbose-mode of the linker will show that the linker uses pythonXY.lib.

The sneaky part: Microsoft Compler has a convinience-pragma #pragma comment(lib, ...) to automatically trigger linking of a library, which is also used in Python-headers:

#               if defined(_MSC_VER)
                        /* So MSVC users need not specify the .lib
                        file in their Makefile (other compilers are
                        generally taken care of by distutils.) */
#                       if defined(_DEBUG)
#                               pragma comment(lib,"python39_d.lib")
#                       elif defined(Py_LIMITED_API)
#                               pragma comment(lib,"python3.lib")
#                       else
#                               pragma comment(lib,"python39.lib")
#                       endif /* _DEBUG */
#               endif /* _MSC_VER */

As you can see, to link against the debug version, one needs to define _DEBUG.

_DEBUG is automatically defined by distutils on Windows, if build_ext is called with options --debug, e.g.

python setup.py build_ext -i --debug

That can be translated to pip as 

pip install --global-option build --global-option --debug XXXXX

which can be interpreted roughly as: trigger build command (which also includes build_ext-command) with option --debug prior to installing.

Another subtility when building debug C-extensions, there is more to it on Windows:

#ifdef _DEBUG
#       define Py_DEBUG
#endif

Having defined Py_DEBUG macro meant incompartible ABIs until Python3.8, because it also assumed Py_TRACE_REFS which leads to different memory layout of PyObject and some additional functionality missing in the release-mode.

However, since Python3.8, one probably can get away with it by providing the missing pythonXY_d.lib/pythonYX.lib as a symlink linking to another version.

ead
  • 32,758
  • 6
  • 90
  • 153
  • Do you know what parts of your answer could be the reason for the downvote? It worked for my use-case and the rest of the answer made sense to me - so I wonder what problems/downsides there could be. – MSeifert Dec 10 '19 at 15:52
  • @MSeifert I don’t know, but would be thankful if somebody would point out the problem. – ead Dec 10 '19 at 17:43