How to compile init.py file using cython on Windows

Question

When I compile an arbitrary __init__.py file on Windows with setup.py build_ext --inplace command, it has an unresolvable external symbol error (i.e. "LINK : error LNK2001: An unresolvable external symbol PyInit___init__").

The local environment:

python3.7,
Cython 0.29.14,
window10 x64,
Microsoft Visual Studio 2017,

ctest/__init__.py

# cython: language_level=3
print('__init__')

setup.py

from distutils.core import setup
from Cython.Build import cythonize


def compile_code(name, filename):
    setup(
        name=name,
        ext_modules=cythonize(filename),
    )


if __name__ == '__main__':
    compile_code('a', 'ctest/__init__.py')

The information printed by the terminal:

Compiling ctest/__init__.py because it changed.
[1/1] Cythonizing ctest/__init__.py
running build_ext
building 'ctest.__init__' extension
creating build
creating build\temp.win-amd64-3.7
creating build\temp.win-amd64-3.7\Release
creating build\temp.win-amd64-3.7\Release\ctest
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Id:\py37\include -Id:\py37\incl
ude "-IC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Pro
gram Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\includ
e\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" /Tcctest/__init__
.c /Fobuild\temp.win-amd64-3.7\Release\ctest/__init__.obj
__init__.c
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTU
AC:NO /LIBPATH:d:\py37\Libs /LIBPATH:D:\ENVS\cpytrantest\libs /LIBPATH:D:\ENVS\cpytrantest\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC
\Tools\MSVC\14.16.27023\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x6
4" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" /EXPORT:PyInit___init__ build\temp.win-amd64-3.7\Release\ctest/__init__.obj /OUT:C:\Users\76923\Deskto
p\cpythonrecord\ctest\__init__.cp37-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.7\Release\ctest\__init__.cp37-win_amd64.lib
LINK : error LNK2001: An unresolvable external symbol PyInit___init__
build\temp.win-amd64-3.7\Release\ctest\__init__.cp37-win_amd64.lib : fatal error LNK1120: An external command that cannot be parsed
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community\\VC\\Tools\\MSVC\\14.16.27023\\bin\\HostX86\\x64\\link.exe' failed with exit status 1120

I would rather not cythonize/compile __init__.py. It seems to work on Linux but there are problems on Windows. — ead, Nov 11 '19 at 09:49
@ead Thank you for your reply. I executed the same command on Linux and it worked, much as I wish, but I need to run it on the window now... — pppig, Nov 11 '19 at 10:09
That might be interesting: https://stackoverflow.com/a/32067984/5769463 implies that it kind of accidentally works - but should one rely on it? However, `__init__.py` is not really a module, so why not leave it uncompiled? What is the benefit of compiling it in the first place? — ead, Nov 11 '19 at 10:34
I recall seeing that you can make it work on Windows if you change symbol visibility (or something like that, I can't remember the details). It does seem mostly pointless though — DavidW, Nov 11 '19 at 12:08
@ead Thanks again.How could that not be a good solution.There's just a little bit of code in the \__init__.py file, it's fine not to compile it.Maybe it's because it worked on Linux, but it's a problem on Windows, so I'm looking for a solution about it. — pppig, Nov 11 '19 at 12:27
@DavidW Thank you for your reply. You're right. It is only because the init file did not compile successfully before (there is no \__init__.py file in the compiled folder) that the program failed to run, so I started to look for methods.Of course I can copy the \__init__.py file over here. — pppig, Nov 11 '19 at 12:48

ead · Accepted Answer · 2020-08-12T07:57:04.630

Maybe this behaviour might be viewed as a small bug in distutils-package (as pointed out by @DavidW there is this open issue: https://bugs.python.org/issue35893). However, it also shows, that cythonizing/compiling __init__.py isn't very popular and uses some undocumented implementation details which might change in the future, so it could be wiser to refrain from meddling with __init__.py.

But if you must...

When a package is imported explicitly, e.g.

import ctest

or implicitly, e.g.

import ctest.something

The FileFinder will see that a package, and not a module, is imported and will try to load ctest/__init__.py instead of ctest.py (which most likely doesn't exists):

    # Check if the module is the name of a directory (and thus a package).
    if cache_module in cache:
        base_path = _path_join(self.path, tail_module)
        for suffix, loader_class in self._loaders:
            init_filename = '__init__' + suffix
            full_path = _path_join(base_path, init_filename)
            if _path_isfile(full_path):
                return self._get_spec(loader_class, fullname, full_path, [base_path], target)

Used suffix, loader_class are for loading __init__.so, __init__.py and __init__.pyc in this order (see also this SO-post). This means, __init__.so will be loaded instead of __init__.py if we manage to create one.

While __init__.py is executed, The property __name__ is the name of the package, i.e. ctest in your case, and not __init__ as one might think. Thus, the name of the init-function, Python-interpreter will call when loading the extension __init__.so is PyInit_ctest in your case (and not PyInit___init__ as one might think).

The above explains, why it all works on Linux out-of-the-box. What about Windows?

The loader can only use symbols from a so/dll which aren't hidden. Per default all symbols built with gcc are visible, but not for VisualStudio on Windows - where all symbols are hidden per default (see e.g. this SO-post).

However, the init-function of a C-extension must be visible (and only the init-function) so it can be called with help of the loader - the solution is to export this symbol (i.e. PyInit_ctest) while linking, in your case it is the wrong /EXPORT:PyInit___init__-option for the linker.

The problem can be found in distutils, or more precise in build_ext-class:

def get_export_symbols(self, ext):
    """Return the list of symbols that a shared extension has to
    export.  This either uses 'ext.export_symbols' or, if it's not
    provided, "PyInit_" + module_name.  Only relevant on Windows, where
    the .pyd file (DLL) must export the module "PyInit_" function.
    """
    initfunc_name = "PyInit_" + ext.name.split('.')[-1]
    if initfunc_name not in ext.export_symbols:
        ext.export_symbols.append(initfunc_name)
    return ext.export_symbols

Here, sadly ext.name has __init__ in it.

From here, one possible solution is easy : to override get_export_symbols, i.e. to add the following to your setup.py-file (read on for a even simpler version):

...
from distutils.command.build_ext import build_ext
def get_export_symbols_fixed(self, ext):
    names = ext.name.split('.')
    if names[-1] != "__init__":
        initfunc_name = "PyInit_" + names[-1]
    else:
        # take name of the package if it is an __init__-file
        initfunc_name = "PyInit_" + names[-2]
    if initfunc_name not in ext.export_symbols:
        ext.export_symbols.append(initfunc_name)
    return ext.export_symbols

# replace wrong version with the fixed:
build_ext.get_export_symbols = get_export_symbols_fixed
...

Calling python setup.py build_ext -i should be enough now (because __init__.so will be loaded rather than __init__.py).

However, as @DawidW has pointed out, Cython uses macro PyMODINIT_FUNC, which is defined as

#define PyMODINIT_FUNC Py_EXPORTED_SYMBOL PyObject*

with Py_EXPORTED_SYMBOL being marked as visible/exported on Windows:

#define Py_EXPORTED_SYMBOL __declspec(dllexport)

Thus, there is no need to mark the symbol as visible at the command line. Even worse, this is the reason for the warning LNK4197:

__init__.obj : warning LNK4197: export 'PyInit_ctest' specified multiple times; using first specification

as PyInit_test is marked as __declspec(dllexport) and exported via option /EXPORT: at the same time.

/EXPORT:-option will be skipped by distutils, if export_symbols is empty, we can use even a simpler version of command.build_ext:

...
from distutils.command.build_ext import build_ext
def get_export_symbols_fixed(self, ext):
    pass  # return [] also does the job!

# replace wrong version with the fixed:
build_ext.get_export_symbols = get_export_symbols_fixed
...

This is even better than the first version, as it also fixes warning LNK4197!

@ead It seems like this should be taken care of by [`PyMODINIT_FUNC`](https://github.com/python/cpython/blob/0b60f64e4343913b4931dc27379d9808e5b78fe1/Include/pyport.h#L670). Would a simpler version that didn't add the `initfunc_name` also work? — DavidW, Nov 14 '19 at 08:53
@DavidW I think you are right. I have no possibility to check that right now, but if I remember right, this is the reason one gets warning LNK4197 from the linker on Windows: the symbol `PyInit_XXX` is once declared as `__declspec(dllexport)` and once via `/EXPORT:`. One of them would be enough. — ead, Nov 14 '19 at 11:09
@ead I guess it's probably there as a backup for people writing there own C API modules who omit PyMODINIT_FUNC, but anyway... doesn't really matter. — DavidW, Nov 14 '19 at 11:29
Here's a link to the bug report for this https://bugs.python.org/issue35893 — DavidW, Jul 31 '20 at 21:02

score 1 · Answer 2 · answered Nov 11 '19 at 15:03

This is a very tentative answer because I have no easy way of testing it on Windows, so if it's wrong then let me know and I'll delete it.

Can you try running (on the commend line):

C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:d:\py37\Libs /LIBPATH:D:\ENVS\cpytrantest\libs /LIBPATH:D:\ENVS\cpytrantest\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" /EXPORT:PyInit_ctest build\temp.win-amd64-3.7\Release\ctest/__init__.obj /OUT:C:\Users\76923\Desktop\cpythonrecord\ctest\__init__.cp37-win_amd64.pyd /IMPLIB:build\temp.win-md64-3.7\Release\ctest\__init__.cp37-win_amd64.lib

All I've done is taken the compilation command that distutils generated and replaced /EXPORT:PyInit___init__ with /EXPORT:PyInit_ctest. The /EXPORT is a Windows specific compiler option that doesn't get added on Linux. It looks like either distutils or Cython passes the name PyInit___init__ to MSVC, but if I look in the actual generated C file then the name appears to be PyInit_ctest, hence the undefined symbol.

If that workaround (doing the compilation independently of distutils) works then you should report the bug to either the distutils or the Cython bug tracker (probably Cython) with these details and hopefully it can be fixed.

It's quite possible that this answer will only get you as far as the issue in your linked question, in which case I don't know if it's worth keeping — DavidW, Nov 11 '19 at 15:13
I don't know when it should execute, when I execute it directly at the terminal or after I execute `setup.py build_ext --inplace` command(It will generate `build\temp.win-amd64-3.7\Release\ctest\__init__.obj`,`__init__.c`and Invalid file `ctest\__init__.cp37-win_amd64.pyd`), it prints `LINK : fatal error LNK1181: Unable to open input file “build\temp.win-amd64-3.7\Release\ctest\\__init__.obj”`, right — pppig, Nov 12 '19 at 03:07
@pppig I'm afraid I don't know then. I don't have Python and MSVC installed on Windows so I can't easily test anything, and this answer was a bit of a guess. My feeling is that it is the right direction to go, but I can't help any further. What I'll do is convert the answer to "community wiki" so it's easier for other people to edit it and make suggestions - if you're lucky someone else will have some ideas. — DavidW, Nov 12 '19 at 11:08

How to compile init.py file using cython on Windows

2 Answers2

Linked

Related

How to compile __init__.py file using cython on Windows

2 Answers2

Linked

Related

How to compile init.py file using cython on Windows