Cython setup.py compilation error in DLL wrapping tutorial: unresolved external symbol

Question

Backstory: For one of my projects that requires heavy computations, I am currently calling from my Python code an .exe coming from a C++ code, which read/write files on disk, and is thus super slow.

To solve this problem, I have recoded the C++ code to be used as a DLL in my Python code, since then there won't be disk access and it should be faster. To do this I plan to use Cython.

Problem: To understand better how it works, I am therefore following this tutorial, by J.J. Hakala who actually posts often here about Cython.

I followed every part, and I have in the same directory on Windows:

the DLL file that I compiled with MS Visual Studio 2015 Express: complexFunLib.dll
the .lib file that comes with the DLL: complexFunLib.lib
the header file for the C++ DLL source: complexFunLib.h
the Cython "header" file: ccomplexFunLib.pxd (yes there is an extra 'c')
the Cython "wrapper" file: complexFunLib.pyx
the Python setup file for creating complexFunLib.pyd with Cython: setup.py

When I tried to run python setup.py --build_ext --inplace first it couldn't find the library so I had to use that trick from another SO question to tell where was the DLL, so in setup.py I modified: libraries=[r'.\complexFunLib'] then that error disappeared.

Which brings me to my current unresolved error:

When I run python setup.py --build_ext --inplace it uses MinGW and raises this error:

EDIT, I PUT THE FULL LOG:

running build_ext

cythoning complexFunLib.pyx to complexFunLib.cpp

building 'complexFunLib' extension

creating build

creating build\temp.win-amd64-3.6

creating build\temp.win-amd64-3.6\Release

D:\MYNAME\AppData\Local\Continuum\Anaconda3\Library\mingw-w64\bin\gcc.exe -mdll -O -Wall -DMS_WIN64 -ID:\MYNAME\AppData\Local\Continuum\Anaconda3\lib\site-packages\numpy\core\include -ID:\MYNAME\AppData\Local\Continuum\Anaconda3\include-ID:\MYNAME\AppData\Local\Continuum\Anaconda3\include -c complexFunLib.cpp -o build\temp.win-amd64-3.6\Release\complexfunlib.o In file included from D:\MYNAME\AppData\Local\Continuum\Anaconda3\lib\site-packages\numpy\core\include/numpy/ndarraytypes.h:1809:0, from D:\MYNAME\AppData\Local\Continuum\Anaconda3\lib\site-packages\numpy\core\include/numpy/ndarrayobject.h:18, from D:\MYNAME\AppData\Local\Continuum\Anaconda3\lib\site-packages\numpy\core\include/numpy/arrayobject.h:4, from complexFunLib.cpp:500:

D:\MYNAME\AppData\Local\Continuum\Anaconda3\lib\site-packages\numpy\core\include/numpy/npy_1_7_deprecated_api.h:13:79: note: #pragma message:

D:\MYNAME\AppData\Local\Continuum\Anaconda3\lib\site-packages\numpy\core\include/numpy/npy_1_7_deprecated_api.h(12) : Warning Msg: Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION

"#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION")
                                                              ^
complexFunLib.cpp: In function 'PyTypeObject* __Pyx_ImportType(const char*, const char*, size_t, int)':

complexFunLib.cpp:7149:53: warning: unknown conversion type character 'z' in format [-Wformat=]
        module_name, class_name, basicsize, size);

                                                ^
complexFunLib.cpp:7149:53: warning: unknown conversion type character 'z' in format [-Wformat=]

complexFunLib.cpp:7149:53: warning: too many arguments for format [-Wformat-extra-args]

writing build\temp.win-amd64-3.6\Release\complexFunLib.cp36-win_amd64.def

D:\MYNAME\AppData\Local\Continuum\Anaconda3\Library\mingw-w64\bin\g++.exe -shared -s build\temp.win-amd64-3.6\Release\complexfunlib.o build\temp.win-amd64-3.6\Release\complexFunLib.cp36-win_amd64.def -L. -LD:\MYNAME\AppData\Local\Continuum\Anaconda3\libs -LD:\MYNAME\AppData\Local\Continuum\Anaconda3\PCbuild\amd64 .\complexFunLib.dll -lpython36 -lmsvcr140 -o "D:\MYNAME\PATH_TO_PROJECT\complexFunLib.cp36-win_amd64.pyd"

build\temp.win-amd64-3.6\Release\complexfunlib.o:complexFunLib.cpp:(.text+0x29b5): undefined reference to `mp_mlt_exp_c4'

build\temp.win-amd64-3.6\Release\complexfunlib.o:complexFunLib.cpp:(.text+0x3ba7): undefined reference to `mp_mlt_exp_c8'

collect2.exe: error: ld returned 1 exit status error: command 'D:\MYNAME\AppData\Local\Continuum\Anaconda3\Library\mingw-w64\bin\g++.exe' failed with exit status 1

And when I run (to try just in case) python setup.py --build_ext --inplace --compiler=msvc since I don't know if it should be compiled also with the same compiler that compiled the DLL, it uses Visual Studio's compiler and raises this error:

EDIT, I PUT THE FULL LOG FOR MSVC:

running build_ext

cythoning complexFunLib.pyx to complexFunLib.cpp

building 'complexFunLib' extension

creating build

creating build\temp.win-amd64-3.6

creating build\temp.win-amd64-3.6\Release

C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -ID:\MYNAME\AppData\Local\Continuum\Anaconda3\lib\site-packages\numpy\core\include -ID:\MYNAME\AppData\Local\Continuum\Anaconda3\include -ID:\MYNAME\AppData\Local\Continuum\Anaconda3\include "-IC:\ProgramFiles (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\8.1\include\shared" "-IC:\Program Files (x86)\Windows Kits\8.1\include\um" "-IC:\Program Files (x86)\Windows Kits\8.1\include\winrt" /EHsc /TpcomplexFunLib.cpp /Fobuild\temp.win-amd64-3.6\Release\complexFunLib.obj

complexFunLib.cpp

d:\MYNAME\appdata\local\continuum\anaconda3\lib\site-packages\numpy\core\include\numpy\npy_1_7_deprecated_api.h(12) : Warning Msg: Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION

C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:. /LIBPATH:D:\MYNAME\AppData\Local\Continuum\Anaconda3\libs /LIBPATH:D:\MYNAME\AppData\Local\Continuum\Anaconda3\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\LIB\amd64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.10240.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\8.1\lib\winv6.3\um\x64" .\complexFunLib.lib /EXPORT:PyInit_complexFunLib build\temp.win-amd64-3.6\Release\complexFunLib.obj "/OUT:D:\MYNAME\PATH_TO_PROJECT\complexFunLib.cp36-win_amd64.pyd" /IMPLIB:build\temp.win-amd64-3.6\Release\complexFunLib.cp36-win_amd64.lib complexFunLib.obj : warning LNK4197: export 'PyInit_complexFunLib' specified multiple times; using first specification

Creating library build\temp.win-amd64-3.6\Release\complexFunLib.cp36-win_amd64.lib and object build\temp.win-amd64-3.6\Release\complexFunLib.cp36-win_amd64.exp

complexFunLib.obj : error LNK2001: unresolved external symbol mp_mlt_exp_c8

complexFunLib.obj : error LNK2001: unresolved external symbol mp_mlt_exp_c4

D:\PATH_TO_PROJECT\complexFunLib.cp36-win_amd64.pyd : fatal error LNK1120: 2 unresolved externals

error: command 'C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\link.exe' failed with exit status 1120

These two errors are, I believe, the same, and the 'undefined references' or 'external symbols' mp_mlt_exp_c4 and mp_mlt_exp_c8 are functions declared and defined in the C++ header and .cpp, and also in the .pxd and .pyx, as can be seen from the tutorial page I linked at the beginning.

I'm really unsure how to resolve this, it looks like a linking problem with Cython machinery, so if anyone knowledgeable in that area has a solution, I'd be glad to hear it :).

EDIT:

The dumpbin asked by @ead:

Dump of file complexFunLib.dll

File Type: DLL

Section contains the following exports for complexFunLib.dll

00000000 characteristics
5B1800BA time date stamp Wed Jun  6 17:41:46 2018
    0.00 version
       1 ordinal base
       2 number of functions
       2 number of names

ordinal hint RVA      name

      1    0 00001000 mp_mlt_exp_c4 = _mp_mlt_exp_c4
      2    1 000010E0 mp_mlt_exp_c8 = _mp_mlt_exp_c8

Summary

    1000 .data
    1000 .gfids
    1000 .rdata
    1000 .reloc
    1000 .rsrc
    1000 .text

Does the `setup.py` output show the actual `g++`/`link` command lines being run? If so, can you add that here? If not… can you find in the docs how to get it to dump more build output for debugging, or do you need us to find that for you? — abarnert, Jun 05 '18 at 16:59
As far as I know you also need *.lib next to you *.dll on windows. From your description and error my guess would be that it is missing — ead, Jun 05 '18 at 17:46
I had tried by adding the *.lib and it changed nothing. As for the full output with the g++ link, it is there, but I'm not in front of that computer right now, so it'll come in about 13 hours, thanks for the patience. — viiv, Jun 05 '18 at 18:37
I added the full log for the compilqtion with Visual Studio. Also, there was indeed an extra line when I didn't add the .lib: `warning: no library file corresponding to '.\complexFunLib' found (skipping)' — viiv, Jun 06 '18 at 07:33
I added the log for the compilation using MinGW too, just in case. I still cant figure out what to do of it. — viiv, Jun 06 '18 at 11:53
@abarnert when you said 'show the actual g++/link command lines being run', is what I posted enough, or is not what you meant? If it isn't I'd be glad to know the actual commands to show more output :) — viiv, Jun 06 '18 at 15:39
That's perfect; it seems like it should be enough for someone who knows Cython building on Windows to either solve this, or tell you what to do next to debug it. I'm not sure I know enough to do that, but I'll give it a look and see if I can help. — abarnert, Jun 06 '18 at 17:26
It looks like, at least in the MSVC build, it seems to be getting the symbols from `complexFunLib.pyx` twice, and not getting the symbols from `complexFunLib.dll`. I'm not sure why. It's definitely passing the `.lib` file explicitly, and the Cython code is definitely getting a different libname `complexFunLib.cp36-win_amd64` as it should, but still, it looks weird. Can you try renaming either the C++ project or the Cython project so they don't have the same base name? — abarnert, Jun 06 '18 at 17:32
If that solves the build, it isn't really an answer to the real problem, but it might make it easier to debug it, and meanwhile it gives you a workaround so you can make progress in parallel. And if it doesn't solve the build (which I think is more likely), at least we've eliminated the possibility, and it may even be easier to spot what's wrong when the names are more visibly different. — abarnert, Jun 06 '18 at 17:33
`export 'PyInit_complexFunLib' specified multiple times` is because this symbol is defined as export at the command line and also via `_declspec(dllexport)` in code - so that is not the problem here. — ead, Jun 07 '18 at 08:39
Initially my C++ Project had the generic name 'ConsoleApplication2', and I got the same errors as I get now (I renamed it 'complexFunLib' now, and recompiled the DLL). So I don't think that is the cause of the problem. @ead how do I verify that the DLL actually exports these symbols? — viiv, Jun 07 '18 at 09:11
@ead I added the dumpbin of the DLL in the first post. It seems to me that the functions are exported well, but I'm no expert, since I didn't know this tool — viiv, Jun 07 '18 at 09:28
I build very seldom on windows, so I don't know whether `./XXX.lib` works. What I know should work: change your setup.py back to `library_dirs=['']\n libraries=['complexFunLib']`. It should find the library and build. If not, find out why it doesn't - don't just change to another version which doesn't work. — ead, Jun 07 '18 at 09:44
I did that and it still produces the exact same error unfortunately. When you say 'don't just change to another version which doesn't work', you mean I have to find why it doesn't find the .lib ? I edited the main post to add that the .lib is in the same directory now — viiv, Jun 07 '18 at 09:54
The same original error "couldn't find the library" or the current error? If it the original error you have to resolve it. If it is the current error - you have to link in verbose mode, too see what realy going on. — ead, Jun 07 '18 at 10:49
How would I enable this verbose mode? Because I would like to be sure it is indeed not a PATH issue, as I indeed get the same error still. — viiv, Jun 07 '18 at 16:06
but how would I pass this to my current command that is `python setup.py build_ext --inplace --compiler=msvc' ? — viiv, Jun 07 '18 at 16:27
I ask because if I add /VERBOSE to the command I cited above, it doesn't work — viiv, Jun 08 '18 at 07:52

Cython setup.py compilation error in DLL wrapping tutorial: unresolved external symbol

0 Answers0

Linked