Error exporting symbol when building Python C Extension in Windows

Question

I'm working on porting a Python module to Windows. I have a toy example as follows.

The folder structure is:

foo/
  libfoo/
    foo.c
  setup.py

setup.py

from setuptools import setup, Extension

sources = ['libfoo/foo.c']

foo = Extension('libfoo',
                 sources = sources,
                 define_macros = None,
                 include_dirs = ['./libfoo'],
                 libraries = None,
                 library_dirs = None,
                 )

setup(name          = 'foo',
      ext_modules      = [foo],
      install_requires = ['setuptools'],
)

libfoo/foo.c (for completeness)

#include <stdio.h>

void foo() {
  printf("Hello World!");
}

When I attempt to install the package, I encounter an error.

C:\Users\user\foo>python setup.py install
running install
running bdist_egg
running egg_info
creating foo.egg-info
writing requirements to foo.egg-info\requires.txt
writing foo.egg-info\PKG-INFO
writing top-level names to foo.egg-info\top_level.txt
writing dependency_links to foo.egg-info\dependency_links.txt
writing manifest file 'foo.egg-info\SOURCES.txt'
reading manifest file 'foo.egg-info\SOURCES.txt'
writing manifest file 'foo.egg-info\SOURCES.txt'
installing library code to build\bdist.win32\egg
running install_lib
running build_ext
building 'libfoo' extension
creating build
creating build\temp.win32-2.7
creating build\temp.win32-2.7\Release
creating build\temp.win32-2.7\Release\libfoo
c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo /Ox
/MD /W3 /GS- /DNDEBUG -I./libfoo -IC:\Python27\include -IC:\Python27\PC /Tclibfo
o/foo.c /Fobuild\temp.win32-2.7\Release\libfoo/foo.obj
foo.c
creating build\lib.win32-2.7
c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL /nologo
/INCREMENTAL:NO /LIBPATH:C:\Python27\libs /LIBPATH:C:\Python27\PCbuild /EXPORT:i
nitlibfoo build\temp.win32-2.7\Release\libfoo/foo.obj /OUT:build\lib.win32-2.7\l
ibfoo.pyd /IMPLIB:build\temp.win32-2.7\Release\libfoo\libfoo.lib /MANIFESTFILE:b
uild\temp.win32-2.7\Release\libfoo\libfoo.pyd.manifest
LINK : error LNK2001: unresolved external symbol initlibfoo
build\temp.win32-2.7\Release\libfoo\libfoo.lib : fatal error LNK1120: 1 unresolv
ed externals
error: command 'c:\\Program Files (x86)\\Microsoft Visual Studio 9.0\\VC\\BIN\\l
ink.exe' failed with exit status 1120

It seems that the distutils package (in this case setuputils) will always export one symbol from a shared extension and that is "init + extension_name" [Link].

It is specified in the Windows Linker "EXPORT" option [Link] but it can't find the symbol.

Help?

EDIT: The C code does not use the Python C API i.e. "#include ". This is because the goal of the project is to take an existing C library and encase in a Python wrapper via a Python extension. The package works on Unix/Linux.

Are you actually trying to build an "libfoo" extension module that would require defining an `initlibfoo` function to be called by Python when importing the extension? Or are you just using setuptools to compile a shared library? If it's the latter, the first thing you need to do for a Windows DLL is define the exported symbols, using either [`__declspec(dllexport)`](https://msdn.microsoft.com/en-us/library/a90k134d%28v=vs.90%29) or a [.DEF file](https://msdn.microsoft.com/en-us/library/d91k01sh%28v=vs.90%29). — Eryk Sun, Jan 09 '16 at 03:55
@eryksun I believe it's the latter. The intention is to build the "libfoo" extension upon installing the package. When the interpreter imports the package "foo" i.e. "import foo", the class object will load the DLL (or .pyd?). Let me know if this doesn't make sense. — nbui, Jan 09 '16 at 04:00
A .pyd is just a DLL. It sounds like this is what you want, i.e. foo.pyd that's linked to and wraps functions in libfoo.dll. Start with getting libfoo.dll working on its own, just manually building it from the command line (without Python or setuptools). Then follow the documentation for [creating an extension module](https://docs.python.org/2/extending/index.html). — Eryk Sun, Jan 09 '16 at 04:08

nbui · Accepted Answer · 2016-01-13T21:46:04.193

After lots of playing around, I was able to understand the issue.

Solution: in Windows, you must define an initialization function for your extension because setuptools will build a .pyd file, not a DLL

To resolve this in Python 2, you define the initlibfoo() method in the source.

#include <Python.h>

PyMODINIT_FUNC initlibfoo(void) {
    // do stuff...
}

To resolve this in Python 3, you define the PyInit_libfoo() method in the source.

#include <Python.h>

PyMODINIT_FUNC PyInit_libfoo(void) {
    // do stuff...
}

so now, foo.c will look like:

#include <stdio.h>
#include <Python.h>

void foo() {
  printf("Hello World!");
}

PyMODINIT_FUNC initlibfoo(void) // Python 2.7
//PyMODINIT_FUNC PyInit_libfoo(void) // Python 3.5
{
    // do stuff...
}

Explanation

When compiling Python extensions for Windows, the Extension class will tell the linker to export a function for other programs to call. That can be seen in the terminal output:

c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL /nologo
/INCREMENTAL:NO /LIBPATH:C:\Python27\libs /LIBPATH:C:\Python27\PCbuild /EXPORT:i
nitlibfoo build\temp.win32-2.7\Release\libfoo/foo.obj /OUT:build\lib.win32-2.7\l
ibfoo.pyd /IMPLIB:build\temp.win32-2.7\Release\libfoo\libfoo.lib /MANIFESTFILE:b
uild\temp.win32-2.7\Release\libfoo\libfoo.pyd.manifest

(emphasis on /EXPORT:initlibfoo)

Now this is a required function for C/C++ Extensions so that they can be imported as modules. In other words, when you import libfoo Python will call the function initlibfoo on the .pyd as part of the initialization of the module. The error encountered here is due to the fact that the linker is told to export the function initlibfoo however it can't find the symbol in the C object files because it was never defined in the source!

Surely, one should be able to decline the option of export a symbol right? Apparently not. Referring back to the Extension class documentation, there's an argument export_symbols. I tried passing None to it and the linker option was still being used. It seems there's no way to circumvent this linker option.

NOT RECOMMENDED: if for some reason, needing to have Python.h bothers you, there's a way around this. You still need to define the "init" methods above but you can do:

void initlibfoo() {} //Python 2.7
void PyInit_libfoo() {} //Python 3.5

But now, you can't use the library via import libfoo. You could use the ctypes module and load it with ctypes.PyDLL('/path/to/pyd'). Essentially, you used Python to build a DLL for you, in which case, it does but it builds a special DLL known as a .pyd file. Then to use it, you must load it in via ctypes module.

score 2 · Answer 2 · answered Jun 24 '19 at 09:17

2

One other possible solution:

from distutils.command.build_ext import build_ext as _du_build_ext
from unittest.mock import Mock
mockobj = _du_build_ext
mockobj.get_export_symbols = Mock(return_value=None)

answered Jun 24 '19 at 09:17

Tomasz Hemperek

111
2
3

score 1 · Answer 3 · answered Dec 15 '21 at 04:07

1

Only need to write a function to convert the path to the corresponding format under different OS. (Windows is \\, Unix-like is /).

answered Dec 15 '21 at 04:07

sun1638650145

11
1

score 0 · Answer 4 · answered Nov 25 '20 at 17:38

I have setup a minimal working python package with ctypes extension here: https://github.com/himbeles/ctypes-example which works on Windows, Mac, Linux and where you do not need to include Python.h in C/C++

It takes the approach of memeplex and of Tomasz Hemperek of overwriting build_ext.get_export_symbols().
It forces the library extension to be the same (.so) for all operating systems.
Additionally, a compiler directive in the c / c++ source code ensures proper export of the shared library symbols in case of Windows vs. Unix.
As a bonus, the binary wheels are automatically compiled by a GitHub Action for all operating systems :-)

Error exporting symbol when building Python C Extension in Windows

4 Answers4

Solution: in Windows, you must define an initialization function for your extension because setuptools will build a .pyd file, not a DLL

Explanation