2

I have a large C/C++ project which I usually just build as a standalone executable. It also requires some libraries (system libraries like pthread, but also MKL from intel for example) which I specify when compiling the project.

Now, I want to use some functions from this project in Python. So my first plan was to simply build the project as a static library and then use that to write a cython wrapper. I.e.

  1. Build the c project: icc -c src/main.cpp .. options and stuff.. linking to all libraries needed .. -o main.o

  2. Create a static library (including the libraries I needed in step 1): ar -rc libmain.a main.o ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a ... /usr/lib/x86_64-linux-gnu/libpthread.a ...

  3. I use the generated static library to build my cython wrapper

  4. when I try to execute a test python script calling a random function from my C program I get this error ImportError: /home/.../main.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_ok_to_fork

It seems like I need to tell cython to link to the corresponding libraries again or something, but I'm not really sure how to resolve this.. Also I don't want the python project to be dependent on some special libraries that I have installed on my system (that's what I try to achieve by adding all the libraries in step 2), is this possible at all?

My setup.py looks like this

from setuptools import setup, Extension
from Cython.Build import cythonize
import os

os.environ["CC"] = "icc"
os.environ["CXX"] = "icc"
setup(
   ext_modules = cythonize([Extension("test", ["test.pyx"], language="c++",
                                  library_dirs=[r'.'], libraries=['main'])])
)
user2224350
  • 2,262
  • 5
  • 28
  • 54
  • A quick search suggests that `__kmpc_ok_to_fork` is in `libiomp5`. Can you try adding "iomp5" to the libraries? – brm May 18 '21 at 20:48
  • no, unfortunately the error still remains unchanged.. – user2224350 May 18 '21 at 21:02
  • Also from adding all the libraries the resulting static library file is getting really huge (like almost 1Gb). There must be a smarter way to do this.. If I don't include the libraries and somehow manage to tell cython to do all the linking, will the -so file generated by cython be independent of all the libraries? Like could I still just copy it to a different machine? – user2224350 May 18 '21 at 21:07
  • 1
    The issue is, that the linker cannot use your static library. The correct way to create it is https://stackoverflow.com/a/50031940/5769463 – ead May 19 '21 at 04:07
  • But usually you don’t use static versions of libraries - this can lead to strange results (as ODR could be violated) – ead May 19 '21 at 04:10
  • Ok I see. I first have to unpack the .a files.. But if creating a static library isn't the proper solution, how would you go about this? – user2224350 May 19 '21 at 07:55
  • This might be a possible solution (distributing needed shared libraries): https://stackoverflow.com/q/63804883/5769463 – ead May 26 '21 at 08:39
  • This is not entirely in the spirit of the question, but I highly recommend using pybind11 for Python/C++ bindings. https://github.com/pybind/pybind11 – unddoch May 26 '21 at 20:53

1 Answers1

1

Create a shared library (.so on linux). Python can only support dynamic or shared libraries. The linker is looking for a .so not a .a.

So your setup.py would be remain the same,

from setuptools import setup, Extension
from Cython.Build import cythonize
import os

os.environ["CC"] = "icc"
os.environ["CXX"] = "icc"
setup(
   ext_modules = cythonize([Extension("test", ["test.pyx"], language="c++",
                                  library_dirs=[r'.'], libraries=['main'])])
)

Related - Using Cython To Link Python To A Shared Library.

Shreyan Avigyan
  • 183
  • 1
  • 11
  • Setup.py doesn’t look for libraries, it only passes names of the libraries to the linker, which will pick the static version (e.g. .a) if there is no shared version. – ead May 26 '21 at 04:26
  • Yes setup.py does pass the name to the linker. Cythonize converts Cython to C. And Python can only work with C code that links with a shared library not a static one. – Shreyan Avigyan May 26 '21 at 07:58
  • If static library was compiled with position independent code, there is no reason it cannot be used in a python-extension (and as said, setup.py does nothing to prevent that) – ead May 26 '21 at 08:16
  • @ead I am a volunteer in CPython and I'm 99% sure that Python can't work with static library. The main point is if the code now works or not. And also notice I edited my answer to change setup.py to linker. – Shreyan Avigyan May 26 '21 at 08:19
  • It is even possible to build a static library during the setup.py and use it in multiple extensions. See for example https://stackoverflow.com/a/57673781/5769463 – ead May 26 '21 at 08:27