Using Cython To Link Python To A Shared Library

Question

I am trying to integrate a third party library written in C with my python application using Cython. I have all of the python code written for a test. I am having trouble finding an example for setting this up.

I have a pyd/pyx file I created manually. The third party has given me a header file (*.h) and a shared library (*.so). As far as I can tell, there are no other dependencies. Can someone provide an example of how to set this up using Cython and disutils?

Thanks

Gauthier Boaglio · Accepted Answer · 2017-11-29T18:57:34.863

Sure !

(In the following, I assume that you already know how to deal with cimport and the interactions between .pxd and .pyx. If this is not completely the case, just ask and I will develop that part as well)

The sample (grabbed from a C++ project of mine, but a C project would work pretty much the same) :

1. The Distutils setup file :

Assuming that the extension to be created will be called myext and the 3rd party shared library is libexternlib.so (note the lib* prefix, here)...

# setup.py file
import sys
import os
import shutil

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

# clean previous build
for root, dirs, files in os.walk(".", topdown=False):
    for name in files:
        if (name.startswith("myext") and not(name.endswith(".pyx") or name.endswith(".pxd"))):
            os.remove(os.path.join(root, name))
    for name in dirs:
        if (name == "build"):
            shutil.rmtree(name)

# build "myext.so" python extension to be added to "PYTHONPATH" afterwards...
setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = [
        Extension("myext", 
                  sources=["myext.pyx",
                           "SomeAdditionalCppClass1.cpp",
                           "SomeAdditionalCppClass2.cpp"
                       ],
                  libraries=["externlib"],          # refers to "libexternlib.so"
                  language="c++",                   # remove this if C and not C++
                  extra_compile_args=["-fopenmp", "-O3"],
                  extra_link_args=["-DSOME_DEFINE_OPT", 
                                   "-L./some/extra/dependency/dir/"]
             )
        ]
)

Note : Your external .so file is linked via the libraries option :

libraries=["externlib"]   # Without the 'lib' prefix and the '.so' extension...

Note : the sources option can be used to get some additional source files compiled.

Important : myext.pxd (do not confound with .pyd - Windows stuff) and myext.pyx should be in the same directory. At compile time the definition file, if it exists, is processed first (more).

2. Then run it as follows :

After having changed directory to the one containing your myext.pxd, your myext.pyx, as well as the above setup.py script :

# setup.sh
# Make the "myext" Python Module ("myext.so")
CC="gcc"   \
CXX="g++"   \
CFLAGS="-I./some/path/to/includes/ -I../../../DEPENDENCIES/python2.7/inc -I../../../DEPENDENCIES/gsl-1.15"   \
LDFLAGS="-L./some/path/to/externlib/"   \
    python setup.py build_ext --inplace

Where :

libexternlib.so is assumed to be located at ./some/path/to/externlib/
yourheader.h is assumed to be located at ./some/path/to/includes/

Note : CFLAGS could also have been setup using the extra_compile_args option :

extra_compile_args=["-I./some/path/to/includes/", "-fopenmp", "-O3"]

Note : LDFLAGS could also have been setup using the extra_link_args option :

extra_link_args=["-L./some/path/to/externlib/", "-DSOME_DEFINE_OPT", "-L./some/extra/dependency/dir/"]

Once distutils is done with the build, you get some new files, specially the myext.cpp, myext.h and most importantly, the myext.so.

3. After that, you're good to go :

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:./some/path/to/externlib/
export PYTHONPATH=$PYTHONPATH:./some/path/to/myext/

# Run some script requiring "myext.so"
python somescript.py

Where your freshly created Python extension can be imported by its name :

# somescript.py
import myext
from myext import PySomeFeature
...

Note about Optimization : By default -O2 is used for compiling the extension, but this can be overloaded (see above setup where -O3 is specified).

Note about Cython paths : If Cython was installed in a custom directory, you might want to add it to your environment, before all :

PYTHONPATH=$PYTHONPATH:../../../DEPENDENCIES/Cython-0.18 export PYTHONPATH;
PATH=$PATH:../../../DEPENDENCIES/Cython-0.18/bin; export PATH;

Well, hope I covered the main points...

Hey, Thanks for all your help. I am getting the following error: ld: library not found for -lMYLIB where MYLIB.so is the file I am trying to link against - any ideas? — josephmisiti, Jun 08 '13 at 16:14
forget it, i just read that the gcc -l command assumes your *.so is prefaced with lib so i removed that and it found it and seems to be working. — josephmisiti, Jun 08 '13 at 16:37
Very good code - except what's with the trailing backslashes toward the end? You don't need 'em (you're in parentheses) - in general, you NEVER need trailing backslashes in proper Python (always do it with parens, braces, or brackets). — Tom Swirly, Feb 03 '14 at 18:07
How can I embed the libexternlib inside the python module (for shipping purpose)? — Dilawar, Feb 27 '14 at 21:11
@Dilawar Not sure what you mean by "shipping". Once the library is released, the only requirement is to have its containing folder's path added to the environment (`$LD_LIBRARY_PATH` and `$PYTHONPATH`- See 3rd point of the above answer). Use absolute paths and add those export commands to your `.profile` or `.bashrc` to make it permanent. — Gauthier Boaglio, Feb 28 '14 at 10:10
useful but `-DSOME_DEFINE_OPT` will be fined for the linker, but usually the compiler needs extra defines — dashesy, Sep 07 '15 at 18:46
@Gauthier Boaglio thanks for answer, it helped. Can you also help with packaging this as a wheel? — skboro, Mar 30 '20 at 11:23

Using Cython To Link Python To A Shared Library

1 Answers1

Linked