Package a pre-built python extension

Question

I am working on a C library (using cmake as a build system) and a corresponding python extension written in cython.

The build process is conducted by cmake, which calls the cython executable to generate a C file. The file is compiled into a python_library.so which links against the native library.so and other dependencies.

The library works as expected, I can set the PYTHONPATH to the build directory, run python and import and execute the wrapped python code.

What remains is the question about how to install / package the python module. As far as I know, the recommended method to create python packages is to use setuptools / distutils inside a setup.py file.

It is of course possible to define a C Extension (optionally using cython) inside the setup.py file. However, I want the compilation to be handled by cmake (it involves some dependent libraries etc.)

So basically, I would like to tell python that the whole package is defined by an existing python_library.so file. Is that at all possible?

Note: there is a related question. But the OP has already figured out how to package the extension.

Would [something like this](https://stackoverflow.com/questions/42585210/extending-setuptools-extension-to-use-cmake-in-setup-py) be the solution? Make setup.py invoke cmake? — DavidW, Jan 06 '19 at 21:53
Thank you for the link, but this is not exactly what I had in mind. In given example the roles of cmake and setup.py are reversed. I am fine with cmake building everything. I just want setup.py to run tests / build an egg or similar. — hfhc2, Jan 06 '19 at 23:01

ead · Accepted Answer · 2019-01-09T09:35:33.300

Obviously, this is not the most robust way to distribute python-packages as it will not work for different OSes or may lead to strange results if there is Python-version mismatch - but nevertheless it is possible.

Let's consider following folder structure:

/
|--- setup.py
|--- my_package
        |-------  __init__.py
        |-------  impl.pyx          [needed only for creation of impl.so]
        |-------  impl-XXX.so       [created via "cythonize -i impl.pyx"]

With the following content:

__init__.py:

from .impl import foo

impl.pyx:

def foo():
    print("I'm foo from impl")

setup.py:

from setuptools import setup, find_packages


kwargs = {
      'name':'my_package',
      'version':'0.1.0',
      'packages':find_packages(),

       #ensure so-files are copied to the installation:
      'package_data' : { 'my_package': ['*.so']},
      'include_package_data' : True,
      'zip_safe' : False  
}


setup(**kwargs)

Now after calling python setup.py install, the package is installed and can be used:

>>> python -c "import my_package; my_package.foo()"
I'm foo from impl

NB: Don't call the test from the folder with the setup file, because then not the installed but local version of my_package can be used.

You might want to have different so-binaries for different Python versions. It is possible to have the same extension compiled for different Python versions - you have to add the right suffix to the resulting shared library, for example:

impl.cpython-36m-x86_64-linux-gnu.so for Python3.6 on my linux machine
impl.cpython-37m-x86_64-linux-gnu.so for Python3.7
impl.cp36-win_amd64.pyd on windows

One can get the suffix for extensions on the current machine using

>>> import importlib
>>> importlib.machinery.EXTENSION_SUFFIXES
['.cp36-win_amd64.pyd', '.pyd']

Thank you for the comment. What do you mean by robustness? Clearly, the library is OS-dependent. I don't expect a package built on Linux (containing a *.so) to run on Windows. But what about different versions? I would expect the Python C API / ABI to be stable?! — hfhc2, Jan 08 '19 at 20:53
@hfhc2 Python 3.X isn't meant to work with so-files build with Python 3.Y. That is the reason, so-files (and pyc-files) have suffix a la `cpython-36m-x86_64-linux-gnu` so there will be no mixup. — ead, Jan 08 '19 at 21:16
Well, I would like to follow that convention. Is it sufficient to name the library according to the above pattern? Can I get the suffix programatically? — hfhc2, Jan 09 '19 at 08:39
@hfhc2 There is an easy way to find it on the local machine (see my update), not sure how to get it without having the targeted machine at hand. But this is also probably could be a separate question — ead, Jan 09 '19 at 09:37

Package a pre-built python extension

1 Answers1