Packaging python with cython extension

Question

I'm trying to build a package that uses both python and cython modules. The problem I'm having deals with imports after building and installing where I'm not sure how to make files import from the .so file generated by the build process.

Before building my folder structure looks like this

root/
├── c_integrate.c
├── c_integrate.pyx
├── cython_builder.py
├── __init__.py
├── integrator_class.py
├── integrator_modules
│   ├── cython_integrator.py
│   ├── __init__.py
│   ├── integrator.py
│   ├── numba_integrator.py
│   ├── numpy_integrator.py
│   ├── quadratic_error.png
│   ├── report3.txt
│   ├── report4.txt
│   └── report5.txt
├── report6.txt
├── setup.py
└── test
    ├── __init__.py
    └── test_integrator.py

Building with python3.5 setup.py build gives this new folder in root

root/build/
├── lib.linux-x86_64-3.5
│   ├── c_integrate.cpython-35m-x86_64-linux-gnu.so
│   ├── integrator_modules
│   │   ├── cython_integrator.py
│   │   ├── __init__.py
│   │   ├── integrator.py
│   │   ├── numba_integrator.py
│   │   └── numpy_integrator.py
│   └── test
│       ├── __init__.py
│       └── test_integrator.py

The setup.py file looks like this

from setuptools import setup, Extension, find_packages
import numpy

setup(
    name = "integrator_package",
    author = "foo",
    packages = find_packages(),
    ext_modules = [Extension("c_integrate", ["c_integrate.c"])],
    include_dirs=[numpy.get_include()],
)

My question is then: how do I write import statements of the functions from the .so file into ìntegrator_class.py in root and cython_integrator and test_integrator located in the build directory. Appending to sys.path seems like a quick and dirty solution that I don't much like.

EDIT: As pointed out in the comments I haven't installed the package. This is because I don't know what to write to import from the .so file

Is [this](https://stackoverflow.com/questions/19048732/python-setup-py-develop-vs-install) relevant? — DavidW, Oct 06 '17 at 11:55
Seems like you don't install the package you've built like @DavidW pointed out. Issue `pip install path/to/root/` or `pip install --editable=path/to/root/` to marry the built files with python. — hoefling, Oct 06 '17 at 11:58
Yup, I haven't installed it - but my problem is that since the `.so` file is named platform specific I don't know how to import it. Especially since the `lib.linux...` folder doesn't have an `__init__.py` file. Aditionally: after I install and just try to rename the `.so` file and import it raises `SystemError: Parent module '' not loaded, cannot perform relative import ` — Copperwire, Oct 06 '17 at 12:20

Pierre de Buyl · Accepted Answer · 2017-10-06T16:28:04.170

In no specific order:

The file setup.py is typically located below the root of a project. Example:
```
library_name/
    __init__.py
    file1.py
setup.py
README
```
Then, the build directory appears alongside the project's source and not in the project source.
To import the file c_integrate.cpython-35m-x86_64-linux-gnu.so in Python, just import "c_integrate". The rest of the naming is taken care of automatically as it is just the platform information. See PEP 3149
A valid module is one of
1. a directory with a modulename/__init__.py file
2. a file named modulename.py
3. a file named modulename.PLATFORMINFO.so
of course located in the Python path. So there is no need for a __init__.py file for a compiled Cython module.

For your situation, move the Cython code in the project directory and either do a relative import import .c_integrate or a full from integrator_modules import c_integrate where the latter only works when your package is installed.

A few of this information can be found in my blog post on Cython modules http://pdebuyl.be/blog/2017/cython-module.html

I believe that this should let you build a proper package, comment below if not.

EDIT: to complete the configuration (see comments below), the poster also

Fixed the module path in the setup.py file so that it is the full module name starting from the PYTHONPATH: Extension("integrator_package.integrator_modules.c_integrat‌or", ["integrator_package/integrator_modules/c_integrator.c"] instead of Extension("c_integrate", ["c_integrate.c"])]
Cythonize the module, build it and use with a same Python interpreter.

Further comment: the setup.py file can cythonize the file as well. Include the .pyx file instead of the .c file as the source.

cythonize(Extension('integrator_package.integrator_modules.c_integrat‌or',
          ["integrator_package/integrator_modules/c_integrator.pyx"],
          include_dirs=[numpy.get_include()]))

Thanks a lot for the answer! I made changes to the tree so it looks like you suggested. I now get an error on importing the cython module `ImportError: dynamic module does not define module export function (PyInit_c_integrator)`. The setup file builds extensions as `ext_modules = [Extension("integrator_package.integrator_modules.c_integrator", ["integrator_package/integrator_modules/c_integrator.c"]` — Copperwire, Oct 06 '17 at 14:03
Solved with the answer from [this](https://stackoverflow.com/questions/36723206/cython-compilation-error-dynamic-module-does-not-define-module-export-function) thread. Turns out `cython` is build sensitive — Copperwire, Oct 06 '17 at 15:16

Packaging python with cython extension

1 Answers1