Calling a cython library with multiple pyx files through c++

Question

I have a python project that I want to call from a c++ application. I would like to bundle all python sources together in a single shared library and link the c++ application to that library. Right now my cython setup.py creates one *.so per python source, which is very inconvenient.

Here is the setup.py file:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize

sourcefiles = ['project_interface.pyx', 'impl_file1.pyx']

setup(
    ext_modules = cythonize(sourcefiles)
)

project_interface.pyx :

# distutils: language = c++

import impl_file1

cdef public void HelloWorld():
    print "Calling Hello World"
    impl_file1.helloworld_func()

impl_file1.pyx :

def helloworld_func():
    print 'Hello World'

I tried to modify setup.py to bundle all python code in a single library like this:

setup(
      ext_modules = cythonize([Extension("my_project", sourcefiles, language='c++')])
)

Unfortunately, when executing void HelloWorld(), the application cannot file impl_file1 anymore. I get :

Calling Hello World
NameError: name 'impl_file1' is not defined
Exception NameError: "name 'impl_file1' is not defined" in 'project_interface.HelloWorld' ignored

The c++ program driving this is:

#include <Python.h>
#include "project_interface.h"

int main(int argc, const char** argv){
    Py_Initialize();
    initproject_interface();
    HelloWorld();
    Py_Finalize();


    return 0;
}

This application works correctly when compiling with multiple *.so files.

Compilation is very straightforward in either cases:

python setup.py build_ext --inplace
mv my_project.so libmy_project.so
g++ main.cpp -o main `python2-config --cflags --ldflags` -L. -lmy_project

Is there any way to get the single shared library solution to work?

Answer 1

There's a number of similar looking questions about bundling multiple Cython modules together (e.g. 1, 2) which isn't really viable because Python uses file paths to handle modules. However, this question isn't quite the same because you're calling it from C++, which gives you an extra option.

You need to use the C API function PyImport_AppendInittab to Python to treat impl_file1 as a builtin module so it doesn't search the path for a file to import. Start by providing a declaration of the import function (since you won't get that from your header file):

extern "C" {
// PyObject* PyInit_impl_file1(); // Python 3
void initimpl_file1(); // Python 2
}

Then, in main, before Py_Initialize, add:

PyImport_AppendInittab("impl_file1", initimpl_file1); // change the name for Python 3

Answer 2

For me (in a slightly different scenario, but I didn't expect a difference) the solution of @DavidW needs some tweaking. Here is my setup:

foo.pyx:

cdef extern from "Python.h":
    int PyImport_AppendInittab(const char *name, object (*initfunc)())


cdef extern from *:
    """
    PyObject *PyInit_bar(void);
    """
    object PyInit_bar()

PyImport_AppendInittab("bar", PyInit_bar)

import bar  # HERE The error happens

bar.pyx:

print("bar imported")

setup.py:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize

sourcefiles = ['foo.pyx', 'bar.pyx']


setup(
    ext_modules = cythonize([Extension("foo", sourcefiles)])
)

Now after building with python setup.py build_ext -i, result in an error:

import foo
ImportError: 'bar' is not a built-in module

coming from here. To work around I have to add the name "bar" to sys.builtin_module_names:

...
import sys
sys.builtin_module_names = list(sys.builtin_module_names)+["bar"]
import bar