Python extension with multiple modules

Question

I am building Python bindings for a standalone C library that I wrote. The file layout of the library is as following:

<project root>
|
`- cpython
|  |
|  `- module1_mod.c
|  `- module2_mod.c
|  `- module3_mod.c
|
`- include
|  |
|  `- module1.h
|  `- module2.h
|  `- module3.h
|
`- src
|  |
|  `- module1.c
|  `- module2.c
|  `- module3.c
|
`- setup.py

I want to obtain a Python package so I can import modules in a namespace such as my_package.module1, my_package.module2, etc.

This is my setup.py so far:

from os import path
from setuptools import Extension, setup


ROOT_DIR = path.dirname(path.realpath(__file__))
MOD_DIR = path.join(ROOT_DIR, 'cpython')
SRC_DIR = path.join(ROOT_DIR, 'src')
INCL_DIR = path.join(ROOT_DIR, 'include')
EXT_DIR = path.join(ROOT_DIR, 'ext')

ext_libs = [
    path.join(EXT_DIR, 'ext_lib1', 'lib.c'),
    # [...]
]

setup(
    name="my_package",
    version="1.0a1",
    ext_modules=[
        Extension(
            "my_package.module1",
            [
                path.join(SRC_DIR, 'module1.c',
                path.join(MOD_DIR, 'module1_mod.c',
            ] + ext_libs,
            include_dirs=[INCL_DIR],
            libraries=['uuid', 'pthread'],
        ),
    ],
)

Importing mypackage.module1 works but the problem is that the external libraries are also needed by module2 and module3 (not all of them for all the modules), and I assume that if I include the same external libs in the other modules, I would get a lot of bloat.

I looked around sample setups in Github but haven't found an example resolving this problem.

What is a good way to organize my builds?

EDIT: This is actually a more severe problem in that I have symbols in module1 that are needed in module2, etc. E.g. an object in module2 requires an object type defined in module1. If I create separate binaries without including all sources for each dependency, the symbols won't be available at linking time, thus increasing redundancy and complexity of keeping track of what is needed for which module.

score 3 · Accepted Answer · answered Dec 19 '20 at 18:37

After a couple of days of digging into Python bug reports and scarcely documented features, I found an answer to this, which resolved both the multiple external dependencies and the internal cross-linking.

The solution was to create a monolithic "module" with all the modules defined inside it, then exposing them with a few lines of Python code in a package initialization file.

To do this I changed the module source files to header files, maintaining most of their methods static and only exposing the PyTypeObject structs and my object type structs so they can be used in other modules.

Then I moved the PyMODINIT_FUNC functions defining all the modules in a "package" module (py_mypackage.c), which also defines an empty module. the "package" module is defined as _my_package.

Finally I added some internal machinery to an __init__.py script that extracts the module symbols from the .so file and exposes them as modules of the package. This is documented in the Python docs :

import importlib.util
import sys

import _my_package


pkg_path = _my_package.__file__


def _load_module(mod_name, path):
    spec = importlib.util.spec_from_file_location(mod_name, path)
    module = importlib.util.module_from_spec(spec)
    sys.modules[mod_name] = module
    spec.loader.exec_module(module)

    return module


for mod_name in ('module1', 'module2', 'module3'):
    locals()[mod_name] = _load_module(mod_name, pkg_path)

The new layout is thus:

<project root>
|
`- cpython
|  |
|  `- my_package
|    |
|    `- __init__.py
|
|  `- py_module1.h
|  `- py_module2.h
|  `- py_module3.h
|  `- py_mypackage.c
|
`- include
|  |
|  `- module1.h
|  `- module2.h
|  `- module3.h
|
`- src
|  |
|  `- module1.c
|  `- module2.c
|  `- module3.c
|
`- setup.py

And setup.py:

setup(
    name="my_package",
    version="1.0a1",
    package_dir={'my_package': path.join(CPYTHON_DIR, 'my_package')},
    packages=['my_package'],
    ext_modules=[
        Extension(
            "_my_package",
            "<all .c files in cpython folder + ext library sources>",
            libraries=[...],
        ),
    ],
)

For the curious, the complete code is at https://notabug.org/scossu/lsup_rdf/src/e08da1a83647454e98fdb72f7174ee99f9b8297c/cpython (pinned at the current commit).

Python extension with multiple modules

1 Answers1

Linked