python import path for sub modules if put in namespace package

Question

I have a python modules written in C, it has a main module and a submodule(name with a dot, not sure this can be called real submodule):

PyMODINIT_FUNC initsysipc(void) {
    PyObject *module = Py_InitModule3("sysipc", ...);
    ...
    init_sysipc_light();
}

static PyTypeObject FooType = { ... };
PyMODINIT_FUNC init_sysipc_light(void) {
    PyObject *module = Py_InitModule3("sysipc.light", ...);
    ...
    PyType_Ready(&FooType);
    PyModule_AddObject(module, "FooType", &FooType);
}

The module is compiled as sysipc.so, and when I put it in current directory, following import works without problem:

import sysipc
import sysipc.light
from sysipc.light import FooType

The problem is I want to put this module inside a namespace package, the folder structure is like this:

company/
company/__init__.py
company/dept/
company/dept/__init__.py
company/dept/sys/
company/dept/sys/__init__.py
company/dept/sys/sysipc.so

all the three __init__.py just includes the standard setuptool import line:

__path__ = __import__('pkgutil').extend_path(__path__, __name__)

in current directory, following imports does not work:

from company.dept.sys import sysipc;
from company.dept.sys.sysipc.light import FooType;

How should I import the types and methods defined in module sysipc.light in this case?

===================================

Update with the actual error:

I have sysipc.so built, if I run python in current directory as this module, import will work as expected:

[root@08649fea17ef 2]# python2
Python 2.7.18 (default, Jul 20 2020, 00:00:00)
[GCC 10.1.1 20200507 (Red Hat 10.1.1-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysipc
>>> import sysipc.light
>>>

If however if I put it into a namespace folder, like this:

company/
company/__init__.py
company/dept
company/dept/__init__.py
company/dept/sys
company/dept/sys/sysipc.so
company/dept/sys/__init__.py

import the submodule will not work:

>>> from company.dept.sys import sysipc
>>> from company.dept.sys import sysipc.light
  File "<stdin>", line 1
    from company.dept.sys import sysipc.light
                                   ^
SyntaxError: invalid syntax
>>> from company.dept.sys.sysipc import light
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name light
>>>

The module is built with this simple code, it is for python2. I also have same example for python3.

Answer 1

Quoting from https://www.python.org/dev/peps/pep-0489/#multiple-modules-in-one-library :

To support multiple Python modules in one shared library, the library can export additional PyInit* symbols besides the one that corresponds to the library's filename.

Note that this mechanism can currently only be used to load extra modules, but not to find them. (This is a limitation of the loader mechanism, which this PEP does not try to modify.) ...

In other words, you need to restructure the project as follows for importlib to be able to find the submodule light in the sysipc package:

company/__init__.py
company/dept/__init__.py
company/dept/sys/__init__.py
company/dept/sys/sysipc/__init__.py
company/dept/sys/sysipc/sysipc.so
company/dept/sys/sysipc/light.so -> sysipc.so  # hardlink

The hardlink between light.so and sysipc.so can be created with:

ln company/dept/sys/sysipc/sysipc.so company/dept/sys/sysipc/light.so

Then in company/dept/sys/sysipc/__init__.py you import all symbols from sysipc.so using:

from .sysipc import *

In addition, you need to change the name of the submodule C extension init function from init_sysipc_light to init_light for Python2, or from PyInit_sysipc_light to PyInit_light for Python3, since importlib loads modules by looking for an exported PyInit_<module name> from the dynamic module and the module name here is only light, i.e., the parent package prefix is not part of the (sub)module name.

Here is the extension code (Python3) and a couple of functions for testing:

#include <Python.h>

PyObject *sysipc_light_foo(PyObject *self, PyObject *args) {
  printf("[*] sysipc.light.foo\n");
  return PyLong_FromLong(0);
}

static PyMethodDef sysipc_light_methods[] = {
    {"foo", (PyCFunction)sysipc_light_foo, METH_VARARGS, "sysipc.light.foo function"},
    {NULL, NULL, 0, NULL}
};

static struct PyModuleDef sysipc_light_module = {
    PyModuleDef_HEAD_INIT,
    "sysipc.light",
    "sysipc child module",
    -1,
    sysipc_light_methods
};

PyMODINIT_FUNC PyInit_light(void)
{
    PyObject *module = NULL;

    module = PyModule_Create(&sysipc_light_module);

    return module;
}

PyObject *sysipc_bar(PyObject *self, PyObject *args) {
  printf("[*] sysipc.bar\n");
  return PyLong_FromLong(0);
}

static PyMethodDef sysipc_methods[] = {
    {"bar", (PyCFunction)sysipc_bar, METH_VARARGS, "sysipc.bar function"},
    {NULL, NULL, 0, NULL}
};

static struct PyModuleDef sysipc_module = {
    PyModuleDef_HEAD_INIT,
    "sysipc",
    "sysipc parent module",
    -1,
    sysipc_methods
};

PyMODINIT_FUNC PyInit_sysipc(void)
{
    PyObject *module = NULL;

    module = PyModule_Create(&sysipc_module);

    PyInit_light();

    return module;
}

test.py:

#!/usr/bin/env python3

from company.dept.sys import sysipc
from company.dept.sys.sysipc import light

sysipc.bar() 
light.foo()

Output:

[*] sysipc.bar
[*] sysipc.light.foo

Answer 2

There are two issues here: first, Py_InitModule and friends expect to create the module being imported. The hint is that the string you pass it is not the fully-qualified name of the module: Python uses the name it already knows to determine where in sys.modules to put the new object. You can, however, use the fully qualified name; other magic attributes like __file__ will have the correct values.

The second issue is that the attribute light needs to be set on the containing module for from imports to work.

Meanwhile, there's no reason to have a separate initialization function (that the interpreter will never call), and combining them avoids the need to recover a pointer to the module later:

static PyTypeObject FooType = { ... };
PyMODINIT_FUNC initsysipc(void) {
    PyObject *module = Py_InitModule3("sysipc", ...);
    ...
    PyObject *const sub = Py_InitModule3("company.dept.sys.sysipc.light", ...);
    ...
    PyType_Ready(&FooType);
    // PyModule_AddObject steals a reference:
    Py_INCREF(FooType);
    PyModule_AddObject(sub, "FooType", &FooType);
    Py_INCREF(sub);
    PyModule_AddObject(module, "light", sub);
}

That said, sysipc will still not be a proper package: at the least, it lacks __path__. If that matters, you might prefer MEE's answer that uses a real (if more complicated) package architecture.