Packaged Python C Extension is missing PyInit_module() in so File

Question

I have created a C library libgac and then wrote Python module implementing bindings with ctype. I call the python module gazepy.

My project file structure is as follows:

pyproject.toml
setup.py
src
  gazepy
    gac          # repo with the C library libgac
    gazepy.py    # python bindings for the C library libgac
    __init__.py  # empty file
tests

Running sudo pip install . generates and installs the so file gazepy.cpython-38-x86_64-linux-gnu.so.

So far so good, however, if I want to import the library in a python3 shell I get the following error:

ImportError: dynamic module does not define module export function (PyInit_gazepy)

My pyproject.toml file:

[build-system]
requires = ["setuptools>=61"]
build-backend = "setuptools.build_meta"

[project]
name = "gazepy"
# ...

My setup.py file:

from setuptools import setup, Extension, find_packages
import subprocess

module_name = "gazepy"
module_path = "src/" + module_name
gac_path = module_path + "/gac"
cglm_path = gac_path + "/cglm"

def main():
    setup(
            name=module_name,
            packages=find_packages(),
            ext_modules=[
                Extension(module_name, [gac_path + '/src/gac.c'],
                    include_dirs=[gac_path  + '/include', cglm_path + '/include'],
                    libraries=['m'])
            ]
    )


if __name__ == "__main__":
    main()

When looking at the so file gazepy.cpython-38-x86_64-linux-gnu.so I cannot find any definition of PyInit_gazepy():

nm /usr/local/lib/python3.8/dist-packages/gazepy.cpython-38-x86_64-linux-gnu.so | grep PyInit

What am I missing here?

StackOverflow/StackExchange is a QA (Questions-and-Answers) network. I suggest moving your solution to the answer box below even if you accept a different answer. — phd, Aug 11 '23 at 15:31
[\[SO\]: How to solve Python-C-API error "This is an issue with the package mentioned above, not pip."? (@CristiFati's answer)](https://stackoverflow.com/a/75307579/4788546) — CristiFati, Aug 23 '23 at 05:51

Ahmed AEK · Accepted Answer · 2023-08-11T11:29:15.640

python extension modules (.so and .pyd files) have higher priority than .py files when using the import command, so python is trying to import the .so file instead of the .py file.

in order for import gazepy to correctly import your python file you should change the name of the extension module module_name to have an underscore _gazepy, this way python is not going to load it, then you should only load your .so using ctypes.

you are compiling your C library as a python extension, which is wrong, as it now depends on python

the above solution will work although your library is not a python extension module, but you should consider Compiling & installing C executable using python's setuptools/setup.py so it won't depend on python, unfortunately this has worse portability than just treating this C library as an extension module, so if you care about producing a clean library you should try compiling it as a C library instead of an extension module, but if you care about low effort portability then keep your current python extension method.

Thanks for the quick reply. Eventually, I ended up compiling and installing the C executable (as you recommended). I will add my final solution to my question. — moiri, Aug 11 '23 at 13:53

score 0 · Answer 2 · answered Aug 25 '23 at 13:12

Here is an answer to my own question (based on Ahmeds reply)

I was not able to make the C extension work. However, as pointed out by Ahmed it would anyways be better to compile the binary and add it to the package.

I ended up doing the following:

I changed my file structure as follows:

pyproject.toml
setup.py
gac          # repo with the C library libgac
gazepy
  gazepy.py    # python bindings for the C library libgac
  __init__.py  # empty file
  bin          # target location for so file
tests

I used the following setup.py script (where I use my own Makefile and then copy the file to the bin folder):

from setuptools import setup, Extension, find_packages
from setuptools.command.sdist import sdist
import subprocess
import os

src_path = "gazepy"
gac_path = "gac"

def compile_and_install_software():
  """Used the subprocess module to compile/install the C software."""
  # compile the software
  subprocess.check_call('make', cwd=gac_path, shell=True)
  os.rename(gac_path + '/build/lib/libgac.so', src_path + '/bin/libgac.so')


class CustomInstall(sdist):
  """Custom handler for the 'install' command."""
  def run(self):
    compile_and_install_software()
    super().run()


def main():
  setup(
         name='gazepy',
         package_data={
             'gazepy':['bin/libgac.so']
         },
         packages=['gazepy', 'gazepy.bin'],
         cmdclass={'sdist': CustomInstall}
  )

if __name__ == "__main__":
  main()

Added the following line to the __init__.py file:
```
from .gazepy import *
```

Packaged Python C Extension is missing PyInit_module() in so File

2 Answers2