python: create wheel from existing c extension

Question

Our build pipeline spits out an *.so / *.pyb python extension through pybind11. As a step in the pipeline I need to package the extension as a wheel for easy distribution through pip. I am trying to come up with a setup.py that takes the existing library and does not rely on re-compiling the binaries through the setup.py (so I really don't want this). It would require a major rewrite of the devops scripts.

When having a folder structure such as:

setup.py
my_module.cpython-39-darwin.so

A very basic setup.py can create a functioning wheel (python setup.py bdist_wheel):

setup(
    name = 'my_module', 
    version='0.9.102', 
    packages=["."],
    package_data={'': ['*.so', '*.pyi']},
)

Unfortunately, the wheel is missing the important python tag and platform name, etc: my_module-0.9.102-py3-none-any.whl vs. my_module-0.9.102-cp39-cp39-macosx_10_13_x86_64.whl

Setting --python-tag and --plat-name works, setting --py-limited-api does not, however.

Through research I found that overwriting the distclass adds the correct tag again, but the Root-Is-Purelib is set back to false. This, unfortunately, creates a broken wheel when installing through pip as it puts the binary in a my_module-0.9.102.data/purelib folder...

Overwriting the is_pure seems to be ignored also:

from setuptools import setup, find_packages, Distribution

class BinaryDistribution(Distribution):
    def is_pure(self):
        return True

    def has_ext_modules(foo):
        return True

setup(
    name = 'my_module', 
    version='0.9.102', 
    packages=["."],
    package_data={'': ['*.so', '*.pyi']},
    distclass=BinaryDistribution
)

What else can I do to wrap my pre-compiled python libraries to wheels for distribution without rewriting lots of the build pipeline?

Have you read https://pybind11.readthedocs.io/en/stable/compiling.html ? — md2perpe, Dec 19 '21 at 14:15
Yes, but as said, I don’t want to compile the project again, the libs are already there — Sc4v, Dec 19 '21 at 15:17
If you don't find a better way, you can always build the whl-file without using setuptools or disttools. It's just a zip-file. — md2perpe, Dec 19 '21 at 16:47

score 1 · Answer 1 · answered Jan 12 '22 at 16:57

Not a perfect solution but a functional wheel is created when using the wheel module instead:

from setuptools import setup

setup(name='my_module',
      packages=['my_module'],
      package_data={'': ['*.so', '*.pyi']},
      )

create wheel with:

pip install wheel
pip wheel .

python: create wheel from existing c extension

1 Answers1