0

I wish to use some C and CUDA code in my Python package (which I then call using ctypes). Because of the CUDA, it doesn't seem to be easy to use the traditional approach of a setuptools Extension, so I instead pre-compile the code to shared libraries and then wish to include them in a Wheel. When I build the Wheel it includes the shared libraries, but it still gives the output Wheel a "none-any" extension, indicating that it is pure Python and platform independent, which is not correct. cibuildwheel then refuses to run auditwheel on it because of this. I have read posts about forcing Wheels to be labelled as platform dependent, but I imagine that I shouldn't have to force it and that I am instead doing something wrong.

Directory structure:

  • mypackage
    • pyproject.toml
    • setup.cfg
    • src
      • mypackage
        • __init__.py
        • aaa.py
        • bbb.c
        • ccc.cu
        • libmypackage_cpu.so
        • libmypackage_cuda.so
        • before_all.sh (script to build the .so files)

pyproject.toml:

[build-system]
requires = ["setuptools>=42"]
build-backend = "setuptools.build_meta"

[tool.cibuildwheel]
build = "cp36-*"
skip = ["pp*", "*686"]
manylinux-x86_64-image = "manylinux2014"

[tool.cibuildwheel.linux]
before-all = "bash {project}/src/mypackage/before_all.sh"

setup.cfg:

[metadata]
name = mypackage

[options]
package_dir =
    = src
include_package_data = True
zip_safe = False
packages = find:
python_requires = >=3.6

[options.packages.find]
where = src

[options.package_data]
mypackage =
    *.so

Am I missing something that is causing the packager to not detect that the Wheel is not pure Python and platform independent, or is forcing the packager to label it otherwise (such as by creating an empty Extension) the normal approach?

user3708067
  • 543
  • 5
  • 12
  • 1
    From the point of view of `setuptools` `package_data` is just data, `setuptools` doesn't look into it to recognize its nature so it sees the package as portable. `Extension` is (almsot) the only way to build platform-dependent package. [This is another way](https://stackoverflow.com/a/45150383/7976758). But whatever way you choose to build your package as platform-dependent I very much doubt `auditwheel` will fix data files. – phd Jun 12 '23 at 14:38

1 Answers1

1

I was able to get it to work using the method referred to by @phd, and I also got it to work using Hatchling instead of Setuptools as the backend. In both cases auditwheel is successful, finding the shared libraries and including their dependencies.

Both methods feel fragile, however. I have come to the realisation that Wheels are intended for Python extensions. I have no need for my compiled code to have any knowledge of Python, and while I could wrap calls to my compiled code using the likes of CFFI or Cython, I don't see an advantage of it in my case and it would introduce a dependence on the Python ABI.

I have thus decided to only provide a source distribution (no Wheels), including the shared libraries for all platforms in the source distribution and loading the correct one for the platform at runtime.

I should note that there are extensions of Setuptools that might also meet my needs, such as setuptools-cuda-cpp to enable Setuptools to compile CUDA, and setuptools_dso to build non-Python shared libraries for inclusion in Wheels with Setuptools.

In case it is useful to someone else, to get Setuptools and Hatchling to produce Wheels that declare themselves to only require Python 3 (not a particular release of CPython), have no Python ABI dependence, and be for the Linux x86_64 platform, I had to make the below changes.

Setuptools:

setup.py:

from setuptools import setup, Distribution

try:
    from wheel.bdist_wheel import bdist_wheel as _bdist_wheel
    class MyWheel(_bdist_wheel):

        def finalize_options(self):
            _bdist_wheel.finalize_options(self)
            self.root_is_pure = False

        def get_tag(self):
            python, abi, plat = _bdist_wheel.get_tag(self)
            python, abi = 'py3', 'none'
            return python, abi, plat

    class MyDistribution(Distribution):

        def __init__(self, *attrs):
            Distribution.__init__(self, *attrs)
            self.cmdclass['bdist_wheel'] = MyWheel

        def is_pure(self):
            return False

        def has_ext_modules(self):
            return True

except ImportError:
    class MyDistribution(Distribution):
        def is_pure(self):
            return False

        def has_ext_modules(self):
            return True

setup(
    distclass=MyDistribution
)

Hatchling:

hatch_build.py:

from hatchling.builders.hooks.plugin.interface import BuildHookInterface
class CustomHook(BuildHookInterface):
    def initialize(self, version, build_data):
        build_data['pure_python'] = False
        build_data['tag'] = 'py3-none-linux_x86_64'

and then, in addition to the other changes to pyproject.toml needed to use Hatchling, the line

[tool.hatch.build.targets.wheel.hooks.custom]

needs to be included to run the hook in the above hatch_build.py.

user3708067
  • 543
  • 5
  • 12