26

I am working on a python2 package in which the setup.py contains some custom install commands. These commands actually build some Rust code and output some .dylib files that are moved into the python package.

An important point is that the Rust code is outside the python package.

setuptools is supposed to detect automatically if the python package is pure python or platform specific (if it contains some C extensions for instance). In my case, when I run python setup.py bdist_wheel, the generated wheel is tagged as a pure python wheel: <package_name>-<version>-py2-none-any.whl. This is problematic because I need to run this code on different platforms, and thus I need to generated one wheel per platform.

Is there a way, when building a wheel, to force the build to be platform specific ?

Adrien Ball
  • 440
  • 1
  • 4
  • 14

5 Answers5

35

Here's the code that I usually look at from uwsgi

The basic approach is:

setup.py

# ...

try:
    from wheel.bdist_wheel import bdist_wheel as _bdist_wheel
    class bdist_wheel(_bdist_wheel):
        def finalize_options(self):
            _bdist_wheel.finalize_options(self)
            self.root_is_pure = False
except ImportError:
    bdist_wheel = None

setup(
    # ...
    cmdclass={'bdist_wheel': bdist_wheel},
)

The root_is_pure bit tells the wheel machinery to build a non-purelib (pyX-none-any) wheel. You can also get fancier by saying there are binary platform-specific components but no cpython abi specific components.

anthony sottile
  • 61,815
  • 15
  • 148
  • 207
20

The modules setuptools, distutils and wheel decide whether a python distribution is pure by checking if it has ext_modules.

If you build an external module on your own, you can still list it in ext_modules so that the building tools know it exists. The trick is to provide an empty list of sources so that setuptools and distutils will not try to build it. For example,

setup(
    ...,
    ext_modules=[
        setuptools.Extension(
            name='your.external.module',
            sources=[]
        )
    ]
)

This solution worked better for me than patching the bdist_wheel command. The reason is that bdist_wheel calls the install command internally and that command checks again for the existence of ext_modules to decide between purelib or platlib install. If you don't list the external module, you end up with the lib installed in a purelib subfolder inside the wheel. That causes problems when using auditwheel repair, which complains about the extensions being installed in a purelib folder.

paulinus
  • 336
  • 2
  • 4
  • 3
    This seems like a neat solution but causes an error in `distutils/_msvccompiler.py` on Windows builds, where I needed it :( – Cas May 22 '19 at 20:35
  • @Cas also add --skip-build option to bdist_* command at the setup.py call! – bartoli Aug 04 '22 at 15:01
  • 1
    This doesn't work for me - even if passing an empty list for `sources`, setuptools tries to launch gcc. – mara004 Aug 18 '22 at 21:34
  • This looked like the most elegant solution to me, but it then changes my python version from py3 to cpy38, any idea why or how to prevent it? – agirault Nov 22 '22 at 18:26
11

Neither the root_is_pure trick nor the empty ext_modules trick worked for me, but after MUCH searching myself, I finally found a working solution in 'pip setup.py bdist_wheel' no longer builds forced non-pure wheels

Basically, you override the 'has_ext_modules' function in the Distribution class, and set distclass to point to the overriding class. At that point, setup.py will believe you have a binary distribution, and will create a wheel with the specific version of python, the ABI, and the current architecture. As suggested by https://stackoverflow.com/users/5316090/py-j:

from setuptools import setup
from setuptools.dist import Distribution

DISTNAME = "packagename"
DESCRIPTION = ""
MAINTAINER = ""
MAINTAINER_EMAIL = ""
URL = ""
LICENSE = ""
DOWNLOAD_URL = ""
VERSION = '1.2'
PYTHON_VERSION = (2, 7)


# Tested with wheel v0.29.0
class BinaryDistribution(Distribution):
    """Distribution which always forces a binary package with platform name"""
    def has_ext_modules(foo):
        return True


setup(name=DISTNAME,
      description=DESCRIPTION,
      maintainer=MAINTAINER,
      maintainer_email=MAINTAINER_EMAIL,
      url=URL,
      license=LICENSE,
      download_url=DOWNLOAD_URL,
      version=VERSION,
      packages=["packagename"],

      # Include pre-compiled extension
      package_data={"packagename": ["_precompiled_extension.pyd"]},
      distclass=BinaryDistribution)
Lucian
  • 461
  • 5
  • 6
9

You can also specify/spoof a specific platform name when building wheels by specifying a --plat-name:

python setup.py bdist_wheel --plat-name=manylinux1_x86_64
Robin De Schepper
  • 4,942
  • 4
  • 35
  • 56
  • 5
    what are all possible platform names ? – 0xB00B Apr 14 '21 at 08:51
  • I have trouble finding an exact list for you. Technically you can give any argument you want, I think it's PyPI that chooses which tags it accepts when you upload it. Maybe https://packaging.python.org/specifications/platform-compatibility-tags/ can help you get started on finding an actual list? – Robin De Schepper May 17 '21 at 16:04
2

I find Anthony Sottile's answer great, but didn't work for me.

My case is that I have to enforce the wheel to be created for x86_64, but any python3, so making root impure actually caused my wheel to be py36-cp36 :(

A better way, IMO, in general is just to use sys.argv:

from setuptools import setup
import sys

sys.argv.extend(['--plat-name', 'x86_64'])
setup(name='example-wheel')
CIsForCookies
  • 12,097
  • 11
  • 59
  • 124