31

Suppose I have a native shared library (.dll or .so), built independently of any Python mechanism, and a Python module using ctypes to interface to the library. Is there a way that I can build those into a .whl package? If so, how?

Assuming this is possible, I think I'd need the wheel package installed and to use python setup.py bdist_wheel but what would my setup.py need to look like?

I'd like to do this so I can upload Wheels for various platforms into a private package index and be able to pip install the appropriate one for the platform I'm on.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
Eric Smith
  • 2,739
  • 2
  • 30
  • 29

3 Answers3

46

Here is a way. For an example, this uses libeay32.dll to expose an md5 package.

The project structure is:

MD5
│   setup.py
│
└───md5
    __init__.py   
    libeay32.dll

The setup.py is:

from setuptools import setup, Distribution
 
 
class BinaryDistribution(Distribution):
    def has_ext_modules(foo):
        return True


setup(
    name='md5',
    version='1.0',
    description='MD5 Library',
    packages=['md5'],
    package_data={
        'md5': ['libeay32.dll'],
    },
    distclass=BinaryDistribution
)

A couple of things to note:

  1. The DLL is listed as package data so that it will be included in the wheel.
  2. A custom distclass is used that indicates this wheel has an extension module, and since the wheel is being built on Windows, that this is a win32 wheel.

The Python ctypes code can load the DLL relative to itself (this code is in __init.py__):

lib_path = os.path.join(os.path.dirname(__file__), 'libeay32.dll')
lib = CDLL(lib_path)

After having installed 'wheel' with pip, I can run python setup.py bdist_wheel to produce dist\md5-1.0-cp34-none-win32.whl. I happen to be using cpython 3.4, but if you want a universal wheel, you can add options={"bdist_wheel": {"universal": True}} into the setup call.

Now I can create and activate a new virtual environment, pip install md5-1.0-cp34-none-win32.whl, and use my package:

>>> import md5
>>> md5.digest('hello')
'8d11aa0625ce42cfe9429d5e93b5ab0a'
wim
  • 338,267
  • 99
  • 616
  • 750
Eric Smith
  • 2,739
  • 2
  • 30
  • 29
  • 3
    Does this work on Linux, where LD_LIBRARY_PATH matters to see and load .so file? – Guillaume Jacquenot Mar 12 '18 at 08:13
  • 2
    You can also pass `has_ext_modules` as an argument to `setup()`. It still expects a callable. Add `has_ext_modules=lambda : True` as a `setup()` argument and you can skip `distclass`. – pullmyteeth May 14 '20 at 13:56
  • The https://github.com/pypa/sampleproject/blob/master/setup.cfg which was referenced no longer builds a universal wheel. I've updated with a working alternative. – wim Mar 29 '22 at 23:10
11

Although the chosen answer was correct at the time of writing, this pull request broke this functionality.

I think this answer correctly solves this problem in newer versions of the wheel package.

(Would have added this as a comment, but lack the reputation needed.)

Thomas D
  • 167
  • 2
  • 8
  • Thanks for the heads up -- I've updated the answer to use `has_ext_modules` rather than `is_pure`. – Eric Smith Nov 04 '16 at 02:19
  • The link to the pull request is dead. I think it may have been [this commit](https://github.com/pypa/wheel/commit/9fc2c53cfe6a14a78c09a0c8863de2d8ddaadcf8) on github, but I'm not 100% sure. – wim Mar 29 '22 at 23:07
1

Use cibuildwheel, this is the solution from pypa, the team behind pip.

cibuildwheel can build wheels with native libraries, and it automatically fix issues like LD_LIBRARY_PATH and incorporate the dll/so files into the bdist wheel.

ospider
  • 9,334
  • 3
  • 46
  • 46