How to avoid building C library with my python package?

Question

I'm building a python package using a C library with ctypes. I want to make my package portable (Windows, Mac and Linux).

I found a strategy, using build_ext with pip to build the library during the installation of my package. It creates libfoo.dll or libfoo.dylib or libfoo.so depending on the target's platform.

The problem with this is that my user needs CMake installed.

Does exist another strategy to avoid building during the installation? Do I have to bundle built libraries in my package?

I want to keep my users doing pip install mylib.

Edit: thank to @Dawid comment, I'm trying to make a python wheel with the command python setup.py bdist_wheel without any success.

How can I create my python wheel for different platform with the embedded library ?

Edit 2: I'm using python 3.4 and working on Mac OS X, but I have access to Windows computer, and Linux computer

Answer 1

You're certainly heading down the right path according to my research... As Daniel says, the only option you have is to build and distribute the binaries yourself.

In general, the recommended way to install packages is covered well in the packaging user guide. I won't repeat advice there as you have clearly already found it. However the key point in there is that the Python community, specifically PyPA are trying to standardize on using platform wheels to package binary extensions. Sadly, there are a few issues at this point:

1. You cannot create distributions for all Linux variants, but you can build wheels for a compatible subset. See https://www.python.org/dev/peps/pep-0513/ for details.
2. The advice on building extensions is somewhat incomplete, reflecting the lack of a complete solution for binary distributions.
3. People then try to build their own library and distribute it as a data file, which confuses setuptools.

I think you are hitting this last issue. A workaround is to force the Distribution to build a platform wheel by overriding is_pure() to always return False. However you could just keep your original build instructions and bdist_wheel should handle it.

Once you've built the wheel, though, you still need to distribute it and maybe other binary packages that it uses or use it. At this point, you probably need to use one of the recommended tools like conda or a PyPI proxy like devpi to serve up your wheels.

EDIT: To answer the extra question about cross-compiling

As covered here Python 2.6 and later allows cross-compilation for Windows 32/64-bit builds. There is no formal support for other packages on other platforms and people have had limited success trying to do it. You are really best off building natively on each of your Linux/Mac/Windows environments.

Answer 2

@rth and @PeterBrittain help me a lot. Here the solution I use:

structure folder :

setup.py
python_package/
    lib/
        libfoo.dylib
        libfoo.dll
    __init__.py
    main.py

setup.py :

from setuptools import setup, dist


class BinaryDistribution(dist.Distribution):
    def is_pure(self):
        return False


setup(
    name='python_package',
    package_data={'python_package': ['lib/libfoo.dylib','lib/libfoo.dll']},
    include_package_data=True,
    distclass=BinaryDistribution,
    packages=['python_package'],
)

main.py :

#!/usr/bin/env python
import platform
from ctypes import CDLL, c_char_p

import pkg_resources

sysname = platform.system()

if sysname == 'Darwin':
    lib_name = "libfoo.dylib"
elif sysname == 'Windows':
    lib_name = "libfoo.dll"
else:
    lib_name = "libfoo.so"
lib_path = pkg_resources.resource_filename('python_package', 'lib/{}'.format(lib_name))
foo = CDLL(lib_path)

bar = foo.bar
bar.restype = c_char_p
bar.argtypes = [c_char_p]

print(bar('hello'))

build wheel :

python setup.py bdist_wheel

It creates a specific plateform wheel rtfdoc-0.0.1-cp34-cp34m-macosx_10_10_x86_64.whl and mac users can do a simple pip install rtfdoc-0.0.1-cp34-cp34m-macosx_10_10_x86_64.whl

This solution is not entirely satisfactory:

• I think I will still need to create a Extension or look at SCons for linux users.
• Update my package will be difficult because of the production of the lib
• I don't know how to manage .dll for 32 bit and 64 bit

Thank you, I learned a lot, and I understand why setup.py in Pillow or Psycopg2 are huge

Answer 3

You can use cibuildwheel to build wheels on Travis CI and/or Appveyor for all kinds of platforms and Python versions. This tool can also deploy your wheels on PyPI or elsewhere.

Answer 4

I prefer my user to install cygwin/mingw32 or cmake and do a pip install mylib [...]. I just want to go further and try to avoid the CMake installation.

As to packaging pre-compiled python modules, see @Peter Brittain's complete answer.

Now, assuming that the user actually has a C compiler installed (whether though cygwin, conda on Windows, or the system one on Linux), and all you want is to avoid the CMake installation, that has no relationship Python packaging.

The question is then how much functionality of CMake you are using and whether the same thing can be accomplished with easier to manage alternatives, see related questions (1), (2) , etc.

Edit: In particular, I was thinking something along the lines of SCons that provides a full build system, but written in Python so it is easier to install in a Python friendly environment than CMake, see full comparison here. A wild guess (I have never used it myself), but since it is a pure python module, you could probably set SCons as a dependency in your setup.py, and fully automate the build of your C code there, so that pip install mylib does everything that's needed. See also, CMake2SCons package that could be useful.

Answer 5

If you don't want the user to build during installation, you have to provide a pre-built binary package, I don't think any other strategy is possible. I've heard good things about py2exe, but I don't use windows so IDK. I've had good experiences with miniconda, but it does require a little extra work to build binary packages from pypi.