Compile C library on pip install

Question

I made a Python application which needs to execute C functions. To do so, I compiled the C functions in a shared library using gcc and called the library in my Python script using ctypes.

I'm trying to pack my application in a pip package but found no way to create the shared library upon pip install.

I tried the following (setup.py) :

from setuptools import setup
from setuptools.command.install import install
import subprocess

class compileLibrary(install):
    def run(self):
        install.run(self)
        command = "cd packageName"
        command += " && git clone https://mygit.com/myAwesomeCLibrary.git"
        command += " && gcc -my -many -options"
    process = subprocess.Popen(command, shell=True)
    process.wait()

setup(
    name='packageName',
    version='0.1',
    packages=['packageName'],
    install_requires=[
        ...
    ],
cmdclass={'install': compileLibrary},

)

This works when doing python setup.py install but, when installing from pip, while the install process is successful, the shared library is not in the package folder (pythonx.x/site-packages/packageName).

I tried using distutils with the same result.

Based on the question Run Makefile on pip install I also tried the following setup.py :

from setuptools import setup
from distutils.command.build import build
import subprocess


class Build(build):
    def run(self):
        command = "cd packageName"
        command += " && git clone https://mygit.com/myAwesomeCLibrary.git"
        command += " && gcc -my -many -options"
        process = subprocess.Popen(command, shell=True)
        process.wait()
        build.run(self)


setup(
    name='packageName',
    version='0.1',
    packages=['packageName'],
    install_requires=[
        ...
    ],
    cmdclass={'build': Build},
)

Here is the architecture of my package

packageName/
├── packageName/
│   ├── __init__.py
│   ├── myFunctions.c
├── MANIFEST.in
├── README.md
├── setup.py

Note : After installing the package with pip, I only have __init__.py and __pycache__ in the install folder (pythonx.x/site-packages/packageName).

How can I create the shared library during pip install so It can be used by my package ?

score 8 · Accepted Answer · answered Nov 18 '17 at 21:06

After hours of research, I found the solution. The main problems were :

Use Extension from setuptools to compile the library
Use a custom install function to git clone and calling the constructor at the end so the git clone occurs before the compilation.

Here is the working setup.py :

from setuptools import setup, Extension
from setuptools.command.install import install
import subprocess
import os

class CustomInstall(install):
    def run(self):
        command = "git clone https://mygit.com/myAwesomeCLibrary.git"
        process = subprocess.Popen(command, shell=True, cwd="packageName")
        process.wait()
        install.run(self)

module = Extension('packageName.library',
                   sources = ['packageName/library.c'],
                   include_dirs = ['packageName/include'],
                   extra_compile_args=['-fPIC'])

setup(
    name='packageName',
    version='0.1',
    packages=['packageName'],
    install_requires=[
        ...
    ],
    cmdclass={'install': CustomInstall},
    include_package_data=True,
    ext_modules=[module],
)

When trying to use this with `python -m build`, I'm finding that this errors with a "file not found" when trying to compile the C files. After some experimentation, it seems the `CustomInstall` section runs *after* the module compilation instead of before (which obviously doesn't work). — Mala, Aug 11 '22 at 19:18

score -1 · Answer 2 · answered Dec 04 '17 at 06:53

-1

you can add the shared library in MANIFEST.in. like this: include *.so *.dylib

answered Dec 04 '17 at 06:53

Hao Zheng

7
1

1

Thank you but I want to compile the library during the pip install. – M-Gregoire Dec 05 '17 at 07:32
1

So you want to using the CustomInstall not the CustomBuild. – Hao Zheng Dec 06 '17 at 09:27

Compile C library on pip install

2 Answers2

Linked