Python 2.7 C++ extensions with CMake and SetupTools?

Question

Overview

I have a C++ project that provides a Python 2.7 wrapper. I'm building that project using CMake and I would like to have a setup.py file so that I can easily pip install git+https://mygitrepo/my_project this project into a virtualenv. I have successfully gotten to the point where I can get pip to build my project, but I have no idea how to specify in my setup.py file or my CMakeLists.txt where my binaries should be installed.

Details

When I build my project with CMake (e.g. mkdir /path/to/my_project/build && cd /path/to/my_project/build && cmake .. && make) everything gets built correctly and I end up with a libmy_project.so and a pymy_project.so file that I can succesfully import and use in Python.

When I build the project with pip in a virtualenv (e.g. pip install /path/to/my_project -v) I can watch CMake and my compiler cranking away to compile things as expected. However they're built in a temp directory and immediately thrown away. How do I tell my setup.py which files to keep, and where will it put them?

Relevant files

My CMakeLists.txt looks like the following:

cmake_minimum_required (VERSION 3.0.0)
project (my_project)

# Find SomePackage
find_package(SomePackage REQUIRED COMPONENTS cool_unit great_unit)
include_directories(${SomePackage_INCLUDE_DIR})

# Find Python
find_package(PythonLibs 2.7 REQUIRED)
include_directories(${PYTHON_INCLUDE_DIR})

# Build libmy_project
add_library(my_project SHARED src/MyProject.cpp)
target_link_libraries(my_project ${SomePackage_LIBRARIES} ${PYTHON_LIBRARIES})

# Build py_my_project
add_library(py_my_project SHARED src/python/pyMyProject.cpp)
set_target_properties(py_my_project PROPERTIES PREFIX "")
target_link_libraries(py_my_project ${SomePackage_LIBRARIES} 
                      ${PYTHON_LIBRARIES} my_project)

file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/__init__.py "__all__ = ['py_my_project']")

and my setup.py file looks like this:

import os
import subprocess

from setuptools import setup, Extension
from setuptools.command.build_ext import build_ext

class CMakeExtension(Extension):
    def __init__(self, name, sourcedir=''):
        Extension.__init__(self, name, sources=[])
        self.sourcedir = os.path.abspath(sourcedir)

class CMakeBuild(build_ext):
    def run(self):
        for ext in self.extensions:
            self.build_extension(ext)

    def build_extension(self, ext):
        if not os.path.exists(self.build_temp):
            os.makedirs(self.build_temp)
        subprocess.check_call(['cmake', ext.sourcedir], cwd=self.build_temp)
        subprocess.check_call(['cmake', '--build', '.'], cwd=self.build_temp)

setup(
    name='my_project',
    version='0.0.1',
    author='Me',
    author_email='rcv@stackoverflow.com',
    description='A really cool and easy to install library',
    long_description='',
    ext_modules=[CMakeExtension('.')],
    cmdclass=dict(build_ext=CMakeBuild),
    zip_safe=False,
)

[This question](https://stackoverflow.com/questions/42585210/extending-setuptools-extension-to-use-cmake-in-setup-py) seems like it's for exactly what you're trying to do. If it's incomplete or wrong or out of date, I can look for other ones; if it solves your problem, we can close this as a dup. — abarnert, Jul 20 '18 at 22:49
Your problem is that you're not calling `super().run()` in the `CMakeBuild` class. You're calling the CMake build ok, but the whole rest of the extension packaging process is skipped. — hoefling, Jul 21 '18 at 06:44
I'm using Python 2.7, and it looks like I can't call `super(CMakeBuild, self)` because `build_ext` (and it's parent `Command`) do not inherit from `object`. What does the `super.run` accomplish that I need? I've gotten very close by adding a `cmake_args = ['-DCMAKE_INSTALL_PREFIX={}'.format(os.path.abspath(os.path.dirname(extdir)))]`. Everything installs just fine now, but now the linux dynamic linker can't find my libmy_project.so. — rcv, Jul 21 '18 at 21:42
Added my solution as an answer below, as there are some slight differences between it and the linked answer. — rcv, Jul 21 '18 at 22:24
On Pytgon 2, add the `object` class to the inherited classes: `class MyCommand(build_ext, object):`. Related: [super() raises "TypeError: must be type, not classobj" for new-style class](https://stackoverflow.com/questions/9698614/super-raises-typeerror-must-be-type-not-classobj-for-new-style-class) — hoefling, Jul 22 '18 at 08:26
Also, I guess you've forgotten some code in your setup script as right now it's identical to the one in linked answer. — hoefling, Jul 22 '18 at 09:16
If you need to call a base class initializer or other method from an old-style class, you have to call it explicitly as an unbound method of th base class, like `build_ext.__init__(self, … args …)`. This does basically the same thing that `super().__init__(… args …)` does in simple single-inheritance chains. — abarnert, Jul 23 '18 at 01:57
Anyway, it looks like your answer is just a trivial backport of the linked answer to Python 2, with less information. So, I think hoefling is right that this is a dup. I doubt there are many people out there that can build C++ Python extensions with cmake and are still on 2.7 a year and half from end-of-life but who have no clue how to backport the code to 2.x. If you disagree, then we need to improve that question, which would probably best be done by you either adding a supplemental answer, or editing the existing one, to explain what needs to be done to make it work for 2.x. — abarnert, Jul 23 '18 at 02:01

score 0 · Accepted Answer · edited Jun 20 '20 at 09:12

The answer to "How do I tell my setup.py which files to keep, and where will it put them?" seems to be self.get_ext_fullpath(ext.name) as is found in this solution as pointed out by abarnert and hoefling. This method returns a temporary root directory into which you can put your build output. Pip then appears to scan that directory and copy everything into your real environment after the build is done.

Unfortunately, the Linux dynamic linker can't find my dependency (libmy_project.so) for my extension, as virtualenv doesn't set LD_LIBRARY_PATH or anything, so I ended up just rolling all of the compilation units into my extension. The full solution is below:

setup.py

import os
import subprocess

from setuptools import setup, Extension
from setuptools.command.build_ext import build_ext

class CMakeExtension(Extension):
    def __init__(self, name, sourcedir=''):
        Extension.__init__(self, name, sources=[])
        self.sourcedir = os.path.abspath(sourcedir)

class CMakeBuild(build_ext):
    def run(self):
        for ext in self.extensions:
            self.build_extension(ext)

    def build_extension(self, ext):
        if not os.path.exists(self.build_temp):
            os.makedirs(self.build_temp)

        extdir = self.get_ext_fullpath(ext.name)
        if not os.path.exists(extdir):
            os.makedirs(extdir)

        # This is the temp directory where your build output should go
        install_prefix = os.path.abspath(os.path.dirname(extdir))
        cmake_args = ['-DCMAKE_LIBRARY_OUTPUT_DIRECTORY={}'.format(install_prefix)]

        subprocess.check_call(['cmake', ext.sourcedir, cmake_args], cwd=self.build_temp)
        subprocess.check_call(['cmake', '--build', '.'], cwd=self.build_temp)

setup(
    name='my_project',
    version='0.0.1',
    author='Me',
    author_email='rcv@stackoverflow.com',
    description='A really cool and easy to install library',
    long_description='',
    ext_modules=[CMakeExtension('.')],
    cmdclass=dict(build_ext=CMakeBuild),
    zip_safe=False,
    install_requires=[
        'cmake',
    ]
)

CMakeLists.txt

cmake_minimum_required (VERSION 3.0.0)
project (my_project)

# Find SomePackage
find_package(SomePackage REQUIRED COMPONENTS cool_unit great_unit)
include_directories(${SomePackage_INCLUDE_DIR})

# Find Python
find_package(PythonLibs 2.7 REQUIRED)
include_directories(${PYTHON_INCLUDE_DIR})

# Build py_my_project
add_library(py_my_project SHARED src/python/pyMyProject.cpp src/MyProject.cpp)
set_target_properties(py_my_project PROPERTIES PREFIX "")
target_link_libraries(py_my_project ${SomePackage_LIBRARIES} 
                      ${PYTHON_LIBRARIES})

file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/__init__.py "__all__ = ['py_my_project']")