Creating a python package (deb/rpm) from cmake

Question

I am trying to create a python package (deb & rpm) from cmake, ideally using cpack. I did read

The installation works just fine (using component install) for my shared library. However I cannot make sense of the documentation to install the python binding (glue) code. Using the standard cmake install mechanism, I tried:

install(
  FILES __init__.py library.py
  DESTINATION ${ACME_PYTHON_PACKAGE_DIR}/project_name
  COMPONENT python)

And then using brute-force approach ended-up with:

# debian based package (relative path)
set(ACME_PYTHON_PACKAGE_DIR lib/python3/dist-packages)

and

# rpm based package (full path required)
set(ACME_PYTHON_PACKAGE_DIR /var/lang/lib/python3.8/site-packages)

The above is derived from:

debian % python -c 'import site; print(site.getsitepackages())'
['/usr/local/lib/python3.9/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.9/dist-packages']

while:

rpm % python -c 'import site; print(site.getsitepackages())'
['/var/lang/lib/python3.8/site-packages']

It is pretty clear that the brute-force approach will not be portable, and is doomed to fail on the next release of python. The only possible solution that I can think of is generating a temporary setup.py python script (using setuptools), that will do the install. Typically cmake would call the following process:

% python setup.py install --root ${ACME_PYTHON_INSTALL_ROOT}

My questions are:

Did I understand the cmake/cpack documentation correctly for python package ? If so this means I need to generate an intermediate setup.py script.
I have been searching through the cmake/cpack codebase (git grep setuptools) but did not find helper functions to handle generation of setup.py and passing the result files back to cpack. Is there an existing cmake module which I could re-use ?

I did read, some alternative solution, such as:

How to build debian package with CPack to execute setup.py?

Which seems overly complex, and geared toward Debian-only based system. I need to handle RPM in my case.

Just thinking of directly creating a whl package like this https://github.com/Mizux/python-native/ that you can uploat to pypi.org etc no rpm involved. otherwise you may be interested in https://github.com/google/or-tools/blob/stable/cmake/python-install.cmake.in — Mizux, Nov 19 '21 at 07:58
@Mizux For your suggestion #1, I cannot upload company code to pypi.org. As for suggestion #2, this closely resemble my own answer. However it does not deal with the complexity associated with absolute path cmake `install()` commands. — malat, Nov 19 '21 at 12:22

malat · Accepted Answer · 2021-11-19T12:49:42.327

As mentionned in my other solution, the ugly part is dealing with absolute path in cmake install() commands. I was able to refactor the code to avoid usage of absolute path in install(). I simply changed the installation into:

install(
  # trailing slash is important:
  DIRECTORY ${SETUP_OUTPUT}/
  # "." syntax is a reliable mechanism, see:
  # https://gitlab.kitware.com/cmake/cmake/-/issues/22616
  DESTINATION "."
  COMPONENT python)

And then one simply needs to:

set(CMAKE_INSTALL_PREFIX "/")
set(CPACK_PACKAGING_INSTALL_PREFIX "/")
include(CPack)

At this point all install path now need to include explicitely /usr since we've cleared the value for CMAKE_INSTALL_PREFIX.

The above has been tested for deb and rpm packages. CPACK_BINARY_TGZ does properly run with the above solution:

https://gitlab.kitware.com/cmake/cmake/-/issues/22925

malat · Answer 2 · 2022-01-11T16:19:18.580

I am going to post the temporary solution I am using at the moment, until someone provide something more robust.

So I eventually manage to stumble upon:

Re-using the above to do an install step instead of a build step can be done as follow:

find_package(Python COMPONENTS Interpreter)

set(SETUP_PY_IN "${CMAKE_CURRENT_SOURCE_DIR}/setup.py.in")
set(SETUP_PY "${CMAKE_CURRENT_BINARY_DIR}/setup.py")
set(SETUP_DEPS "${CMAKE_CURRENT_SOURCE_DIR}/project_name/__init__.py")
set(SETUP_OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/build-python")

configure_file(${SETUP_PY_IN} ${SETUP_PY})

add_custom_command(
  OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/setup_timestamp
  COMMAND ${Python_EXECUTABLE} ARGS ${SETUP_PY} install --root ${SETUP_OUTPUT}
  COMMAND ${CMAKE_COMMAND} -E touch ${CMAKE_CURRENT_BINARY_DIR}/setup_timestamp
  DEPENDS ${SETUP_DEPS})

add_custom_target(target ALL DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/setup_timestamp)

And then the ugly part is:

install(
  # trailing slash is important:
  DIRECTORY ${SETUP_OUTPUT}/
  DESTINATION "/" # FIXME may cause issues with other cpack generators
  COMPONENT python)

Turns out that the documentation for install() is pretty clear about absolute paths:

https://cmake.org/cmake/help/latest/command/install.html#introduction

DESTINATION
[...]
As absolute paths are not supported by cpack installer generators,
it is preferable to use relative paths throughout.

For reference, here is my setup.py.in:

from setuptools import setup

if __name__ == '__main__':
    setup(name='project_name_python',
          version='${PROJECT_VERSION}',
          package_dir={'': '${CMAKE_CURRENT_SOURCE_DIR}'},
          packages=['project_name'])

You can be fancy and remove the __pycache__ folder using the -B flag:

COMMAND ${Python_EXECUTABLE} ARGS -B ${SETUP_PY} install --root ${SETUP_OUTPUT}

You can be extra fancy and add debian option such as:

if(CPACK_BINARY_DEB)
  set(EXTRA_ARG "--install-layout" "deb")
endif()

use as:

COMMAND ${Python_EXECUTABLE} ARGS -B ${SETUP_PY} install --root ${SETUP_OUTPUT} ${EXTRA_ARG}

Creating a python package (deb/rpm) from cmake

2 Answers2