How to use libraries within my CMake project that need to be installed first?

Question

I have a problem with my CMake build system. There are CMakeLists.txt files defining runtimes or libraries or using ExternalProjects_Add() to download and build external code. Because of dependencies, those projects have to find each other. Now I want to have a CMakeLists.txt at the top level that builds all those at once. In order to find a project, is must be installed. But finding projects is already done at configuration time in CMake.

repository
├─project
│ ├─game (Depends on engine, uses EngineConfig.cmake once installed)
│ │ ├─CMakeLists.txt
│ │ ├─include
│ │ ├─src
│ │ └─textures
│ ├─engine (Depends on boost, uses built-in FindBoost.cmake)
│ │ ├─CMakeLists.txt
│ │ ├─include
│ │ └─src
│ ├─boost (Not the source code, just an ExternalProject_Add call)
│ : └─CMakeLists.txt
│
├─build
│ ├─game
│ ├─engine
│ ├─boost (Source will be downloaded and built here)
│ : ├─download
│   ├─source
│   :
│
├─install
│ ├─game
│ │ ├─bin
│ │ └─textures
│ ├─engine
│ │ ├─include
│ │ │ └─engine
│ │ │   ├─EngineConfig.cmake (Needed to find the library)
│ │ │   :
│ │ │
│ │ └─lib
│ ├─boost (Layout is up to the external library)
│ : └─ ...
│
└─CMakeLists.txt (Calls add_subdirectory for all inside the project folder)

Run a CMake process for every project: Using execute_process(${CMAKE_COMMAND} ...), I can configure and build each project after another at configure time. However, this means I always have to run CMake after editing the code and cannot compile from within the IDE I generated project files for.

Linking to CMake targets: Running a CMake process for all external libraries is okay since I don't work on them. My own libraries could be used by calling target_link_libraries() with their target names. However, linking isn't enough. My libraries include directories of external libraries. Those must be made available to the using project, as well.

How can I use libraries within my CMake project that need to be installed first?

Answer 1

You can classify your projects into three groups:

1. External dependencies you are not working on in this super-project
2. Projects you're working on but are too complex to add them as a subdirectory, for example having too many targets or other reasons. (You don't seem to have such a project in your example.)
3. Projects you're working on: these will be added as a subdirectory of the super-project.

You need to configure, build and install the project in groups #1 and #2 before configuring the super-project:

• You can do it before running the super-project's CMakeLists.txt, for example, from a shell-script
• Or, as you mentioned, from within the super-project's CMakeLists.txt, using execute_process(${CMAKE_COMMAND} ...). You can do it conditionally using the result of an appropriate find_package(... QUIET) command.

You need to decide if projects in group #3, like engine will be used solely in projects that uses them as subdirectories or you intend to use them as standalone libraries, built in their own build trees.

Also, you mentioned that: "My libraries include directories of external libraries". Let's cover all such possible libraries the engine can be dependent on:

• say, LIB1 and LIB2 are private and public external dependencies of engine and their config-modules export old-school LIB1_* and LIB2_* variables
• LIB3 and LIB4 are private and public external dependencies of engine and their config-modules export the LIB3 and LIB4 imported libraries

By public and private dependencies I mean whether the particular library is used or not used on the interface of engine.

Now, if engine is to be used only as a subdirectory then the relevant section of engine/CMakeLists.txt is:

add_library(engine ...)
target_include_directories(engine
    PRIVATE
        ${LIB1_INCLUDE_DIRS}
    PUBLIC
        $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
        ${LIB2_INCLUDE_DIRS})
target_compiled_definitions(engine
    PRIVATE ${LIB1_DEFINITIONS}
    PUBLIC ${LIB2_DEFINITIONS})
target_link_libraries(engine
    PRIVATE LIB3
    PUBLIC LIB4)

in repository/CMakeLists.txt:

add_subdirectory(engine)
add_subdirectory(game)

in game/CMakeLists.txt:

add_executable(game ...)
target_link_libraries(game engine)

The include dirs of both of the engine and of its public dependencies will be correctly forwarded to game.

If engine will also be built in its own build tree (in another project) you need to add the exporting code to engine/CMakeLists.txt and maybe a custom config-module that calls find_package (or find_dependency) for its dependencies. See How to use CMake to find and link to a library using install-export and find_package? for details. One issue not discussed in that answer is finding the dependencies of a library in the library's config module:

The referenced SO answer simply installs the <lib>-targets.cmake script, generated by the install(EXPORT ...) command, as the config-module:

install(EXPORT engine-targets
    FILE engine-config.cmake
    DESTINATION lib/cmake/engine)

This solution is fine when engine has no further dependencies. If it does, they need to be found at the beginning of the config module, which should be written manually.

engine/engine-config.cmake:

include(CMakeFindDependencyMacro)
find_dependency(some-dep-of-engine)
include(${CMAKE_CURRENT_LIST_DIR}/engine-targets.cmake)

and in engine/CMakeLists.txt:

install(EXPORT engine-targets
    FILE engine-targets.cmake
    DESTINATION lib/cmake/engine)
install(FILES engine-config.cmake
    DESTINATION lib/cmake/engine)

Note: The CMakeFindDependencyMacro has been introduced in CMake 3.0. With an older CMake you can use find_package instead of find_dependency (handling of QUIET and REQUIRED options will not be forwarded to the dependency).

Answer 2

When export library from engine project you need to specify its include directories. Code below is a simplification of example provided at http://www.cmake.org/cmake/help/v3.0/manual/cmake-packages.7.html#creating-packages. Paths are adjasted for use installation prefix install/engine for build and install engine component.

engine/CMakeLists.txt:

...
install(TARGETS engine EXPORT engineTargets
    DESTINATION lib
    INCLUDES DESTINATION include
)

set(ConfigPackageLocation lib/cmake/engine)

install(EXPORT engineTargets
    FILE EngineTargets.cmake
    DESTINATION ${ConfigPackageLocation}
)

install(FILES cmake/EngineConfig.cmake
    DESTINATION ${ConfigPackageLocation}
)

engine/cmake/EngineConfig.cmake:

include("${CMAKE_CURRENT_LIST_DIR}/EngineTargets.cmake")

This provides interface of the exported target. So when it will be linked by executable, the executable gets proper INCLUDE_DIRECTORIES property:

CMakeLists.txt:

# Need for `find_package` to find `EngineConfig.cmake`.
set(CMAKE_PREFIX_PATH <path-pointed-to-install/engine>)

game/CMakeLists.txt:

find_package(Engine)
add_executable(game ...)
target_link_libraries(game engine)

Answer 3

Thanks @Tsyvarev and @tamas.kenez you for the two good answers. I ended up using the super-build pattern. The top-level project doesn't do much at configure time. At build time, it runs external CMake processes to configure, build and install the projects.

Usually, this is implemented using ExternalProject_Add() instead of add_subdirectory() to add the projects. I found add_custom_command() to work better since it doesn't do additional tasks in the background like creating stamp files and so on.

# add_project(<project> [DEPENDS project...])
function(add_project PROJECT)
    cmake_parse_arguments(PARAM "" "" "DEPENDS" ${ARGN})
    add_custom_target(${PROJECT} ALL DEPENDS ${PARAM_DEPENDS})
    # Paths for this project
    set(SOURCE_DIR  ${CMAKE_CURRENT_SOURCE_DIR}/${PROJECT})
    set(BUILD_DIR   ${CMAKE_CURRENT_BINARY_DIR}/${PROJECT})
    set(INSTALL_DIR ${CMAKE_INSTALL_PREFIX}/${PROJECT})
    # Configure
    escape_list(CMAKE_MODULE_PATH)
    escape_list(CMAKE_PREFIX_PATH)
    add_custom_command(TARGET ${TARGET}
        COMMAND ${CMAKE_COMMAND}
            --no-warn-unused-cli
            "-DCMAKE_MODULE_PATH=${CMAKE_MODULE_PATH_ESCAPED}"
            "-DCMAKE_PREFIX_PATH=${CMAKE_PREFIX_PATH_ESCAPED}"
            -DCMAKE_BINARY_DIR=${BUILD_DIR}
            -DCMAKE_INSTALL_PREFIX=${INSTALL_DIR}
            -DCMAKE_CXX_FLAGS=${CMAKE_CXX_FLAGS}
            ${SOURCE_DIR}
        WORKING_DIRECTORY ${BUILD_DIR})
    # Build
    add_custom_command(TARGET ${TARGET}
        COMMAND ${CMAKE_COMMAND}
            --build .
            --target install
        WORKING_DIRECTORY ${BUILD_DIR})
    # Help later find_package() calls
    append_global(CMAKE_PREFIX_PATH ${INSTALL_DIR})
endfunction()

Here are the two helper functions. It took me quite some time to figure out the right way to pass list parameters to other CMake processes without them being interpreted and passes as multiple parameters.

# escape_list(<list-name>)
function(escape_list LIST_NAME)
    string(REPLACE ";" "\;" ${LIST_NAME}_ESCAPED "${${LIST_NAME}}")
    set(${LIST_NAME}_ESCAPED "${${LIST_NAME}_ESCAPED}" PARENT_SCOPE)
endfunction()

# append_global(<name> value...)
function(append_global NAME)
    set(COMBINED "${${NAME}}" "${ARGN}")
    list(REMOVE_DUPLICATES COMBINED)
    set(${NAME} "${COMBINED}" CACHE INTERNAL "" FORCE)
endfunction()

The only downside is that every project needs to have an install target for this. So you need to add a dummy install command like install(CODE "") to projects that have no install command otherwise, e.g. those who just call ExternalProject_Add.