Use find_package() on external project

Question

I have an External project called messages. I am using ExternalProject_Add in order to fetch and build the project.

If i use find_package(messages REQUIRED) in top level CMakeLists.txt the cmake .. fails because it couldn't find the package installation files, which is logical as they are only build during make command invocation.

I am not sure, if there is way use find_package() on ExternalProjects. If so, please point me to an example.

Thanks BhanuKiran

Answer 1

You have misunderstood how ExternalProject is supposed to work. You cannot find_package(messages REQUIRED) because it hasn't been built yet. ExternalProject merely creates the build steps necessary to build the subproject.

You have two options:

1. Use add_subdirectory or FetchContent in place of ExternalProject. In this case, you don't need a find_package call. This effectively adds the sub-project to the main build and imports the subproject's targets.
2. Use two ExternalProject calls: one for messages and another for main_project, which depends on messages. If messages uses the export(EXPORT) function, you can point CMAKE_PREFIX_PATH or messages_ROOT to the build directory. Otherwise you'll need to run the install step for messages and set up an install prefix inside your build directory. Then the find_project(messages REQUIRED) call inside main_project will succeed. This will likely require re-structuring your build.

Generally speaking, ExternalProject is only useful for defining super-builds, which are chains of CMake builds that depend on one another. And super builds are only useful when you need completely different configure-time options, like different toolchains (eg. you're cross compiling, but need a code generator to run on the build machine). If that's not the case, prefer FetchContent or add_subdirectory with a git submodule.

It is best to use FetchContent with CMake 3.14+ since it adds the FetchContent_MakeAvailable macro that cuts down on boilerplate.

Docs:

https://cmake.org/cmake/help/latest/module/ExternalProject.html https://cmake.org/cmake/help/latest/module/FetchContent.html

Answer 2

Since I like keeping my CMake file agnostic on how I get my packages.

I was using FetchContent and added this file (Findalib.cmake):

if(NOT alib_POPULATED)
    set(alib_BUILD_TESTS OFF CACHE BOOL INTERNAL)
    set(alib_BUILD_EXAMPLES OFF CACHE BOOL INTERNAL)
    set(alib_BUILD_DOCS OFF CACHE BOOL INTERNAL)
    FetchContent_MakeAvailable(alib)
endif()
set(alib_FOUND TRUE)

Then, in my CMake files:

find_package(alib REQUIRED)
target_link_libraries(my-executable PUBLIC alib::alib)

That way, packages are only declared in my file in which I declare dependencies, and I fetch them only if I try to find them.

Answer 3

Coming from to @Alex Reinking's answer, It is generally better to have less boiler plate, and use ExternalProject only for super-builds. However sometimes libraries do not support FetchContent() like @Blackclaws points out. That being said, it is somewhat better to use ExternalProject_Add() and this will be clear to the us as we keep developing C++ projects or our project gets larger.

Here is an alternate solution without FetchContent_Declare() or find_package(), albeit it is not tested thoroughly. It is possible to do this in pure CMake without external package managers. See the Details section below. What would be helpful to know beforehand are the targets (.lib/.dll/.so/.dylib) and include directories, namespaces, etc... that are exported by the utility which is returned by the ExternalProject_Add() command.

You can modify this example to include more of the available features from the official docs

TL;DR: Details

While configuring the project, we can use a dummy library to make cmake to trigger the build without causing errors with ExternalProject_Add() and pass in steps to create a Debug config of a target that we can use in our projects with a Release config of another target, and so on.

1. You need to specify the build and install args to the ExternalProject_Add() via the CMAKE_CACHE_ARGS and not CMAKE_ARGS as they are superimposed by the transient args are carried forward (especially in IDEs like Visual Studio Code. If your project uses Debug build, but the ExternalProject specifies Release build)

2. Append to the CMAKE_INSTALL_PREFIX the Location/to/the/install/directory/of/your/target

3. Create a temporary directory file(MAKE_DIRECTORY Location/to/the/include/directory/of/your/target).This will suppress the Location is non-existent errors at configure time.

4. The next is to create a dummy imported interface library that is a target that links into the your main executables. You need to set IMPORTED_LOCATION and INTERFACE_INCLUDE_DIRECTORIES and other properties this step. The docs are here, in cmake-properties.

5. Finally, add_dependencies(dummy_target_name exported_target_name) to remove errors being thrown in the output

(The returned target is called a utility in Visual Studio Code with the CMakeTools Extension. I will call it utility here as well. Not sure if it is called a utility in native cmake)

Example

We will build GLFW with ExternalProject_Add().

Initial Project Structure

Project
    |
    |-src
    |   |-main.cpp
    |   
    |-CMakeLists.txt
    |
    |-cmake
    |   |-BuildExternalProject.cmake
    |   |-FindGLFW.cmake

Main CMakeLists.txt:

cmake_minimum_required(VERSION 3.25.2)
project("My_Project" VERSION 1.0.0 LANGUAGES C CXX)

set(CMAKE_MODULE_PATH "${PROJECT_SOURCE_DIR}/cmake")
include(BuildExternalProject)
include(FindGlfw)

add_executable(exec src/main.cpp)

target_link_libraries(exec
    PUBLIC
        glfw_target
)

FindGLFW.cmake

BuildExternalProject(
    NAME            glfw
    LIB_NAME        glfw_target
    GIT_REPO        https://github.com/glfw/glfw.git
    GIT_TAG         3.3.8
    INSTALL_PATH    ${PROJECT_SOURCE_DIR}/external
    BUILD_TYPE      Release
    ARCHITECTURE    x64
)

cmake/BuildExternalProject.cmake:

include(ExternalProject)

macro(append_cmake_prefix_path)
    list(APPEND CMAKE_PREFIX_PATH ${ARGN})
    string(REPLACE ";" "|" CMAKE_PREFIX_PATH "${CMAKE_PREFIX_PATH}")
endmacro()

function(BuildExternalProject)
    set(one_value_args
        NAME            # name of directory in <INSTALL_DIR>/NAME"
        LIB_NAME        # For example, the glfw library needs us to link against the target glfw3 not glfw
        GIT_REPO        # URL : https://github.com/user/repo.git
        GIT_TAG         # Release tags: "3.3.8","2.3.2" ...
        INSTALL_PATH        # "${PROJECT_SOURCE_DIR}/external" in this example
        BUILD_TYPE      # "Release", "Debug" ...
        ARCHITECTURE    # "x64","x86"...
    )
    set(multi_value_args
        EXTRA_CMAKE_ARGS
        EXT_BUILD_ARGS
        EXT_INSTALL_ARGS
    )

    # Parse arguments.
    cmake_parse_arguments(
        _ARGS # Main prefix for parsed args
        "${options}"
        "${one_value_args}"
        "${multi_value_args}"
        ${ARGN} # Number of Args
    )

    # Set new path
    set(INSTALL_PATH_NEW_PREFIX ${_ARGS_INSTALL_PATH}/${_ARGS_ARCHITECTURE}-${_ARGS_BUILD_TYPE}/${_ARGS_NAME})

    set(INSTALL_PATH_LIB ${INSTALL_PATH_NEW_PREFIX}/lib)
    set(INSTALL_PATH_INCLUDE ${INSTALL_PATH_NEW_PREFIX}/include)

    set(_UTILITY_NAME ${_ARGS_NAME}_utility)

    ExternalProject_Add(
        ${_UTILITY_NAME}
        PREFIX ${_UTILITY_NAME}
        GIT_REPOSITORY ${_ARGS_GIT_REPO}
        GIT_TAG ${_ARGS_GIT_TAG}

        CMAKE_CACHE_ARGS # CMAKE_ARGS does not use the FORCE option, this is a problem especially in IDEs
            "-DCMAKE_BUILD_TYPE:STRING=${_ARGS_BUILD_TYPE}"

        # This is where the install will
        "-DCMAKE_INSTALL_PREFIX:PATH=${INSTALL_PATH_NEW_PREFIX}"

        # Addition parsed args
        ${_ARGS_EXTRA_CMAKE_ARGS}

        # Commands to build
        BUILD_COMMAND ${CMAKE_COMMAND} --build <BINARY_DIR> --target install --config ${_ARGS_BUILD_TYPE}
        COMMAND ${_ARGS_EXT_BUILD_ARGS}

        # Commands to install
        INSTALL_COMMAND ${CMAKE_COMMAND} --build <BINARY_DIR> --target install --config ${_ARGS_BUILD_TYPE}
        COMMAND ${_ARGS_EXT_INSTALL_ARGS}

        CONFIGURE_HANDLED_BY_BUILD ON # No config steps are passed
        BUILD_ALWAYS FALSE
        UPDATE_DISCONNECTED TRUE
    )

    # We append to `CMAKE_INSTALL_PREFIX`
    append_cmake_prefix_path(${INSTALL_PATH_NEW_PREFIX})

    # We make the temporary directory
    file(MAKE_DIRECTORY ${INSTALL_PATH_INCLUDE})

    # We make the new dummy imported library
    add_library(${_ARGS_LIB_NAME} INTERFACE IMPORTED GLOBAL ${_UTILITY_NAME})

    # Setup the include and libs/dlls,etc
    target_include_directories(${_ARGS_LIB_NAME} INTERFACE ${INSTALL_PATH_INCLUDE})
    target_link_libraries(${_ARGS_LIB_NAME} INTERFACE ${INSTALL_PATH_LIB}/*.lib)

    add_dependencies(${_ARGS_LIB_NAME} ${_UTILITY_NAME})
endfunction()

Final Project Structure

Project
    |
    |-src
    |   |-main.cpp
    |   
    |-external
    |   |-x64-Debug
    |   |   |-glfw                                          # Everything under this is auto generated
    |   |       |-include/GLFW/..         
    |   |       |-lib                     
    |   |       |   |-cmake/*Config.cmake; *.cmake          # This is where we need to point CMAKE_INSTALL_PREFIX
    |   |       |   |                                       # so find_package() can find it you need it to in the future
    |   |       |   |-pkgconfig/*.pc                    
    |   |                            
    |   |-x64-Release                                          
    |       |-glfw                                          # Everything under this is auto generated
    |           |-include/GLFW/..         
    |           |-lib                     
    |           |   |-cmake/*Config.cmake; *.cmake          # This is where we need to point CMAKE_INSTALL_PREFIX
    |           |   |                                       # so find_package() can find it you need it to in the future
    |           |   |-pkgconfig/*.pc
    |                                
    |   
    |
    |-CMakeLists.txt
    |
    |-cmake
    |   |-BuildExternalProject.cmake
    |   |-FindGLFW.cmake

FetchContent_Declare vs ExternalProject_Add

1. Since FetchContent_Declare() makes the target available at configure time, and if the source is very large, you might not want to download or build all the libs at configure time, especially if your project needs only a few sub-targets from the external project. These parameters can be passed in as the optional args to the subproject via ExternalProject_Add().

2. From the docs of FetchContent_Declare

   The contentOptions can be any of the download, update or patch options that the ExternalProject_Add() command understands. The configure, build, install and test steps are explicitly disabled and therefore options related to them will be ignored.

3. ExternalProject_Add() provides a larger set of options to run at configure, build, install, test, etc. It is a project in itself, and can help directly expose the fully or partially build binaries, libs, etc.

Sources

1. https://cmake.org/cmake/help/latest/module/ExternalProject.html
2. https://www.youtube.com/watch?v=nBptg3SHPGU&ab_channel=JeffersonAmstutz
3. https://github.com/jeffamstutz/superbuild_ospray
4. https://github.com/geospace-code/h5fortran/blob/main/cmake/hdf5.cmake
5. https://github.com/geospace-code/h5fortran/blob/main/cmake/h5fortran.cmake
6. https://github.com/deepmind/mujoco/blob/main/cmake/FindOrFetch.cmake