CMake declare dependency of function on ExternalProject_Add

Question

TLDR:

My problem is that CMake starts executing this function before downloading the repository. I would like to declare a dependency for that function on ExternalProject_Add so that CMake understands that it should download, build and then run the function.

Context:

I have a cmake module SomeModule.cmake which is supposed to add flatbuffers as an external project from its repository and build it. The build would produce flatbuffers compiler executable which I intend to use in some/directory/CMakeLists.txt file to generate c++ header files from a flatbuffers schema. So in that same CMake module that I use ExternalProject_Add, I have declared a CMake function that generates header files from a given set of schema files and somewhere in some/directory/CMakeLists.txt I call that function.

My problem is that CMake starts executing this function before downloading the repository. I would like to declare a dependency for that function on ExternalProject_Add so that CMake understands that it should download, build and then run the function.

Enough talk. Here's relevant parts of the code:

SomeModule.cmake:

include(ExternalProject)

set(flatbuffers_CMAKE_ARGS
    "-DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}"
    "-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}"
    "-DFLATBUFFERS_BUILD_TESTS=OFF"
    "-DFLATBUFFERS_BUILD_FLATC=ON"
    "-DFLATBUFFERS_BUILD_FLATHASH=OFF"
    "-DCMAKE_INSTALL_PREFIX=${OTS_DEPENDENCIES}"
)

ExternalProject_Add(
    flatbuffers
    GIT_REPOSITORY  "https://github.com/google/flatbuffers.git"
    GIT_TAG         "v1.9.0"
    SOURCE_DIR      "${OTS_DEPDENDENCIES_DIR}/flatbuffers"
    BINARY_DIR      "${OTS_DEPDENDENCIES_DIR}/flatbuffers"
    CMAKE_ARGS      "${flatbuffers_CMAKE_ARGS}"
    INSTALL_COMMAND ""
)

ExternalProject_Get_Property(flatbuffers SOURCE_DIR)
ExternalProject_Get_Property(flatbuffers BINARY_DIR)
set(flatbuffers_SOURCE_DIR ${SOURCE_DIR})
set(flatbuffers_BINARY_DIR ${BINARY_DIR})
set(flatbuffers_INCLUDE_DIR ${flatbuffers_SOURCE_DIR}/include)
set(flatbuffers_FLATC_EXECUTABLE ${flatbuffers_BINARY_DIR}/flatc)
# please assume that the variables above are all set to appropriate values

function(FlatbuffersGenerateCpp SCHEMA_FILES GENERATED_DIR GENERATED_CXX)
    foreach(SCHEMA_FILE ${SCHEMA_FILES})
        get_filename_component(NAME ${SCHEMA_FILE} NAME_WE)
        set(GENERATED_HEADER_FILE_PATH ${GENERATED_DIR}/${NAME}_generated.h)
        message(STATUS "attempting to generate: ${GENERATED_HEADER_FILE_PATH}")
        add_custom_command(
            DEPENDS ${flatbuffers_FLATC_EXECUTABLE}
            OUTPUT ${GENERATED_HEADER_FILE_PATH}
            COMMAND ${flatbuffers_FLATC_EXECUTABLE} -o ${GENERATED_DIR} -c ${SCHEMA_FILE}
            COMMENT "generating flatbuffers c++ header file: ${GENERATED_HEADER_FILE_PATH}"
        )
        list(APPEND GENERATED_FILES ${GENERATED_HEADER_FILE_PATH})
    endforeach()
    message(STATUS "generated c++ header files: ${GENERATED_FILES}")
    set(${GENERATED_CXX} ${GENERATED_FILES} PARENT_SCOPE)
endfunction()

And some/directory/CMakeLists.txt:

# cmake module path is properly set so the following works:
include(SomeModule)

set(flatbuffers_GENERATED_INCLUDES_DIR
    ${CMAKE_BINARY_DIR}/generated/config/flatbuffers
)

FlatbuffersGenerateCpp(
    "${flatbuffers_SCHEMAS}"
    "${flatbuffers_GENERATED_INCLUDES_DIR}"
    flatbuffers_GENERATED_CXX
)

add_library(
    my_framework
SHARED
    ${THE_PUBLIC_HEADER_FILES}
    ${THE_IMPL_SOURCE_FILES}
    ${THE_IMPL_HEADER_FILES}
    ${flatbuffers_GENERATED_CXX}
)

add_dependencies(my_framework flatbuffers ${flatbuffers_GENERATED_CXX})

target_include_directories(my_framework PRIVATE ${flatbuffers_INCLUDE_DIR})
target_include_directories(my_framework PRIVATE ${CMAKE_SOURCE_DIR})
target_include_directories(my_framework PRIVATE ${CMAKE_BINARY_DIR}/generated)
set_source_files_properties(${flatbuffers_GENERATED_CXX} PROPERTIES GENERATED TRUE)

CMake functions are executed at *configure* stage, so you need to build the external project at configure stage too. (There is no "dependencies" during this stage - CMake scripts are processes sequentially, line by line). See [that question](https://stackoverflow.com/questions/37553280/how-to-build-cmake-externalproject-while-configurating-main-one) about building external project at configure stage. — Tsyvarev, Jul 12 '18 at 15:49
Thank you for your comment. You are always helpful. :) I honestly was not aware of the distinction between configure and build stage in CMake. I would start looking into the mentioned question. — Pejman, Jul 12 '18 at 16:28

score 2 · Accepted Answer · edited Jul 12 '18 at 19:52

I did start modifying my code based on comment posted by Tsyvarev:

CMake functions are executed at configure stage, so you need to build the external project at configure stage too.

While I trusted that his proposed solution would work, I was slightly uncomfortable and kept thinking that there has to be a more elegant solution. I consulted with a colleague and came up with a better solution which is as simple as the following diff (which removes ${flatbuffers_GENERATED_CXX}).

- add_dependencies(my_framework flatbuffers ${flatbuffers_GENERATED_CXX})
+ add_dependencies(my_framework flatbuffers)

we reviewed that the problem with the code in question is that as is, CMake reads add_dependencies(my_framework flatbuffers ${flatbuffers_GENERATED_CXX}) and understands that it needs ${flatbuffers_GENERATED_CXX} as target to build my_framework so it proceeds with running the function. But there is no way for it to understand that the function depends on the external project. Now if we remove the explicit dependency declaration of ${flatbuffers_GENERATED_CXX}, CMake defers running the function to after resolving other dependencies (flatbuffers target) which will effectively download and build the external project prior to running the project.

This solution has a slight disadvantage that we are removing the explicit dependency declaration which (i think) always helps readability. — Pejman, Jul 12 '18 at 18:19
Hm, it seems that the problem "CMake starts executing this function before downloading the repository" isn't the one you have actually faced: It is OK to run the function, which creates *rules* (`add_custom_command`) for creating the files, *without* having the `flatbuffers` project downloaded. What do you need is to have `flatbuffers` being downloaded **before** `add_custom_command` will actually produce the files. And this is *build stage*, not a *configure* one. The code was right in general: you create a target which depends both on `flatbuffers` and files generating by the command. — Tsyvarev, Jul 12 '18 at 20:07
Note, that `add_dependencies` is for CMake **targets** only; it doesn't work with regular *files*. So it was wrong to use `${flatbuffers_GENERATED_CXX}` with this CMake command in any case. — Tsyvarev, Jul 12 '18 at 20:09

CMake declare dependency of function on ExternalProject_Add

1 Answers1