Using an ExternalProject download step with Ninja

Question

This seems to be a common problem without a clear answer.

The situation is: we have a 3rd party dependency that we want to install at build time when building a target that depends on it. That's roughly:

ExternalProject_Add(target-ep
    DOWNLOAD_COMMAND <whatever>
    BUILD_COMMAND ""
    INSTALL_COMMAND ""
    CONFIGURE_COMMAND "")

add_library(target-imp STATIC IMPORTED)
set_target_properties(target-imp PROPERTIES
    INTERFACE_INCLUDE_DIRECTORIES /path/to/install/include
    IMPORTED_LOCATION /path/to/install/lib/libwhatever.a)

add_library(target INTERFACE)
target_link_libraries(target INTERFACE target-imp)
add_dependencies(target target-ep)

(It takes three to tango here because of cmake issue 15052)

When using Unix Makefiles as the generator, this works great. Only installs dependencies on demand, all the builds work correctly.

However, on Ninja, this fails immediately with something like:

ninja: error: '/path/to/install/lib/libwhatever.a', needed by 'something', missing and no known rule to make it

This is because Ninja scans dependencies differently from Make (see ninja issue 760). So what we have to do is actually tell Ninja that this external dependency exists. We can do that:

ExternalProject_Add(target-ep
    DOWNLOAD_COMMAND <whatever>
    BUILD_BYPRODUCTS /path/to/install/lib/libwhatever.a
    BUILD_COMMAND ""
    INSTALL_COMMAND ""
    CONFIGURE_COMMAND "")

Which unfortunately also fails with:

No build step for 'target-ep'ninja: error: mkdir(/path/to/install): Permission denied

This is because my download step has permissions to write to that path, but whatever mkdir command is being run by the underlying add_custom_command() from with ExternalProject_Add() does not.

So:

1. Is this possible at all with Ninja and CMake? (Version is not an issue, I can use the latest CMake if that solves the problem)
2. If there is some way to workaround with explicitly listing BUILD_BYPRODUCTS, is there a way to simply communicate that the entire directory that will get installed is a byproduct? That is, /path/to/install/* is a byproduct?

Answer 1

The hidden mkdir step of ExternalProject (which all other steps directly or indirectly depend on) always tries to create the full set of directories, even if they won't be used. You can see this here. For reference, it does this:

ExternalProject_Add_Step(${name} mkdir
  COMMENT "Creating directories for '${name}'"
  COMMAND ${CMAKE_COMMAND} -E make_directory ${source_dir}
  COMMAND ${CMAKE_COMMAND} -E make_directory ${binary_dir}
  COMMAND ${CMAKE_COMMAND} -E make_directory ${install_dir}
  COMMAND ${CMAKE_COMMAND} -E make_directory ${tmp_dir}
  COMMAND ${CMAKE_COMMAND} -E make_directory ${stamp_dir}${cfgdir}
  COMMAND ${CMAKE_COMMAND} -E make_directory ${download_dir}
  COMMAND ${CMAKE_COMMAND} -E make_directory ${log_dir}   # This one only since CMake 3.13
  )

The default install location on Unix systems is probably going to be /usr/local, so if you don't have write permissions to all of the directories it tries to make, then that may be related to your problem. I suggest you check the permissions of each of these locations and make sure they either already exist or are writable. Alternatively, you could specify an install directory that is local to the build tree so that even though it won't be used, it can at least always be created (see example further below).

If you use Ninja, it will be more rigorous in its dependency checking than make. You have target-ep doing the download that provides libwhatever.a, so you do need BUILD_BYPRODUCTS to tell Ninja that target-ep is what creates that file. As you've found out, if you don't then target-imp will point at a library that won't initially exist and Ninja rightly complains that it is missing and it doesn't know how to create it. If you provide BUILD_BYPRODUCTS, it makes sense that the build step shouldn't be empty, so you probably need to do something as a build step, even if it is just a BUILD_COMMAND that doesn't actually do anything meaningful.

The following modified definition of target-ep should hopefully get things working for you:

ExternalProject_Add(target-ep
    INSTALL_DIR ${CMAKE_CURRENT_BUILD_DIR}/dummyInstall
    DOWNLOAD_COMMAND <whatever>
    BUILD_BYPRODUCTS /path/to/install/lib/libwhatever.a
    BUILD_COMMAND ${CMAKE_COMMAND} -E echo_append
    INSTALL_COMMAND ""
    CONFIGURE_COMMAND "")

Your original question also creates a dependency on the wrong target. target-imp should depend on target-ep, but you had target depend on target-ep instead. The correct dependency can be expressed by this:

 add_dependencies(target-imp target-ep)

With the BUILD_BYPRODUCTS option, Ninja already knows the above dependency, but it is needed for other generators, including make.

You haven't specified what your <whatever> download command does, but I'm assuming it is responsible for ensuring that the library will exist at /path/to/install/lib/libwhatever.a when it has executed. You could also try making the DOWNLOAD_COMMAND empty and putting <whatever> as the BUILD_COMMAND instead.

To address your specific questions:

1. Is this possible at all with Ninja and CMake? (Version is not an issue, I can use the latest CMake if that solves the problem)

Yes, I verified that the approach mentioned above works with Ninja 1.8.2 for a dummy test project on macOS using CMake 3.11.0. I would expect it to work with CMake 3.2 or later (that's when support for the BUILD_BYPRODUCTS option was added).

2. If there is some way to workaround with explicitly listing BUILD_BYPRODUCTS, is there a way to simply communicate that the entire directory that will get installed is a byproduct? That is, /path/to/install/* is a byproduct?

Unlikely. How would Ninja know what is expected to be in such a directory? The only way to get reliable dependencies would be to explicitly list each file that was expected to be there, which you do using BUILD_BYPRODUCTS in your case.

Answer 2

If you're willing to download at configuration time, you could follow this post. It uses google-test as the example, but I've used the same technique for other dependencies. Just put your ExternalProject code in a separate file, say "CMakeLists.txt.dependencies" and then launch another cmake with execute_process. I use configure_file first to inject configuration information into the external project and to copy it into the build tree.

configure_file(CMakeLists.txt.dependency.in dependency/CMakeLists.txt)
execute_process(COMMAND "${CMAKE_COMMAND}" -G "${CMAKE_GENERATOR}" .
        WORKING_DIRECTORY "${CMAKE_BINARY_DIR}/dependency" )
execute_process(COMMAND "${CMAKE_COMMAND}" --build .
        WORKING_DIRECTORY "${CMAKE_BINARY_DIR}/dependency" )

I do this at configuration time so find_package and find_library commands can work on the dependencies.

And now it doesn't matter what generator you use.