cmake add_custom_command introduces false dependency unless add_custom_target also used

Question

I'm using cmake to build some libraries, all of which generate some of their files.

I've created the generated files using add_custom_command(), but I've discovered something which seems like a false dependency. If a downstream library has a generated file and links to an upstream library, the downstream library sources will not start to compile until the upstream library is completely built. With many libraries (more than 50) in my project, this false dependency causes serialization in the build.

What's curious is that I also noticed that if an explicit add_custom_target() for the generated file is used with add_dependencies(), the false dependency no longer exists, and the files in the downstream library will compile concurrently with the ones in the upstream library. So I have a workaround, but is this expected behavior?

Using Cmake 1.19, Ninja 1.10.2.

The following is a minimal CMakeLists.txt file that shows what happens. The WORKS option conditionally adds the add_custom_target() and add_dependencies() clauses which cause it to work (quickly). The files foo.c and bar.c are empty, and I'm building in a subdirectory of the CMakeLists.txt directory.

I put a file called /tmp/x which is a wrapper around cc that sleeps to show the serialization occur:

#!/bin/bash -e

echo "mycc" $(date)
sleep 4
exec /usr/bin/cc $*

Here is the CMakeLists.txt:

cmake_minimum_required(VERSION 3.19)

project(test)
option(WORKS "false dependency gone if WORKS set to ON" OFF)

add_library(foo
  foo.c
  )

add_library(bar
  bar.c
  bargen.h
  )

target_include_directories(bar PRIVATE ${CMAKE_CURRENT_BINARY_DIR})
add_custom_command(OUTPUT bargen.h
        COMMAND touch bargen.h
   )
if (${WORKS})
  add_custom_target(generate_file
        DEPENDS bargen.h
  )
  add_dependencies(bar generate_file)
endif()

target_link_libraries(bar PUBLIC foo)

Build it like this and you will see the serialization: bar.c will not start to compile until after foo.c has finished compiling (in fact, not until after the foo library is built). (I'm explicitly choosing '-j 4' to ensure Ninja will try to build in parallel).

cmake -DWORKS=OFF -G Ninja -DCMAKE_C_COMPILER=/tmp/x .. && ninja clean && ninja -j 4 -v | grep mycc

This shows the following output:

-- Configuring done
-- Generating done
-- Build files have been written to: /Users/rhb/Downloads/cmake-anomaly/build
[1/1] Cleaning all built files...
Cleaning... 5 files.
mycc Sat Jan 2 15:15:25 EST 2021
mycc Sat Jan 2 15:15:29 EST 2021

Now build it like this and you'll see that bar.c compiles concurrently with foo.c:

cmake -DWORKS=ON -G Ninja -DCMAKE_C_COMPILER=/tmp/x .. && ninja clean && ninja -j 4 -v | grep mycc

-- Configuring done
-- Generating done
-- Build files have been written to: /Users/rhb/Downloads/cmake-anomaly/build
[1/1] Cleaning all built files...
Cleaning... 5 files.
mycc Sat Jan 2 15:15:37 EST 2021
mycc Sat Jan 2 15:15:37 EST 2021

For `add_library` you should use the path to the file; his should be `"${CMAKE_CURRENT_BINARY_DIR}/bargen.h"`, not `bargen.h`. Furthermore you need to mark this file as generated to prevent cmake from complaining about a missing source: `set_source_files_properties("${CMAKE_CURRENT_BINARY_DIR}/bargen.h" PROPERTIES GENERATED 1)` — fabian, Jan 04 '21 at 18:52
Unfortunately (at least for cmake/19) neither of your suggestions address the problem, or are necessary for functionality. As you can see from the output above, cmake finds the bargen.h file (because there is no ambiguity for where it is located). There is no complaint about a missing source. And, according to the cmake documentation, the GENERATED property is automatically marked on a file which is the OUTPUT of an add_custom_command(). https://cmake.org/cmake/help/latest/prop_sf/GENERATED.html — Rick, Jan 07 '21 at 22:29

score 0 · Answer 1 · answered Feb 20 '23 at 17:16

This was reported as a cmake issue and recently fixed with a new argument to add_custom_command. This fix will be released with the next cmake version (3.27).

DEPENDS_EXPLICIT_ONLY

.. versionadded:: 3.27

Indicate that the command's DEPENDS argument represents all files required by the command and implicit dependencies are not required.

Without this option, if any target uses the output of the custom command, CMake will consider that target's dependencies as implicit dependencies for the custom command in case this custom command requires files implicitly created by those targets.

Only the Ninja Generators actually use this information to remove unnecessary implicit dependencies.

cmake add_custom_command introduces false dependency unless add_custom_target also used

1 Answers1