cmake add_dependencies, add_custom_command and add_custom_target not working together

Question

I have been trying to get the simple use case of getting cmake to to generate some files with a script and then build an executable out of those generated files. After spending hours reading the documentation and various stack overflow answers, it seems like this is how everything should be arranged, yet cmake refuses to configure the project.

The idea is that things are done in following steps:

During configure time, the generate_files.sh is generated with correct paths
During build time, the target gen is built first and then target test is built

Step 1 is being performed properly, however step 2 isn't. Cmake complains that the source files for target test aren't found, even though it should build gen first and then it would find the sources.

What am I doing wrong?

Directory structure:

.
├── build
├── CMakeLists.txt
├── config
│   └── generate_files.sh.in
├── extern
│   ├── CMakeLists.txt
│   └── gen
├── scripts
└── src
    ├── CMakeLists.txt
    └── main.cpp

Top level CMakeLists:

cmake_minimum_required(VERSION 3.13)
project(test VERSION 0.1.0.0)

add_subdirectory(src)
add_subdirectory(extern)

configure_file("${CMAKE_SOURCE_DIR}/config/generate_files.sh.in" "${CMAKE_SOURCE_DIR}/scripts/generate_files.sh")

src/CMakeLists:

add_executable(test main.cpp)

extern/CMakeLists:

set(generated_sources
    ${CMAKE_CURRENT_SOURCE_DIR}/gen/generated.h
    ${CMAKE_CURRENT_SOURCE_DIR}/gen/generated_1.cpp
    ${CMAKE_CURRENT_SOURCE_DIR}/gen/generated_2.cpp
    )

set(generated_directories
    ${CMAKE_CURRENT_SOURCE_DIR}/gen)

add_custom_command(
    OUTPUT ${generated_sources}
    COMMAND "${CMAKE_SOURCE_DIR}/scripts/generate_files.sh"
    )

add_custom_target(gen
    DEPENDS ${generated_sources}
    )

add_dependencies(test gen)

target_sources(test
    PRIVATE
        ${generated_sources}
)

target_include_directories(test
    PRIVATE
        ${generated_directories}
)

config/generate_files.sh.in:

#! /bin/bash

echo " const int get_trouble_code();
const int get_higher_trouble_code();" > ${CMAKE_SOURCE_DIR}/extern/gen/generated.h
echo " #include \"generated.h\"
const int get_trouble_code(){return 1;}" > ${CMAKE_SOURCE_DIR}/extern/gen/generated_1.cpp
echo " #include \"generated.h\"
const int get_higher_trouble_code(){return 1+1;}" > ${CMAKE_SOURCE_DIR}/extern/gen/generated_2.cpp

src/main.cpp:

#include "generated.h"

int main()
{
  get_trouble_code();
  get_higher_trouble_code();

  return 0;
}

Edit: I found a way to make it work, but now I am even more confused. Adding the generated files as a library instead of custom target seems to do the trick. The following changes to extern/CMakeLists work, but could someone explain why? :

set(generated_sources
    ${CMAKE_CURRENT_SOURCE_DIR}/gen/generated.h
    ${CMAKE_CURRENT_SOURCE_DIR}/gen/generated_1.cpp
    ${CMAKE_CURRENT_SOURCE_DIR}/gen/generated_2.cpp
    )

set(generated_directories
    ${CMAKE_CURRENT_SOURCE_DIR}/gen)

add_custom_command(
    OUTPUT ${generated_sources}
    COMMAND "${CMAKE_SOURCE_DIR}/scripts/generate_files.sh"
    )

#comment out this stuff
#add_custom_target(gen
#   DEPENDS ${generated_sources}
#    )
#
#add_dependencies(test gen)
#
#target_sources(test
#    PRIVATE
#        ${generated_sources}
#)

#add as library instead
add_library(gen ${generated_sources})
target_link_libraries(test PRIVATE gen)

target_include_directories(test
    PRIVATE
        ${generated_directories}
)

Immediate red-flag: setting the output of `add_custom_command` to be in the source tree. — Alex Reinking, Feb 19 '23 at 19:19
I'm guessing Alex is implying that you should output generated files in the build tree, and if the output varies for each config in a multi-config setup, also in a per-multi-config subdirectory or with a per-multi-config filename prefix/suffix. (which is good advice. makes it easier to not have to write extra gitignore rules) — starball, Feb 19 '23 at 19:21
I'd try adding `${generated_sources}` to the `BYPRODUCTS` of the custom target. If that works, let me know and I can write up an answer. I'd also suggest that you specify your `config/generate_files.sh.in` in the `DEPENDS` argument of the custom command, since you'll probably want the custom command to be re-run if you edit that script file. fun extra readings somewhat related: [by craig scott](https://crascit.com/2017/04/18/generated-sources-in-cmake-builds/) — starball, Feb 19 '23 at 19:22
@AlexReinking I know that to avoid polluting the source tree, one should generate in the build tree, however, this seems like a "best practice" and should not break configuring and building by cmake isn't it? I am happy to be wrong, I probably thought wrong. — learning2code, Feb 19 '23 at 19:24
I am constantly amazed by the bugs I find in CMake by doing not-best-practice things. — Alex Reinking, Feb 19 '23 at 19:25
@AlexReinking I am not sure how this helps, best practices are found somewhere else outside of documentation, which means they shouldn't cause bugs. — learning2code, Feb 19 '23 at 19:26
@Tsyvarev you are right! This is my mistake that creeped in when I was reducing my problem to a minimal working example. I will change this. — learning2code, Feb 21 '23 at 10:26

score 2 · Accepted Answer · answered Feb 19 '23 at 19:35

CMake complains that the source files for target test aren't found ...

This is because the source file is absent, and its GENERATED property is not set.

Normally, add_custom_command sets GENERATED property for all files listed in its OUTPUT. But before CMake 3.20 that property was local to the directory:

add_custom_command is called from extern/CMakeLists.txt, but
target test evaluates its sources in top-level CMakeLists.txt, where the target is created. (Irrespective where target_sources is called).

In CMake 3.20 the GENERATED property becomes global (for use this feature your project should have corresponding cmake_minimum_required).

If updating CMake is not an option, then you could set GENERATED property manually in top-level CMakeLists.txt. (This will hurts locality of your CMakeLists.txt .. so it is better to update CMake.)

Technically, you trigger the problem issued in that bugreport.

You the awesome! Changing cmake minimum version to 3.20 `cmake_minimum_required(VERSION 3.20)` solved the issue!! Thank you so much!!!! — learning2code, Feb 19 '23 at 19:48
This is a good answer that directly addresses the observed error, so +1. For the benefit of future readers, my answer shows a much more straightforward approach that works very far back and is more hygienic in its management of the source and binary trees. — Alex Reinking, Feb 19 '23 at 19:59

score 0 · Answer 2 · answered Feb 19 '23 at 19:31

Here's a minimal working example that I think is much less complicated and follows best practices. Here, there's no need to configure_file your generation script. There's also no need to write to the source directory, which you should never do.

The link between the custom command and your target is clearly established, too. Plus, the include paths are set up correctly.

Files

CMakeLists.txt:

cmake_minimum_required(VERSION 3.25)
project(example)

set(gendir "${CMAKE_CURRENT_BINARY_DIR}/extern/gen")

set(genfiles generated.h generated_1.cpp generated_2.cpp)
list(TRANSFORM genfiles PREPEND "${gendir}/")

add_custom_command(
  OUTPUT ${genfiles}
  COMMAND "${CMAKE_CURRENT_SOURCE_DIR}/generate_files.sh"
)

add_executable(app main.cpp ${genfiles})
target_include_directories(app PRIVATE "${gendir}")

generate_files.sh:

#!/bin/bash

set -e -o pipefail

mkdir -p extern/gen

echo "const int get_trouble_code();
const int get_higher_trouble_code();" > extern/gen/generated.h
echo "#include \"generated.h\"
const int get_trouble_code(){return 1;}" > extern/gen/generated_1.cpp
echo "#include \"generated.h\"
const int get_higher_trouble_code(){return 1+1;}" > extern/gen/generated_2.cpp

main.cpp:

^{(same as above, but reproduced here for completeness)}

#include "generated.h"

int main()
{
  get_trouble_code();
  get_higher_trouble_code();

  return 0;
}

Build output

Here's what I see when it builds (successfully!)

$ cmake -G Ninja -S . -B build -DCMAKE_BUILD_TYPE=Debug
-- The C compiler identification is GNU 11.3.0
-- The CXX compiler identification is GNU 11.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /home/alex/test/build
$ cmake --build build --verbose
[1/5] cd /home/alex/test/build && /home/alex/test/generate_files.sh
[2/5] /usr/bin/c++  -I/home/alex/test/build/extern/gen -g -MD -MT CMakeFiles/app.dir/extern/gen/generated_1.cpp.o -MF CMakeFiles/app.dir/extern/gen/generated_1.cpp.o.d -o CMakeFiles/app.dir/extern/gen/generated_1.cpp.o -c /home/alex/test/build/extern/gen/generated_1.cpp
[3/5] /usr/bin/c++  -I/home/alex/test/build/extern/gen -g -MD -MT CMakeFiles/app.dir/extern/gen/generated_2.cpp.o -MF CMakeFiles/app.dir/extern/gen/generated_2.cpp.o.d -o CMakeFiles/app.dir/extern/gen/generated_2.cpp.o -c /home/alex/test/build/extern/gen/generated_2.cpp
[4/5] /usr/bin/c++  -I/home/alex/test/build/extern/gen -g -MD -MT CMakeFiles/app.dir/main.cpp.o -MF CMakeFiles/app.dir/main.cpp.o.d -o CMakeFiles/app.dir/main.cpp.o -c /home/alex/test/main.cpp
[5/5] : && /usr/bin/c++ -g  CMakeFiles/app.dir/main.cpp.o CMakeFiles/app.dir/extern/gen/generated_1.cpp.o CMakeFiles/app.dir/extern/gen/generated_2.cpp.o -o app   && :
$ ./build/app
$ # doesn't crash

Thank you, but this completely changes the usecase of the question. Often one cannot copy entire code generators to build directories. The minimal working example works becasue its a simple bash script generating a couple files. This cannot work where a large codegenerator project already exists generating hundreds of files depending on various other user input through a GUI. This is why a separate CMakeLists and target method was chosen. The key question is why add_dependencies did not work with the custom target. This answer simply omits these two commands :( — learning2code, Feb 19 '23 at 19:45
"Often one cannot copy entire code generators to build directories." -- that's not happening, here... nothing is being copied at all. In your original post, `configure_file` was copying your generation script, so this answer actually copies _less_ than before... — Alex Reinking, Feb 19 '23 at 19:47
Seems like I am unable to communicate properly to get my point across :(. The accepted answer solved my issue without needed to massively change my setup. — learning2code, Feb 19 '23 at 19:51
Without the full context of your real use-case, it's impossible for me to tell whether the complexity in your question is accidental or not. My answer at least shows how things _ought_ to be done when your example is representative of what needs doing: the "simple use case of getting cmake to to generate some files with a script and then build an executable out of those generated files". — Alex Reinking, Feb 19 '23 at 19:55
Stack overflow rules told me that one should include a "minimal working example". That is what I did. I reduced my complex problem to the minimum required that produces the same error. I am sure I have followed all the rules and have been respectful of everyone. In the end, it was a bug in cmake, so I do not think I need to justify my code in any manner. — learning2code, Feb 19 '23 at 20:06

cmake add_dependencies, add_custom_command and add_custom_target not working together

2 Answers2

Files

Build output