add_custom_command -- update dependency list over rebuilds

Question

See last status update

Initial conditions

• code generator that generates set of c++ sources taking one input file as parameter
• input file may include other input files

• already solved task of getting list of output files, parsing input codegen files to get full list of codegen inputs. I.e. add_custom_command is provided with correct set of dependencies for the first time:

  add_custom_command(OUTPUT ${generatedSources}
                     COMMAND ${codegenCommand} ARGS ${codegenArgs}
                     DEPENDS ${codegenInputFiles})
  

Problem scenario

• current system works well until someone modifies one of codegen input files to include new input file or remove include of existing one. This case, it is required to update list of codegen input file provided to add_custom_command as dependencies, but I have no idea how

What is missing

• ability to update add_custom_command dependencies over project rebuilds

Is there any way to solve it without making full project rebuild ?

UPDATE - Alternative (better?) problem description

I`ve found similar not answered question on cmake mailing list, post it here for better clarity: http://article.gmane.org/gmane.comp.programming.tools.cmake.user/52279

I am trying to get a code generation tool to behave "the same as" a C source file with respect to dependencies. By that I mean, suppose you have a C file "a.c". Because it can #include files, every time the content of a.c changes, its dependencies may have changed also. The dependencies get rescanned with -MMD. I would like some way to emulate this for my code generator. First I tried add_custom_command, which takes a fixed DEPENDS list, determined at the time the custom command is defined. Concretely, I mean something like this:

function(add_generated_library)
   figure_out_dependencies(deps ${ARGN})
   add_custom_command(... DEPENDS ${deps})
endfunction()

But that only captures the dependencies at build-system-generation time. Every time the custom command runs, the DEPENDS list may need to change, because the changes may imply new dependencies.How should I go about doing this?

UPDATE 2 - Possible solution

Following I consider as facts - there are voices across the web regarding cmake support of dynamic dependencies, which is required for smooth integration of many non-trivial code generation tools - there`s no ready-for-use optimal solution available, as what we actually need is hook to add support of custom DSL to IMPLICIT_DEPENDS

From cmake manual:

The IMPLICIT_DEPENDS option requests scanning of implicit dependencies of an input file. The language given specifies the programming language whose corresponding dependency scanner should be used. Currently only C and CXX language scanners are supported. The language has to be specified for every file in the IMPLICIT_DEPENDS list. Dependencies discovered from the scanning are added to those of the custom command at build time.

Solution below adheres (hopefully) to following criteria:

• avoid not necessary dependency scanning on rebuild
• avoid not necessary code generator run on rebuild
• allow providing cmake functions to clients to register their models and generate code/create libraries from that code, without imposing any project structure requirements (i.e. no sub-project responsible for code generation, models are distributed across project hierarchy using project-specific policy)

Solution idea

It is not possible to register custom language scanner, but it is possible to reuse existing one. The idea is that dependencies / hierarchy of custom model files get reflected as hierarchy of "C" header files. Each hierarchy node get added on registering of model file and C file includes match model file includes. If model file includes get changed, C file includes get changed. Therefore, each codegen invocation would depend on just one generated C header reflecting passed model. Each reflected file will have a dependency on the model file and get touched on model file change.

To sum up : probably, my wording is not that clear at this point, but with respect to other people needs and community helped me to investigate this problem, I will post generic solution (+ link to github or new cmake wiki page) without my project specifics once it is ready (in 1-3 days).

Answer 1

Can you show how you initialize the variable codegenInputFiles? You can probably use a file(GLOB ... ) or file(GLOB_RECURSE ... ) command there. See documentation.

Note, though, that you will have to rerun cmake to have your command being generated. Are you working with git? Then you can have a hook that forces a cmake call every time you pull (so that if somebody modified the codegenInputFiles your auto generated files will be updated).

---

After clarification of the problem, you should be able to find a workaround by using IMPLICIT_DEPENDS instead of DEPENDS. Limitations:

1. It can only work if your input file is C/C++ (check the syntax, as you must specify the language for each file specified)
2. You might need to check your cmake version supports that command, even though it looks that it has been around for a while
3. It's only supported for Makefile generator, which sounds pretty bad...

---

EDIT

After some iterations, I finally got what is your problem. I propose the following solution: separate the file generation in a separate cmake subproject. When you will build your main project (by calling make), you will trigger both cmake and make for your subproject. Calling cmake is necessary for keeping updated the dependencies, while calling make to actually build your auto-generated sources.

Here I show an example of a project and a subproject, with the project invoking cmake and make for the subproject.

Structure:

.
├── CMakeLists.txt
├── a.cpp
├── build
└── subProject
    └── CMakeLists.txt

File content

./CMakeLists.txt:

cmake_minimum_required(VERSION 2.8)

add_custom_target(subProjectTarget ALL)
add_custom_command(TARGET subProjectTarget PRE_BUILD COMMAND mkdir -p ${CMAKE_BINARY_DIR}/subProject && cd ${CMAKE_BINARY_DIR}/subProject && ${CMAKE_COMMAND} ${CMAKE_SOURCE_DIR}/subProject && make)

include_directories(${CMAKE_BINARY_DIR}/subProject)
add_executable (dummy a.cpp)
add_dependencies (dummy subProjectTarget)

./a.cpp (Note that b.h doesn't exist yet)

#include "b.h"

int main () {
}

./SubProject/CMakeLists.txt

cmake_minimum_required(VERSION 2.8)
file(WRITE ${CMAKE_BINARY_DIR}/b.h "//I am a dummy file\n")

Building the project (using default make)

me@here:~/example/build$ cmake ..
-- The C compiler identification is GNU 4.8.2
-- The CXX compiler identification is GNU 4.8.2
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Configuring done
-- Generating done
-- Build files have been written to: /home/me/example/build

me@here:~/example/build$ make
Scanning dependencies of target subProjectTarget
-- The C compiler identification is GNU 4.8.2
-- The CXX compiler identification is GNU 4.8.2
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Configuring done
-- Generating done
-- Build files have been written to: /home/me/example/build/subProject
[  0%] Built target subProjectTarget
Scanning dependencies of target dummy
[100%] Building CXX object CMakeFiles/dummy.dir/a.cpp.o
Linking CXX executable dummy
[100%] Built target dummy

Note that the second time the cmake calls is on the subproject.

At the next call everything is even faster:

me@here:~/example/build$ make
-- Configuring done
-- Generating done
-- Build files have been written to: /home/me/example/build/subProject
[  0%] Built target subProjectTarget
Scanning dependencies of target dummy
[100%] Building CXX object CMakeFiles/dummy.dir/a.cpp.o
Linking CXX executable dummy
[100%] Built target dummy

(Although here the file b.h is written every time causing a.cpp to be recompiled)

This stub can be very much improved by using cmake commands to generate the output directories (and not mkdir) and cascading the generator chosen for the main project (here I am assuming everything is using make)

Let me know if you need any further clarification.

Answer 2

I think that ${codegenInputFiles} should contain a list of hardcoded source files and include files required by the custom command. Documentation of file(GLOB ...) states:

We do not recommend using GLOB to collect a list of source files from your source tree. If no CMakeLists.txt file changes when a source is added or removed then the generated build system cannot know when to ask CMake to regenerate.

The hard work (for which we are paid) is to keep the ${codegenInputFiles} up to date (causing the full project rebuild). Anyway, you would have similar problem if someone created a new source file and hadn't added it to the ${codegenInputFiles}, right? So I believe the additional dependency on include file should be treated the same.