1

From the question Defining lex include files for (f)lexers we are already knowing that there is no lex distribution known that has an include file facility for e.g. rules into a lex file.

The suggestion there was to write an own pre-processor.

When defining an own pre-processor one generates, of course, dependencies on the included file(s), so when such a file is changed one wants to do a rebuild of the "lex" file. With C / C++ the dependencies are handled (to the best of my knowledge) automatically (probably by some CMake script).

  • How to accomplish the dependency rules for the file(s) included in a lex file?

Edit

Say I have an original .l file configimpl.l that includes the files aa.inc and bb.inc, by converting the original .l file (${CMAKE_CURRENT_LIST_DIR}/${lex_file}.l) by means of a python script into a new .l (${GENERATED_SRC}/${lex_file}.l) file (and we also have an output file ${GENERATED_SRC}/${lex_file}.corr but that is not relevant here). Of course all "cmake" variables etc. have been set properly.

My code for the preprocessing (i.e. including the include files into the .l file) is:

    add_custom_command(
        COMMAND ${PYTHON_EXECUTABLE} ${CMAKE_CURRENT_LIST_DIR}/pre_lex.py ${CMAKE_CURRENT_LIST_DIR}/${lex_file}.l  ${GENERATED_SRC}/${lex_file}.l ${GENERATED_SRC}/${lex_file}.corr ${CMAKE_CURRENT_LIST_DIR}
        DEPENDS ${CMAKE_CURRENT_LIST_DIR}/pre_lex.py ${CMAKE_CURRENT_LIST_DIR}/${lex_file}.l ${LEX_INC_FILES}
        OUTPUT  ${GENERATED_SRC}/${lex_file}.corr ${GENERATED_SRC}/${lex_file}.l
    )

and this works fine, except when the aa.inc or the bb.inc changes the "generated included" ${GENERATED_SRC}/${lex_file}.l is not rebuild.

Based on the question: Dynamic dependency within custom source files I already tried to create something, but failed.

    add_custom_command(
        COMMAND ${PYTHON_EXECUTABLE} ${CMAKE_CURRENT_LIST_DIR}/dep_lex.py ${CMAKE_CURRENT_LIST_DIR}/${lex_file}.l  ${GENERATED_SRC}/${lex_file}.d ${GENERATED_SRC}/${lex_file}.l ${CMAKE_CURRENT_LIST_DIR}
        DEPENDS ${CMAKE_CURRENT_LIST_DIR}/dep_lex.py ${CMAKE_CURRENT_LIST_DIR}/${lex_file}.l
        OUTPUT  ${GENERATED_SRC}/${lex_file}.d
    )
    set_source_files_properties(${GENERATED_SRC}/${lex_file}.d PROPERTIES GENERATED 1)
    add_custom_command(
        COMMAND ${PYTHON_EXECUTABLE} ${CMAKE_CURRENT_LIST_DIR}/pre_lex.py ${CMAKE_CURRENT_LIST_DIR}/${lex_file}.l  ${GENERATED_SRC}/${lex_file}.l ${GENERATED_SRC}/${lex_file}.corr ${CMAKE_CURRENT_LIST_DIR}
        DEPENDS ${CMAKE_CURRENT_LIST_DIR}/pre_lex.py ${CMAKE_CURRENT_LIST_DIR}/${lex_file}.l
        DEPFILE ${GENERATED_SRC}/${lex_file}.d
        OUTPUT  ${GENERATED_SRC}/${lex_file}.l ${GENERATED_SRC}/${lex_file}.corr
    )

So I first run a dependency detection step that creates a file ${GENERATED_SRC}/${lex_file}.d with as content:

${GENERATED_SRC}/generated_src/configimpl.l: ${CMAKE_CURRENT_LIST_DIR}/aa.inc
${GENERATED_SRC}/generated_src/configimpl.l: ${CMAKE_CURRENT_LIST_DIR}/bb.inc

in this generated file the paths are, of course, set to the full paths. After this the original preprocessing step is run but now with the extra DEPFILE ${GENERATED_SRC}/${lex_file}.d that should signal that there are some extra dependencies.

When I change the aa.inc or the bb.inc the ${GENERATED_SRC}/${lex_file}.l is not rebuild, what did I miss?

Used CMake version: cmake version 3.22.1 used generator: NMake Makefiles

albert
  • 8,285
  • 3
  • 19
  • 32
  • Such long lines, they could use newlines. `Say I have an original .l file` I do not understand the sentence. What is the input, what is the output, and what is going on. `what did I miss?` DEPFILE is a byproduct of the command, not a separate stage. _One_ command, generates the output file _and_ at the same time generates the DEPFILE. I can only throw you at my M4 adventure: https://gitlab.com/Kamcuk/kcmakelib/-/blob/master/cmake/k/m4.sh#L11 and https://gitlab.com/Kamcuk/kcmakelib/-/blob/master/cmake/k/m4.cmake#L93 . – KamilCuk Jun 11 '22 at 09:49
  • `is not rebuild, what did I miss?` I do not understand, but there can't be any `${GENERATED_SRC}` variable expansions in depfile, and at best, it's best to keep all paths absolute. Also, it's best, at least to my experience with Make and Ninja, to have _one line_. Make it one long line. (I remember some parser using a hashmap and overwriting the previous entry) – KamilCuk Jun 11 '22 at 09:57
  • @KamilCuk thanks for the comment the `{GENERATED_SRC}` is expanded but the part `Also, it's best, at least to my experience with Make and Ninja, to have one line.` looks like to be the solution. I tried it, quickly, manually in my `.d` file and this looks like to work!! Now I have to implement this in my `dep_lex.py` script.:w – albert Jun 11 '22 at 10:02
  • @KamilCuk I just tested with my `dep_lex.py` file and the `one line` approach works, can you make an answer out of it? (and maybe also add this information to to your original answer, https://stackoverflow.com/a/62713715/1657886, it is implied here but when it is more explicit ...) – albert Jun 11 '22 at 10:08

1 Answers1

1

Make the DEPFILE output as one long line, then it works the best. From memory, I think the parser for the depfile is not smart enough to handle multiple lines. When in doubt - see the output of -MT -MD of gcc, the format should be the same.

KamilCuk
  • 120,984
  • 8
  • 59
  • 111