"ctest" versus "make check": bad build time versus broken option passing

Question

To register tests under CMake, we need

enable_testing()

or

include(CTest)

and then for each single test (name fooTest, executable foo)

add_executable(foo <foo_sources>)
add_test(fooTest foo)

Tests can then be run with the command ctest.

Additionally, we can run tests with the command make check, provided we add once

add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND})

and for each test we extend the above by a keyword EXCLUDE_FROM_ALL and a command add_dependencies:

add_executable(foo EXCLUDE_FROM_ALL <foo_sources>)
add_test(fooTest foo)
add_dependencies(check foo)

Ideally, this would make make check an alias of ctest. It does not so for at least two reasons:

(1) make check is flawed because it does not pass options to ctest [2]. In particular, ctest -j4 will run 4 tests in parallel, whereas make -j4 check will work in one thread on target check, and the other three threads will remain idle.

(2) ctest is flawed [3,4] because all tests are build under the all target, i.e. along with the main application. This may be desired behavior in some situations, but in other situations it ought to be possible to postpone the build until the tests are to be run.

Does this correctly summarize the current state of affairs?

Is there any way around (to eat the cake and have it)?

[1] https://gitlab.kitware.com/cmake/community/-/wikis/doc/tutorials/EmulateMakeCheck [2] http://comments.gmane.org/gmane.comp.programming.tools.cmake.user/47300 [3] CMake & CTest : make test doesn't build tests [4] http://public.kitware.com/Bug/view.php?id=8774

Answer 1

First, let me remark that ctest and make test are only simple command line tools, for simple testing tasks. If you want a tool for serious testing, use CDash, Buildbot, Jenkins or whatever.

Concerning the flaws of CTest: It is intentional, that the call for CTest does not build the tests. It is a bad idea in several scenarios:

• Compiling tests can take more resources then running the tests itself. This might be true with respect to memory consumption, read/writes to the hard disk or compilation time. So compiling and linking in parallel might be bad, but executing the tests in parallel might be beneficial.
• How to handle compilation or linking failure? Report it as failing? Report is as not compiling? Continuing with compiling the other tests or aborting immediately?

Autotools did it the way you want it and people got used to it. But why should it be a unit? Why not having two commands? What's the benefit of mixing two tasks and making it more difficult for project with special needs?

I came to the conclusion, to create a target build-tests or similar, and follow the decision made by the CMake developers to decouple building test and executing tests. Then I can decide whether I want parallel builds, how to treat compilation failures (e.g., passing -k to make) and so on.
The only downside is, that this target is only present in the top level directory and cannot be used in sub-directories.
To get such a target built-in by CMake would be a good feature request. Ranting on SO does no good.

Answer 2

CTest is not flawed at all, but the way you use CMake and CTest seems "flawed". The invocation of the command line interface (CLI) tool ctest is in general not related to the invokation of a CMake build targets (with the exception of the target test).

In my opinion the custom check target solution described in the CMake Wiki should not be used, since it changes the default behavior of CMake and is not configurable.

Instead the following approach using the built-in option BUILD_TESTING should be used:

include(CTest)
if(BUILD_TESTING)
  find_package(GTest MODULE REQUIRED)
  add_executable(example_test example_test.cpp)
  target_link_libraries(
    example_test
    PRIVATE
      GTest::GTest
      GTest::Main
  )
  add_test(NAME example_test COMMAND example_test)
endif()

include(CTest) defines in the option BUILD_TESTING, which allows to control whether to build all tests of the project or not.

Quote from the official documentation:

CMake will generate tests only if the enable_testing() command has been invoked. The CTest module invokes the command automatically when the BUILD_TESTING option is ON.

The above can be used on the CLI as follows:

1. Create tests (default):

    cmake -Hexample-testing -B_builds/example-testing/release -G"Unix Makefiles" -DCMAKE_BUILD_TYPE=Release
    cmake --build _builds/example-testing/release --config Release
   

   In this case the commands cd _builds/example-testing/release and ctest / cmake --build . --target test build and run the test(s).

2. Do not create tests, setting -DBUILD_TESTING=OFF:

    cmake -Hexample-testing -B_builds/example-testing/release-no-tests -G"Unix Makefiles" -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF
    cmake --build _builds/example-testing/release-no-tests --config Release
   

   In this case the commands cd _builds/example-testing/release-no-tests and ctest run no test(s), since no test(s) have been built. The command cmake --build . --target test fails since it has not been created during configure phase of CMake.

We are only scratching the surface here. Refer to ctest --help, e.g. there are a lot of --build-<...> options that allow finer control regarding testing/building, though I have not any experience with that.

I highly recommend reading the following:

• CTest - CMake Documentation
• CTest Commands - CMake Documentation
• ctest(1) - CMake Documentation
• add_test - CMake Documentation

If you really want to enable building of tests, but via a separate target that is not invoked by default and run the test not via CTest but directly you can do the following:

include(CTest)
if(BUILD_TESTING)
  find_package(GTest MODULE REQUIRED)
  option(
    BUILD_TESTING_EXCLUDE_FROM_ALL
    "Do not build the testing tree together with the default build target."
    OFF
  )

  if(BUILD_TESTING_EXCLUDE_FROM_ALL)
    set(add_executable_args_for_test EXCLUDE_FROM_ALL)
  endif()

  # The "build_test" target is used to build all test executables.
  add_custom_target(
    build_test
    # Workaround for printing the COMMENT, it does not work without a NOOP
    # COMMAND.
    COMMAND ${CMAKE_COMMAND} -E echo
    COMMENT "Building tests..."
    VERBATIM
  )
  add_executable(example_test ${add_executable_args_for_test} example_test.cpp)
  target_link_libraries(
    example_test
    PRIVATE
      GTest::GTest
      GTest::Main
  )
  add_test(NAME example_test COMMAND example_test)
  add_dependencies(build_test example_test)

  # The "check" target is used to build AND run all test executables.
  add_custom_target(
    check
    # Either invoke the test(s) indirectly via "CTest" (commented) or directly.
#   COMMAND ${CMAKE_CTEST_COMMAND}
    COMMAND example_test
    COMMENT "Building and running test..."
    VERBATIM
  )
  # Alternative to the COMMAND in the add_custom_target. Leads to the same
  # behavior as calling "CTest" directly.
#  add_custom_command(
#    TARGET check
#    COMMAND ${CMAKE_COMMAND} ARGS --build ${CMAKE_BINARY_DIR} --target test
#    VERBATIM
#  )
   add_dependencies(check build_test)
endif()

• Note that the above code does not invoke CTest or the target test in order to run the test, but the test directly.
• Please read the comments and the commented code for alternative approaches using CTest that are similar to the approach described in the question.
• It's easy to enhance the above code to support more than one test executable.

IMHO, Kitware should remove the entire CMake Wiki, since the Wiki contains almost only out-of-date information for CMake versions < 3.0. Most information in it cannot be considered as Modern CMake.