Expected build-failure tests in CMake

Question

Sometimes it's good to check that certain things fail to build, e.g.:

// Next line should fail to compile: can't convert const iterator to iterator.
my_new_container_type::iterator it = my_new_container_type::const_iterator();

Is it possible to incorporate these types of things into CMake/CTest? I'm looking for something like this in CMakeLists.txt:

add_build_failure_executable(
    test_iterator_conversion_build_failure
    iterator_conversion_build_failure.cpp)
add_build_failure_test(
    test_iterator_conversion_build_failure
    test_iterator_conversion_build_failure)

(Of course, these specific CMake directives don't exist, to the best of my knowledge.)

Note that your specific example can be tested in C++ directly with `static_assert(!std::is_convertible_v and !std::is_constructible_v);`, testing that both implicit and explicit conversions fail. This has the advantage of correctly failing your tests if the member types do not exist. — David Stone, Mar 29 '19 at 23:28

score 45 · Accepted Answer · answered May 12 '15 at 12:56

You can do this more or less as you described. You can add a target which will fail to compile, then add a test which invokes cmake --build to try to build the target. All that remains is to set the test property WILL_FAIL to true.

So, say you have your tests in a file named "will_fail.cpp" which contains:

#if defined TEST1
non-compiling code for test 1
#elif defined TEST2
non-compiling code for test 2
#endif

Then you can have something like the following in your CMakeLists.txt:

cmake_minimum_required(VERSION 3.0)
project(Example)

include(CTest)

# Add a couple of failing-to-compile targets
add_executable(will_fail will_fail.cpp)
add_executable(will_fail_again will_fail.cpp)
# Avoid building these targets normally
set_target_properties(will_fail will_fail_again PROPERTIES
                      EXCLUDE_FROM_ALL TRUE
                      EXCLUDE_FROM_DEFAULT_BUILD TRUE)
# Provide a PP definition to target the appropriate part of
# "will_fail.cpp", or provide separate files per test.
target_compile_definitions(will_fail PRIVATE TEST1)
target_compile_definitions(will_fail_again PRIVATE TEST2)

# Add the tests.  These invoke "cmake --build ..." which is a
# cross-platform way of building the given target.
add_test(NAME Test1
         COMMAND ${CMAKE_COMMAND} --build . --target will_fail --config $<CONFIGURATION>
         WORKING_DIRECTORY ${CMAKE_BINARY_DIR})
add_test(NAME Test2
         COMMAND ${CMAKE_COMMAND} --build . --target will_fail_again --config $<CONFIGURATION>
         WORKING_DIRECTORY ${CMAKE_BINARY_DIR})
# Expect these tests to fail (i.e. cmake --build should return
# a non-zero value)
set_tests_properties(Test1 Test2 PROPERTIES WILL_FAIL TRUE)

You can obviously wrap all of this into a function or macro if you have a lot of these to write.

May I ask how you know all this stuff? I like cmake, but as opposed to other build systems, I never found a comprehensive online tutorial for it. — Ami Tavory, May 12 '15 at 14:59
Mainly working with CMake in a medium sized codebase, following the Mailing List, and having a go at answering StackOverflow questions :) The docs are pretty good now, and are being improved, although CTest's docs are way behind CMake's in my opinion. — Fraser, May 12 '15 at 15:22
Nice. Note that if we want to check for a precise error (like the one resulting of a failed static assertion), we can set the test property: `PASS_REGULAR_EXPRESSION`. `WILL_FAIL` shall not be used simultaneously. — Luc Hermitte, Mar 13 '16 at 14:31
Watch out. The WILL_FAIL option don't work as expected if your program crash or call abort. Which mean that for example a failing assert using munit (https://nemequ.github.io/munit/) will still fail your test even with WILL_FAIL. — ninjaconcombre, Nov 13 '20 at 21:36

score 14 · Answer 2 · answered Jun 03 '18 at 11:21

@Fraser's answer is a good approach, in particular the WILL_FAIL property is good advice. There is an alternative to making the failing target part of the main project though. The use case in the question is pretty much what the ctest --build-and-test mode is meant for. Rather than making the expected-to-fail target part of the main build, you can put it in its own separate mini project which is then built as part of a test. An example of how this might look in the main project goes something like this:

add_test(NAME iter_conversion
    COMMAND ${CMAKE_CTEST_COMMAND}
            --build-and-test
                ${CMAKE_CURRENT_LIST_DIR}/test_iter
                ${CMAKE_CURRENT_BINARY_DIR}/test_iter
            --build-generator ${CMAKE_GENERATOR}
            --test-command ${CMAKE_CTEST_COMMAND}
)
set_tests_properties(iter_conversion PROPERTIES WILL_FAIL TRUE)

This has the advantage that it will be part of the project's test results and will therefore be more likely to get executed regularly as part of normal testing processes. In the above example, the test_iter directory is essentially it's own separate project. If you need to pass information to it from the main build, you can do that by adding --build-options to define cache variables to pass to it's CMake run. Check the latest docs for recently corrected/clarified help on this area.

Just wanted to mention that this also very helpful if you want to group multiple test cases from the mini project to a single test in the main project. You might want to set `--build-config $` as well. — oreiche, Jan 10 '21 at 15:13

score 1 · Answer 3 · answered Oct 05 '22 at 07:14

For the particular example in the question I agree with the comment. This is should be tested with a static_assert rather than a build failure test.

For a way of adding build failure test with CMake (in the cases where this is a good idea) I recently created a library which more or less allows adding such tests with a single CMake function call. It expands on the accepted answer and allows build-failure test and checks that the failure happened because of a specific error (the expected error can be provided in CMake or in a source file): https://github.com/iboB/icm/blob/master/icm_build_failure_testing.cmake

Expected build-failure tests in CMake

3 Answers3

Linked