Unit testing as part of the build

Question

I have a CMake project separated in three parts:

Compilation of my libraries
Compilation of many unit tests programs testing each precise subpart of those libraries
Compilation of program samples using those libraries

My question is about the 2^nd part. My unit tests executables are short binary programs with a main returning 0 on success, 1 on fail. I would like to integrate their running as part of the build.

Is it possible to use CMake to do one of those two solutions:

Run every unit test program and check return values and generate an error if any fail.
Generate a test script that would run and check every unit test program and check return values to yield an error if any fail.

I'm not looking for a complete CMake script code, a simple indication on what is possible with a link to corresponding documentation would be enough.

score 15 · Accepted Answer · edited May 23 '17 at 11:33

15

See a similar problem and my answer here.

Mainly my recommendation is to add a POST_BUILD step to your unit test targets that runs ctest. If a POST_BUILD step does fail (return code is not 0), the build will fail.

Something like:

set(UNIT_TEST MyLibUnitTestTargetName)
add_test(NAME ${UNIT_TEST} COMMAND ${UNIT_TEST})
add_custom_command(
     TARGET ${UNIT_TEST}
     COMMENT "Run tests"
     POST_BUILD 
     WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
     COMMAND ${CMAKE_CTEST_COMMAND} -C $<CONFIGURATION> -R "^${UNIT_TEST}$" --output-on-failures
)

This has the advantage that it runs like you would call ctest.

The short version - without add_test() / ctest - would be:

set(UNIT_TEST MyLibUnitTestTargetName)
add_custom_command(
     TARGET ${UNIT_TEST}
     COMMENT "Run tests"
     POST_BUILD 
     COMMAND ${UNIT_TEST}
)

References:

edited May 23 '17 at 11:33

Community

1
1

answered Oct 02 '15 at 07:25

Florian

39,996
9
133
149

There's no need for the genex in `add_test`. Target names are automatically understood to mean the executable produced by the target. – Angew is no longer proud of SO Oct 02 '15 at 07:36
@Angew Thanks for the hint. Changed it to directly use the target name as `COMMAND` for `add_test()` and the short `add_custom_command()` version. – Florian Oct 02 '15 at 07:53
4

This code causes binary removal, in case of tests failure for _make_ and _ninja_ – ivaigult Apr 27 '17 at 14:40
I'm not sure why, but I had to add a space like this: `-R "^${UNIT_TEST}$ "`. Otherwise `--output-on-failures` was getting treated as part of the regex resulting in "No tests were found!!!" – jacobq Aug 29 '19 at 10:17

score 4 · Answer 2 · answered Aug 27 '19 at 09:06

If you want your test binary to be preserved, even if tests fail, you can try something like this:

set(UNIT_TEST MyLibUnitTestTargetName)
add_test(NAME ${UNIT_TEST} COMMAND ${UNIT_TEST})
add_custom_target(run_unit_test ALL
    COMMAND ${CMAKE_CTEST_COMMAND} --output-on-failure
    DEPENDS ${UNIT_TEST})

This will create a target that always runs your unit tests. However, because it's a separate target failure of that command will result in build failure, but not in removing the binary produced by your original UNIT_TEST target.

Unit testing as part of the build

2 Answers2

Linked