CMake testing causing error when tests fail

Question

I've been looking around for a solution to this but cannot seem to find out why this happens. I use CMake for a small project and I wanted to add unit tests cmakes add_test. It works well when tests are passing but when they fail, I get an odd error message that I cannot interpret.

I've made a minimal example to illustrate what happens.

CMakeLists.txt

cmake_minimum_required (VERSION 2.6)

project (TestExample)

enable_testing()

add_executable(succeedExample succeed.cpp)
add_executable(failExample fail.cpp)

add_test(NAME succeedTest COMMAND succeedExample)
add_test(NAME failTest COMMAND failExample)

succeed.cpp

int main()
{
    return 0;
}

fail.cpp

int main()
{
    return 1;
}

I use Visual Studio 2015, and when building the RUN_TESTS target, I get the following error:

Severity    Code    Description Project File    Line    Suppression State
Error   MSB3073 The command "setlocal
"C:\Program Files (x86)\CMake\bin\ctest.exe" --force-new-ctest-process -C Debug
if %errorlevel% neq 0 goto :cmEnd
:cmEnd
endlocal & call :cmErrorLevel %errorlevel% & goto :cmDone
:cmErrorLevel
exit /b %1
:cmDone
if %errorlevel% neq 0 goto :VCEnd
:VCEnd" exited with code 8. RUN_TESTS   C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets  133

Yet if I change the fail.cpp to return 0; I get

1>------ Build started: Project: RUN_TESTS, Configuration: Debug Win32 ------
1>  Test project C:/Users/.../projects/cmaketestexample/build
1>      Start 1: succeedTest
1>  1/2 Test #1: succeedTest ......................   Passed    0.01 sec
1>      Start 2: failTest
1>  2/2 Test #2: failTest .........................   Passed    0.02 sec
1>
1>  100% tests passed, 0 tests failed out of 2
1>
1>  Total Test time (real) =   0.03 sec
========== Build: 1 succeeded, 0 failed, 1 up-to-date, 0 skipped ==========

Can anyone point me towards a solution to fix this? Am I missing something obvious?

Thanks :)

EDIT

Visual studio version is 14.0 (Visual Studio 2015 community)

Cmake 3.5.2 - yet I set the version to 2.6 in the CMakeLists.txt.

EDIT2

running ctest -C Debug on command line:

Test project C:/Users/.../projects/cmaketestexample/build
    Start 1: succeedTest
1/2 Test #1: succeedTest ......................   Passed    0.02 sec
    Start 2: failTest
2/2 Test #2: failTest .........................***Failed    0.02 sec

50% tests passed, 1 tests failed out of 2

Total Test time (real) =   0.07 sec

The following tests FAILED:
          2 - failTest (Failed)
Errors while running CTest

Changing fail.cpp to

#include <iostream>
int main()
{
    std::cout << "Does it work now?" << std::endl;
    return 1;
}

makes no difference.

EDIT3

A similar error occur on Ubuntu 14.04.4 LTS using

cmake version 3.2.2
GNU Make 3.81
g++ (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4

You could try to run `ctest -C Debug` on the command line in your binary output directory to see potential error messages. Or - just a guess - try to add any `stdout`/`stderr` output to your failing test and see if it makes a difference. For alternatives see e.g. [here](http://stackoverflow.com/questions/5709914/using-cmake-how-do-i-get-verbose-output-from-ctest). — Florian, Oct 09 '16 at 19:42
Please try CMake 3.6 and if that still fails, try 3.7-rc1. Visual Studio 2015 is maybe too new for CMAke 3.5. — usr1234567, Oct 10 '16 at 07:45
I updatted the question yet again. I tried on Ununtu using make and g++, still getting an error. I also tried setting the CMake version higher without luck :/ — user1291510, Oct 10 '16 at 07:56

score 2 · Accepted Answer · edited May 23 '17 at 10:27

I've given you code a try and what you see is the little script that CMake embeds the ctest call into. And ctest does fail - return a non-zero value - if any of tests it is executing does fail. And this behavior of ctest is by intention to stop any scripts/builds running the tests if encountering errors.

The possible error return codes of ctest are a combination of the following bits:

// provide some more detailed info on the return code for ctest
enum {
  UPDATE_ERRORS    = 0x01,
  CONFIGURE_ERRORS = 0x02,
  BUILD_ERRORS     = 0x04,
  TEST_ERRORS      = 0x08,
  MEMORY_ERRORS    = 0x10,
  COVERAGE_ERRORS  = 0x20,
  SUBMIT_ERRORS    = 0x40
};

So I see the following options:

Ignore the strange error, the output you're looking for is just above the error message:

1>------ Build started: Project: RUN_TESTS, Configuration: Debug Win32 ------
1>  Test project C:/temp/StackOverflow/TestError/VS2015
1>      Start 1: succeedTest
1>  1/2 Test #1: succeedTest ......................   Passed    0.03 sec
1>      Start 2: failTest
1>  2/2 Test #2: failTest .........................***Failed    0.03 sec
1>
1>  50% tests passed, 1 tests failed out of 2
1>
1>  Total Test time (real) =   0.10 sec
1>
1>  The following tests FAILED:
1>        2 - failTest (Failed)
1>  Errors while running CTest
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: The command "setlocal
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: "C:\Program Files\CMake\bin\ctest.exe" --force-new-ctest-process -C Debug
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: if %errorlevel% neq 0 goto :cmEnd
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: :cmEnd
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: endlocal & call :cmErrorLevel %errorlevel% & goto :cmDone
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: :cmErrorLevel
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: exit /b %1
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: :cmDone
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: if %errorlevel% neq 0 goto :VCEnd
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: :VCEnd" exited with code 8.
========== Build: 0 succeeded, 1 failed, 1 up-to-date, 0 skipped ==========

Add your own custom target calling ctest on VS without a script, giving just one error:

if (CMAKE_CONFIGURATION_TYPES)
    add_custom_target(
        RUN_CTEST
            COMMAND ${CMAKE_CTEST_COMMAND} -C $<CONFIGURATION> --force-new-ctest-process 
            WORKING_DIRECTORY "${CMAKE_BINARY_DIR}"
    )
endif()

Add your own custom target command that does ignore the return value by moving the ctest call into a separate script (see references).

References

I feel I might overlook something, but in my visual studio, the output is replaced with the errors, so I do not see the output, can I force this? Also, the error is 8, so I guess its the TEST_ERRORS enum. But it all seems odd to me, shoudl CMake not be able to evaluate tests to fail without failing itself? — user1291510, Oct 10 '16 at 20:36
@user1291510 Yes, in Visual Studio's "Error List" window you won't see the test results. You need to check the "Build Output" window. And yes, I just checked the `ctest` code and there is no option to *not* fail when a test fails. — Florian, Oct 10 '16 at 20:49

CMake testing causing error when tests fail

1 Answers1