38

I run unit tests by building with ninja and then running ninja test from my build tree:

cmake -G Ninja /source/tree
ninja
ninja test

However, to run valgrind I need to run it manually:

valgrind rel/path/to/test

I'd like valgrind to run automatically when I run ninja test. According to the cmake documentation "setting up [valgrind tests] is extremely easy", but when I run

ctest -D NightlyMemoryCheck

I just get this error:

Cannot find file: /home/arman/tinman/deb/DartConfiguration.tcl
   Site: 
   Build name: (empty)
WARNING: No nightly start time found please set in CTestConfig.cmake or DartConfig.cmake
Problem initializing the dashboard.

I get a similar error when I follow the instructions from this SO question:

How do I make ctest run a program with valgrind without dart?

I don't know what dart is, but according the website it's some kind of online testing doodad.

Clearly extremely easy is not easy enough for me. Does anyone know of a solution that is so supremely easy that you'd have to be some kind of IT warlock to make it not work?

starball
  • 20,030
  • 7
  • 43
  • 238
quant
  • 21,507
  • 32
  • 115
  • 211
  • I think you need to use the `ctest_memcheck` command, cf. https://cmake.org/cmake/help/v3.7/command/ctest_memcheck.html#command:ctest_memcheck and https://cmake.org/cmake/help/v3.7/manual/ctest.1.html?highlight=valgrind#ctest-memcheck-step – usr1234567 Nov 05 '16 at 09:27

1 Answers1

64

Here is a self contained example that shows how to add valgrind tests to a CMake project. The example consists of a single C++ source file main.cpp:

#include <iostream>

int main()
{
    double* leak = new double[10];
    std::cout << "Hello!" << std::endl;
}

The code contains an intentional leak which should be picked up by valgrind. We also need a CMakeLists.txt file which requires CMake >= 2.8:

cmake_minimum_required(VERSION 2.8)

project (ValgrindExample)

include (CTest)
add_executable(example main.cpp)
add_test(example_test example)

Here it is important to include the CTest module with include instead of just enabling tests with enable_testing(). The CTest module takes care of setting up the machinery for being able to run memory checks with tests (e.g., it locates the valgrind executable).

Now we can open a shell session in the project folder and create a Ninja build tree:

$ mkdir build; cd build
$ cmake -G Ninja -DCMAKE_BUILD_TYPE=Debug ..

We can build and run tests without valgrind in the regular way:

$ ninja
[2/2] Linking CXX executable example
$ ninja test
[0/1] Running tests...
...
100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.01 sec

To run tests with valgrind we have to use CMake's ctest executable with the test action memcheck:

$ ctest -T memcheck
...
1/1 MemCheck #1: example_test .....................   Passed    0.77 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.77 sec
-- Processing memory checking output: 
Memory checking results:
Memory Leak - 2

ctest prints a summary of the memory checking results. The detailed output of valgrind is located in a temporary directory in the build tree:

$ cat ./Testing/Temporary/MemoryChecker.*.log
==4565== 80 bytes in 1 blocks are definitely lost in loss record 37 of 64
==4565==    at 0x10000B681: malloc (in /usr/local/Cellar/valgrind/3.12.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)
==4565==    by 0x1000507DD: operator new(unsigned long) (in /usr/lib/libc++.1.dylib)
==4565==    by 0x100000F93: main (main.cpp:5)
...

It is not possible to automatically run valgrind when you run ninja test because CMake's built-in test target cannot be modified and always runs tests in the regular way. We can however add a custom CMake target which invokes ctest with the -T memcheck option and then prints the detailed valgrind report:

add_custom_target(test_memcheck
    COMMAND ${CMAKE_CTEST_COMMAND} 
        --force-new-ctest-process --test-action memcheck
    COMMAND ${CMAKE_COMMAND} -E cat "${CMAKE_BINARY_DIR}/Testing/Temporary/MemoryChecker.*.log"
    WORKING_DIRECTORY "${CMAKE_BINARY_DIR}")

--test-action is the verbose version of the switch -T.

We then can invoke valgrind testing from Ninja with

$ ninja test_memcheck

and get the results as if we ran valgrind manually.

sakra
  • 62,199
  • 16
  • 168
  • 151
  • 1
    So there is a default command used in this example with test-action memcheck. If I were to want to specify a custom suppressions file, or using custom parameters all together to pass to valgrind(such as running a full leak check), how would one go about doing so? – Dwight Jan 17 '17 at 02:39
  • Set the variable `CTEST_MEMORYCHECK_SUPPRESSIONS_FILE`. See https://cmake.org/cmake/help/v3.7/variable/CTEST_MEMORYCHECK_SUPPRESSIONS_FILE.html – sakra Jan 17 '17 at 08:43
  • 2
    How to cat valgrind log only when valgrind check failed? How to exit with -1? – johnsam Oct 01 '18 at 23:39
  • 4
    @sakra How do you set the `CTEST_MEMORYCHECK_SUPPRESSIONS_FILE` variable? I tried to use the `-D` option to ctest, but that doesn't seem to do anything. Any ideas? I really want to be able to set the suppressions file and the memcheck options from the command line when running ctest. – dcmm88 Oct 09 '18 at 23:57
  • 1
    The custom target is missing `WORKING_DIRECTORY "${CMAKE_BINARY_DIR}"`. – Peter Sep 11 '20 at 08:40
  • 1
    @dcmm88 I got this working by using `MEMORYCHECK_SUPPRESSIONS_FILE` instead. – jyn Oct 28 '20 at 18:44