31

I'm using CTest and want to pass command-line arguments to the underlying tests at runtime. I know there are ways to hard code command-line arguments into the CMake/CTest script, but I want to specify the command-line arguments at runtime and have those arguments passed through CTest to the underlying test.

Is this even possible?

simon
  • 1,125
  • 1
  • 10
  • 20
jlconlin
  • 14,206
  • 22
  • 72
  • 105

3 Answers3

9

I've figured out a way to do it (using the Fundamental theorem of software engineering). It's not as simple as I'd like, but here it is.

First, create a file ${CMAKE_SOURCE_DIR}/cmake/RunTests.cmake with the content

if(NOT DEFINED ENV{TESTS_ARGUMENTS})
    set(ENV{TESTS_ARGUMENTS} "--default-arguments")
endif()
execute_process(COMMAND ${TEST_EXECUTABLE} $ENV{TESTS_ARGUMENTS} RESULT_VARIABLE result)
if(NOT "${result}" STREQUAL "0")
    message(FATAL_ERROR "Test failed with return value '${result}'")
endif()

Then, when you add the test, use

add_test(
    NAME MyTest
    COMMAND ${CMAKE_COMMAND} -DTEST_EXECUTABLE=$<TARGET_FILE:MyTest> -P ${CMAKE_SOURCE_DIR}/cmake/RunTests.cmake
)

Finally, you can run the test with custom arguments using

cmake -E env TESTS_ARGUMENTS="--custom-arguments" ctest

Note that if you use bash, you can simplify this to

TESTS_ARGUMENTS="--custom-arguments" ctest

There are some problems with this approach, e.g. it ignores the WILL_FAIL property of the tests. Of course I wish it could be as simple as calling ctest -- --custom-arguments, but, as the Stones said, You can't always get what you want.

Kevin
  • 1,064
  • 12
  • 14
  • 4
    I've made a feature requrest to CMake here: https://gitlab.kitware.com/cmake/cmake/issues/20470 – AMA Mar 18 '20 at 12:36
0

I'm not sure I fully understand what you want, but I still can give you a way to pass arguments to tests in CTest, at runtime.

I'll give you an example, with CTK (the Common Toolkit, https://github.com/commontk/CTK):

In the build dir (ex: CTK-build/CTK-build, it's a superbuild), if I run: ('-V' for Verbose, and '-N' for View Mode only)

ctest -R ctkVTKDataSetArrayComboBoxTest1 -V -N

I get:

UpdateCTestConfiguration  from : /CTK-build/CTK-build/DartConfiguration.tcl
Parse Config file:/CTK-build/CTK-build/DartConfiguration.tcl
 Add coverage exclude regular expressions.
 Add coverage exclude: /CMakeFiles/CMakeTmp/
 Add coverage exclude: .*/moc_.*
 Add coverage exclude: .*/ui_.*
 Add coverage exclude: .*/Testing/.*
 Add coverage exclude: .*/CMakeExternals/.*
 Add coverage exclude: ./ctkPixmapIconEngine.*
 Add coverage exclude: ./ctkIconEngine.*
UpdateCTestConfiguration  from :/CTK-build/CTK-build/DartConfiguration.tcl
Parse Config file:/CTK-build/CTK-build/DartConfiguration.tcl
Test project /CTK-build/CTK-build
Constructing a list of tests
Done constructing a list of tests

178: Test command: /CTK-build/CTK-build/bin/CTKVisualizationVTKWidgetsCppTests "ctkVTKDataSetArrayComboBoxTest1"
Labels: CTKVisualizationVTKWidgets
  Test #178: ctkVTKDataSetArrayComboBoxTest1

Total Tests: 1

You can copy-paste the "Test command" in your terminal:

/CTK-build/CTK-build/bin/CTKVisualizationVTKWidgetsCppTests "ctkVTKDataSetArrayComboBoxTest1"

And add the arguments, for example "-I" for interactive testing:

/CTK-build/CTK-build/bin/CTKVisualizationVTKWidgetsCppTests "ctkVTKDataSetArrayComboBoxTest1" "-I"

Tell me if it helps.

matthieu
  • 1,412
  • 1
  • 11
  • 33
0

matthieu's answer gave me the clue to get it to work for me.

For my code I did the following:

Type the command ctest -V -R TestMembraneCellCrypt -N to get the output:

...
488: Test command: path/to/ctest/executable/TestMembraneCellCrypt
Labels: Continuous_project_ChasteMembrane
  Test #488: TestMembraneCellCrypt
...

Then I copied the Test command and provided the arguments there:

path/to/ctest/executable/TestMembraneCellCrypt -e 2 -em 5 -ct 10

I'll note that the package I'm using (Chaste), is pretty large so there might be things going on that I don't know about.

Bamboo
  • 973
  • 7
  • 17
  • This however doesn't execute the test in the exact same environment as ctest would, e.g. if environment variables have to be set. – David Faure Oct 01 '20 at 12:19
  • Depends on how the CMakeLists.txt are written. In our project, the test command also contains setting the environment variables. This was done specifically to make ctest -N more useful. – user7610 Oct 21 '20 at 09:45