How to force run a disabled test from ctest?

Question

I have a c++ project that leverages cmake, ctest, and gtest.

One of my tests needs to be disabled for the purposes of regressions, but I would like to run it manually.

I disabled it by using the gtest trick of naming the test DISABLED_test_name.

When I try to run it, I see the following:

$ ctest -R aie2_210a_ddr_2kernel_ddr_gmio_bd_api.maize_xrt --verbose
UpdateCTestConfiguration  from :/scratch/bryanloz/aie_controller_hax/test/build/DartConfiguration.tcl
Parse Config file:/scratch/bryanloz/aie_controller_hax/test/build/DartConfiguration.tcl
UpdateCTestConfiguration  from :/scratch/bryanloz/aie_controller_hax/test/build/DartConfiguration.tcl
Parse Config file:/scratch/bryanloz/aie_controller_hax/test/build/DartConfiguration.tcl
Test project /scratch/bryanloz/aie_controller_hax/test/build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 8
    Start 8: aie2_210a_ddr_2kernel_ddr_gmio_bd_api.maize_xrt
1/1 Test #8: aie2_210a_ddr_2kernel_ddr_gmio_bd_api.maize_xrt ...***Not Run (Disabled)   0.00 sec
No tests were found!!!

How do I override this behavior?

Add the correct options to the `add_test` command? https://github.com/google/googletest/blob/main/docs/advanced.md#temporarily-enabling-disabled-tests — fabian, Sep 21 '22 at 17:04
@ixSci I use the CMake's GoogleTest module which provides the gtest_discover_tests macro: https://cmake.org/cmake/help/latest/module/GoogleTest.html — CraigDavid, Sep 22 '22 at 17:05
As a quick workaround/hack you can locate a file named like `ProjectUnderTest[1]_tests.cmake` where your generated files are, open it, locate your test name there in line which looks something like this `set_tests_properties(YourTestName PROPERTIES DISABLED TRUE)` and change `TRUE` to `FALSE`. This file is autogenerated so it will be reverted whenever the file containing `gtest_discover_tests` is changed (or cmake rerun). There have to be an easier and more robust way involving CTest script but I can't come up with the solution offhand, it needs verification. — ixSci, Sep 22 '22 at 17:38

Ari · Answer 1 · 2022-09-22T00:54:59.877

3

You can use --gtest_also_run_disabled_tests. This command line flag forcibly runs all the DISABLED_ tests.

Note that if you have multiple disabled tests and you only want to enable one of them using the command line, you can combine this flag with --gtest_filter.

Edit for CTest:

I am not familiar with CTest, but apparently there is not an easy way to pass command-line arguments in CTest at runtime. Maybe you can use this workaround to do so, or find the executable and pass the argument to it directly See this answer.

As an alternative to passing the command line argument, you can set the GTEST_ALSO_RUN_DISABLED_TESTS environment variable to a value other than 0. See this.

edited Sep 22 '22 at 00:54

answered Sep 21 '22 at 17:04

Ari

7,251
11
40
70

ctest intercepts the command line args, and doesn't forward them to the executable (gtest main). i.e. This doesn't work: ctest -R aie2_210a_ddr_2kernel_ddr_gmio_bd_api.maize_xrt --verbose --gtest_also_run_disabled_tests – CraigDavid Sep 22 '22 at 00:14
1

@WilderField I edited my answer with some suggestions. – Ari Sep 22 '22 at 00:55
If OP is using automatic capabilities of the test discovery of GoogleTest CMake module nothing described it this answer will help. – ixSci Sep 22 '22 at 04:52
@Ari the ENV is my favorite option at the moment. Will try it tomorrow. Thanks for your help! – CraigDavid Sep 23 '22 at 07:22

CraigDavid · Answer 2 · 2022-09-29T19:26:16.033

There is no good solution for this. I will just rebuild the test executables when I need to test locally. I can also invoke the gtest executable directly from the appropriate work directory.

Even the suggested ENV is not respected in the CTest scenario:

GTEST_ALSO_RUN_DISABLED_TESTS=1 ctest --verbose -R aie2_210a_ddr_2kernel_ddr_gmio_bd_api.maize_xrt
UpdateCTestConfiguration  from :/scratch/bryanloz/aie_controller_hax/test/build/DartConfiguration.tcl
Parse Config file:/scratch/bryanloz/aie_controller_hax/test/build/DartConfiguration.tcl
UpdateCTestConfiguration  from :/scratch/bryanloz/aie_controller_hax/test/build/DartConfiguration.tcl
Parse Config file:/scratch/bryanloz/aie_controller_hax/test/build/DartConfiguration.tcl
Test project /scratch/bryanloz/aie_controller_hax/test/build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 9
    Start 9: aie2_210a_ddr_2kernel_ddr_gmio_bd_api.maize_xrt
1/1 Test #9: aie2_210a_ddr_2kernel_ddr_gmio_bd_api.maize_xrt ...***Not Run (Disabled)   0.00 sec
No tests were found!!!

How to force run a disabled test from ctest?

2 Answers2