1

I have a CMake project in which I am trying to enable code coverage (Using Gcov, Lcov, and Geninfo). In order to do this, I am using a module CodeCoverage.cmake to set this up for my target. Here is the code that sets this up.

if (ENABLE_COVERAGE AND NOT CMAKE_CONFIGURATION_TYPES)
    if (NOT BUILD_TESTS)
        set(BUILD_TESTS On CACHE STRING "" FORCE)
    endif (NOT BUILD_TESTS)

    include(CodeCoverage)

    include_directories(include ${PROJECT_BINARY_DIR})
    add_subdirectory(src) # <--
    # Defines target Project-Name-lib, which is a 
    # library consisting of all sources files except main.cpp
    # Also defines Project-Name, which is the executable

    target_compile_options(Project-Name-lib PRIVATE "--coverage")
    # Only add coverage flags to the library, _NOTHING_ else

    SETUP_TARGET_FOR_COVERAGE(NAME coverage 
        EXECUTABLE tests 
        DEPENDENCIES coverage)
    # Function from the module to enable coverage
else (ENABLE_COVERAGE AND NOT CMAKE_CONFIGURATION_TYPES)
    # ... Normal build ...
endif (ENABLE_COVERAGE AND NOT CMAKE_CONFIGURATION_TYPES)

if (BUILD_TESTS)
    include(CTest)
    enable_testing()
    add_subdirectory(tests)
endif (BUILD_TESTS)

This approach works really well, and when I use make coverage to run my program, then the reports successfully generate and can be viewed in the browser. However, there is one problem. This approach only enables coverage for the library, not any of the local header files I am using. For example, if I had a header like the following:

class HelloWorld
{
public:
    HelloWorld();        

    std::string hello() const; // <-- Implementation in header.cpp
    std::string world() const; // <-- Implementation in header.cpp

    int generateRandomNumber() const; // <-- Implementation in header.cpp

    int headerFunction(int test) const
    {
        if (test == 45)
            return 45;
        else
            return 4;
    }
private:
    std::string hello_;
    std::string world_;
};

And a corresponding test case (Using Catch 2.2.1):

SECTION("function headerFunction()")
{
    REQUIRE(helloWorld.headerFunction(33) == 4);
    // All  branches are NOT covered
}

Then all test cases pass (as expected), but the function in the header file does not show up in index.html for the coverage. Only the functions that are defined in header.cpp do. Because of this, my code coverage incorrectly shows up as 100%. How should I change my CMake code so that the functions that are defined in a header file are also included as part of the coverage report?

Arnav Borborah
  • 11,357
  • 8
  • 43
  • 88
  • Most likely, the function defined in the header file is optimized, see that question: https://stackoverflow.com/questions/32330219/gcov-is-not-generating-coverage-information-for-header-files. Original `APPEND_COVERAGE_COMPILER_FLAGS` sets optimization level `-O0`, but you forget to do the same in your CMake code. – Tsyvarev Mar 14 '18 at 07:45
  • @Tsyvarev Nope, I changed my command to: `target_compile_options(Project-Name-lib PRIVATE "-g;-O0;--coverage;-fprofile-arcs;-ftest-coverage")`. This doesn't change anything. I'm guessing the error is caused by the fact that the include files are in `include/project`, so the coverage doesn't look at it. But shouldn't the `#include` statement cause coverage to be evaluated for that header? Currently, all header files are just completely ignored. – Arnav Borborah Mar 14 '18 at 12:23
  • Most likely, CMake is unrelated here. `.gcno` files are generated for every *source* file by the compiler and they have no "exclude that header" feature. `.gcda` files are generated when code is actually executed. It is `gcov`/`lcov` tool which processes `.gcda` and `.gcno` files and prepare html report. – Tsyvarev Mar 14 '18 at 13:13
  • @Tsyvarev So what should I do? I have a failing build on [my full project](https://github.com/arnavb/cpp-project-template). – Arnav Borborah Mar 14 '18 at 13:19
  • Ok, I fixed my build, but I still cannot get the full [codecov report](https://codecov.io/gh/arnavb/cpp-project-template) for my header files. At this point, I feel like giving up on my Code Coverage... – Arnav Borborah Mar 14 '18 at 13:28
  • When I worked with `lcov` last time, I noted that different Linux distros provide `lcov` with different default settings. That time it was coverage for *conditions* which differs, but header's coverage could be the very thing which different `lcov` process differently. Am I correctly understand you, that your header file is shown in the coverage report, but it is completely white (as if it contains no executable code)? – Tsyvarev Mar 14 '18 at 13:38
  • @Tsyvarev No, the file itself doesn't show up _at all_. I get a top level directory `src`, which contains my `.cpp` files (Which have 100% coverage). I don't get anything else, no include folder or anything. – Arnav Borborah Mar 14 '18 at 13:39
  • @Tsyvarev The code coverage report I linked above shows 100% coverage, even though I have a function in my header file in which all branches are _not_ executed. – Arnav Borborah Mar 14 '18 at 13:44
  • Weird... I suggest you to play with toy example about the coverage: single source, single header, compile them directly with `gcc` (no CMake). You may got `lcov` usage from docs or from your project by run `make VERBOSE=1 coverage`. – Tsyvarev Mar 14 '18 at 13:46
  • @Tsyvarev Ok, I followed the exact steps for 1 source (src/Test.cpp) and 1 header file (includeTest.h) and was able to get the expected output with the GCC command: `g++-5 src/main.cpp -Iinclude/ --coverage -lgcov ` and copying the commands from `make VERBOSE=1 coverage`. However, now this is working too well. I am also getting Code coverage for `iostream`. ;(. Working or not, I still have no idea how to apply this to my CMake project... – Arnav Borborah Mar 14 '18 at 17:41
  • @Tsyvarev Ok, major update. Turns out the solution in https://stackoverflow.com/a/49256108/6525260 doesn't actually work. I created a minor test project in which I compared calling `append_coverage_compiler_flags` and doing what I did in this post. The first led to complete coverage; The latter did not. So I need to apply flags to the header files? I'm unsure what to do right now. – Arnav Borborah Apr 15 '18 at 16:59
  • `So I need to apply flags to the header files?` - There is no such thing like "compiler flags for a *header* file" - compiler flags are applied for the whole **source** file. It seems exactly the reason of your problem: your test, which calls the inline function, is compiled without coverage flags. So coverage counters aren't updated for that calls. When `main.cpp`, compiled with coverage flags, calls the inline function, this updates coverage counters. – Tsyvarev Apr 15 '18 at 17:10
  • @Tsyvarev but when I link the library for functions that are in cpp files (Using `target_link_libraries`), doesn't the function in the header file get the same coverage flags? If not, then I'm probably going to open an issue in Catch to see how to speed it up. – Arnav Borborah Apr 15 '18 at 17:18
  • Inline functions are compiled into **different instances** for different object files. Instance of the function which is compiled with `main.cpp` has coverage flags, thus it has coverage counters. Instance of the function which is compiled with `test.cpp` has no coverage flags and no coverage counters. A *non-inline* function has the only instance, corresponded to the `.c` file defining it. If this `.c` is compiled with coverage flags, any call to the function would update coverage counters. – Tsyvarev Apr 15 '18 at 17:37
  • @Tsyvarev That's actually really enlightening. But now still, I have no idea how to fix this... I need coverage flags, but I can't apply them without a compilation freeze... – Arnav Borborah Apr 15 '18 at 17:46
  • I know no ideal fixing for your case. Possible approaches could be: 1. Ignore non-covered inline functions. Usually, inline functions are one-liners, which could be verified without tests. 2. Have special tests which checks inline functions and compiled with coverage flags. If such tests are written separately from the tested libraries, they should know in advance which functions are inline. – Tsyvarev Apr 15 '18 at 17:53
  • @Tsyvarev The problem is; It isn't just inline functions. All functions in header files (SUCH AS TEMPLATE FUNCTIONS) are ignored by the coverage. So if I had a header only library (It's possible), then I can't even use Catch for the reason specified above. For now, I am considering writing separate tests and have opened an issue (https://github.com/catchorg/Catch2/issues/1253) in Catch. I also have a repository with this issue isolated: https://github.com/arnavb/test-codecov – Arnav Borborah Apr 15 '18 at 18:04

0 Answers0