LCOV/GCOV branch coverage with C++ producing branches all over the place

Question

We are using LCOV/GCOV to produce test coverage of our projects. Recently we tried to enable branch-coverage additionally. But it looks like, this just doesn't yield the results we expected from a high-level developer view.

Using branch-coverage with C++ blows the report up with branches all over the place. We suspect (as the searching for the issues indicates) that mostly exception handling code creates these "hidden branches". And GCOV/LCOV doesn't seem to skip over these.

I created a small test project to show the problem: https://github.com/ghandmann/lcov-branch-coverage-weirdness

Currently we use Ubuntu 16.04. with:

• gcc v5.4
• lcov & genhtml v1.12

Our production code is built with c++11 enabled. The minimal example isn't built with c++11 enabled, but as we experimented a bit with all different options (c++ standard, optimization, -fno-exceptions) we didn't come up with a passable result.

Anybody got some ideas? Tipps? Are we using anything the wrong way? Is this - as stated somewhere else - really expected behavior?

Update:

As also pointed out on the gcc-help mailing list, these "hidden branches" occur because of exception handling. So adding the -fno-exceptions switch to gcc produces 100% branch coverage for "simple" programs. But when exceptions are disabled, gcc refuses to compile code which actually uses exceptions (e.g. try-catch, throw). Therefore for real production code this is not an option. Looks like, you have to simply declare ~50% coverage to be the new 100% in this case. ;)

Answer 1

The thing is that GCC also records branch information for each line where a scope exit due to some thrown exception is possible (e.g. on Fedora 25 with GCC 6.3.1 and lcov 1.12).

The value of this information is limited. The main use case for branch coverage data are complicated if-statements that have a multi-clausal logical expression like this:

if (foo < 1 && (bar > x || y == 0))

Say you are interested to verify whether your test suite also covers the bar > x case or if you just have test cases where y == 0.

For this, branch coverage data collection and the visualization by lcov's genhtml is useful. For simple if-statements like

if (p == nullptr) {
  return false;
}
return true;

you don't need branch coverage data because you see whether the branch was taken or not via looking at the coverage of the following lines.

The input of genhtml that is generated by lcov is in a relatively simple text format (cf. geninfo(1)). Thus, you can post-process it such that all lines that start with BRDA: and don't belong to an if-statement are removed. See for example filterbr.py which implements this approach. See also gen-coverage.py for the other lcov/genhtml processing steps and an example project where the resulting trace file is uploaded to codecov (codecov doesn't use genhtml but can import lcov trace files and displays branch coverage data).

(Non-)Alternatives

• disabling exceptions is only an option when your C++ code doesn't use any
• compiling with something like -O1 -fno-omit-frame-pointer -fno-optimize-sibling-calls somewhat reduces the number of recorded branch coverage data, but not much
• Clang supports GCOV style coverage collection but also implements a different approach, called 'source-based code coverage' (compile with -fprofile-instr-generate -fcoverage-mapping and post-process with llvm-profdata and llvm-cov). That toolchain doesn't support branch coverage data, though (as of 2017-05-01).
• by default, lcov+genhtml don't generate branch coverage data - sometimes you don't really need it (see above), thus, it is then an option to disable it (cf. --rc lcov_branch_coverage=0 and --no-branch-coverage)

Answer 2

GCC will add a bunch of exception handling stuff. Especially when you do function calls.

You can fix this by adding -fno-exceptions -fno-inline to your build.

I should add, you probably only want these flags on for testing. So something like this:

g++ -O0 --coverage -fno-exceptions -fno-inline main.cpp -o test-coverage 

Answer 3

There's a PR in progress that addresses these limitations. https://github.com/linux-test-project/lcov/pull/86.

This paper explains the theory behind the implementation.

Answer 4

You can try g++ -O3 --coverage main.cpp -o testcov. I have tried this with g++-5.4 on your file and it works fine, meaning exceptions are discarded with standard printf and string calls.

In fact, any optimization flag other than O0 will cause gcov to ignore exceptions generated for plain standard library calls in a CPP file. I am not sure if normal exceptions will also be optimized away (I don't think so, but haven't tried it yet).

But, I am not sure if you have any requirement in your project that only O0 should be used with your code and not O1, O2, O3 or even Os.

Answer 5

I did some work to add filtering to geninfo/lcov/genhtml to remove branches in C/C++ code which are associated with source code lines which appear to not contain any conditionals - using some relatively simple regexps. The filters appear to work in our code base/on the products where the modified lcov tools have been used. The filters are not perfect, and are easily defeated by a bloody-minded user.

I recently got approval to upstream my lcov updates. You can find them at https://github.com/henry2cox/lcov

This version adds support for differential coverage as well as date and owner binning.

A peripheral additional change was to add 'filtering', as described above - mainly because branch coverage seemed not usable for C++ code without it.
You can find the (admittedly hacky, and easily circumvented) regexps in method lcovutil::ReadCurrentSource::containsConditional - near line .../bin/lcovutil.pm:1122

While not perfect: this hack seems to work for our code. Your mileage may vary.

This is tested with perl/5.12.5 and gcc/8.3.0 and 9.2.0. It may work with other versions as well (please let me know if you find portability issues, so I can fix them).

Answer 6

I just came across the same problem and I want to get rid of these uncovered branches due to exceptions. I found a suitable solution for me:

I just avoid using "throw exception" in my code, which I want to cover, directly. I have designed a class, which offers some methods which throw the exceptions instead. As the exception class is not that complex, I don't really care about the coverage, so I just exclude everything with LCOV_EXCL_START and LCOV_EXCL_STOP. Alternatively I could also turn off the branch coverage only for that exception class.

I admit, it's not a straightforward solution, but for my purposes it's perfect due to other reasons, too (I need that exception class to be flexible, so that I can offer different implementation of it: once throwing an exception, another time doing something else).