What happens to the return values while using Google benchmark?

Question

I am benchmarking some functions in our software using the Google-benchmark. Let us say the function signature is something like below. The return type can be any other derived data type.

std::map<uint32_t, bool> func(Obj& o1, Obj& o2);

The benchmark function looks something like this.

static void BM_Func(benchmark::State& state) {

// Prepare the objects o1 and o2
for (auto _ : state)
  func(Obj& o1, Obj& o2);
}
BENCHMARK(BM_Func);

BENCHMARK_MAIN();

Now, the code compiles and I am able to collect the benchmark results. However, I have below questions.

What happens to the return values? Should I be bothered at all if I am not using these values anywhere again in the benchmark function?
Should I call the function instead like this benchmark::DoNotOptimize( func(Obj& o1, Obj& o2) ); to avoid optimization? I do not really understand when to call the function with benchmark::DoNotOptimize

Marek R · Answer 1 · 2019-11-28T16:22:28.090

You have to remember that there is "AS IF RULE". So compiler can do anything with code as long as visible effects of execution remains unchanged.

In case of performance measurements there is a risk that compiler will remove function under test since removing it will not have impact on visible effects of execution.

So it is recommended your test should look like this:

static void BM_Func(benchmark::State& state) {
    // Prepare the objects o1 and o2
    for (auto _ : state) {
        auto result = func(Obj& o1, Obj& o2);
        benchmark::DoNotOptimize(result);
    }
}

benchmark::DoNotOptimize will hide from compiler fact that result is in fact not used.

Note that benchmark test has to be build with optimization flags set up exactly like on production release. So optimization must be enabled and obsolete (from compiler point of view) code will be removed.

BTW there is nice online tool quick-bench which uses google benchmark, note that it also shows assembly results to be able verify what has been optimized by compiler (in case some code was removed).

Benchmarking without optimization flag, at least to me makes little sense. So, I always compile with -O3 — talekeDskobeDa, Nov 28 '19 at 16:26
I use quick-bench to test out small algorihms. But, rather prefer to have Google-benchmark integrated into our code base just like Google unit-testing — talekeDskobeDa, Nov 28 '19 at 16:28
I use `quick-bench` as a demo tool for answers on SO. My point was there is a UI which allows you to see what has been optimized and what actually is tested. — Marek R, Nov 28 '19 at 16:30

score 3 · Accepted Answer · answered Nov 28 '19 at 16:15

The danger with not using benchmark::DoNotOptimize is that the compiler might realize that func has absolutely no side effects. It would then correctly conclude that your code is equivalent to for (auto _ : state) /* do nothing */;. You of course didn't want to measure nothing.

Using benchmark::DoNotOptimize prevents the compiler from the above realization. It has no choice but to actually call func to obtain a result object (although the same considerations apply - if it can inline func and func always returns e.g. true, then the rest of func might get optimized out).

If the returned object is large, then destructing it might take a meaningful amount of time. Since this happens inside the benchmarking loop in your code, this time will be included in your benchmark. Avoiding that is pretty nontrivial though, and any "real" user of the function will have to incur this time too, so the answer is "nothing extraordinary happens with those objects".

Does this mean, in most of the cases we have to use DoNotOptimize? If I am reusing the return value, to let's say call another function inside the for loop, then I would not use the DoNotOptimize — talekeDskobeDa, Nov 28 '19 at 16:24
@talekeDskobeDa It is safest to use it, yes. If the compiler cannot see what's inside `func` (e.g. because its definition is in a different translation unit and you are building without link-time/whole-program optimization) then using `DoNotOptimize` won't help, but that's playing a dangerous game against the compiler. — Max Langhof, Nov 28 '19 at 16:27

What happens to the return values while using Google benchmark?

2 Answers2