10

I like to include google benchmark tests within out continuous integration environment.

How can I check if the code under test has not become slower than it was before? Do I have to manually process the json output of the benchmark?

Is there an elegant way of accounting for different machines?

Is there an elegant way to check for every build on the same machine the performance? Is there a better way than writing a bash script and compare the output of the actual performance with a reference performance?

There has been a similar question, but no good answer to it.

Is there a way to integrate it into Jenkins.

Vertexwahn
  • 7,709
  • 6
  • 64
  • 90
schorsch312
  • 5,553
  • 5
  • 28
  • 57
  • In case you are still looking for a way: Have you tried [compare_bench.py tool](https://github.com/google/benchmark/blob/master/docs/tools.md#compare_benchpy) – Mike van Dyke Jul 25 '18 at 08:21

2 Answers2

4

To integrate Google Benchmark in Jenkins, I used this neat plugin:

https://plugins.jenkins.io/benchmark

I had the tests output .json files

benchmarks.exe --benchmark_out=benchmarktest_output.json

and wrote a custom JSON schema for the plugin to understand the output:

{
  "description": "Google Benchmark JSON schema",
  "failure": { "value": true },
  "type": "object",
  "properties": {
    "benchmarks": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "name": { "type": "name" },
          "real_time": { "type": "result" },
          "cpu_time": { "type": "result" },
          "iterations": { "type": "parameter" }
        }
      }
    }
  }
}

I'm not able to display all the output in the plugin though. E.g., I haven't figured out how to display the time unit together with both real and cpu time, only with one of them at the time.

jonei
  • 96
  • 6
0

With this python tool one can automate the evaluation of google benchmark tests and thus integrate them into a continuous integration tool

Thanks you @MikeVanDyke for the hint.

schorsch312
  • 5,553
  • 5
  • 28
  • 57
  • Could you elaborate on that ? Especially When a new git commit leads to performance degration, I want the CI to trigger a warning, even stop the new commit from being merged. Is that possible ? – Lewis Chan Sep 23 '19 at 12:26
  • I wonder how to use `compare.py` to compare the performance between the base commit version and your commit version ? – Lewis Chan Sep 23 '19 at 12:32