4

I'm looking for ways to detect changes in runtime performance of my code in an automatic way. This would act in a similar way that JUnit does, but instead of testing the code's functionality it would test for sudden changes in speed. As far as I know there are no tools right now to do this automatically.

So the first question is: Are there any tools available which will do this?

Then the second questions is: If there are no tools available and I need to roll my own, what are the issues that need to be addressed?

If the second question is relevant, then here are the issues that I see:

  1. Variability depending on the environment it is run on.
  2. How do detect changes since micro benchmarks in Java have a large variance.
  3. If Caliper collects the results, how to get the results out of caliper so that they can be saved in a custom format. Caliber's documentation is lacking.
lessthanoptimal
  • 2,722
  • 2
  • 23
  • 25
  • I see one question: "Is there an existing library which does this already?". Is that *the* question? – The Nail Jan 19 '12 at 22:59
  • That is one question. If there is not and I need to roll my own what are the issues that need to be addressed. I'll modify the question a bit. – lessthanoptimal Jan 19 '12 at 23:20
  • 1
    Caliper's *everything* is lacking, which I both apologize for and promise is changing in the coming (small number of) months. – Kevin Bourrillion Jan 21 '12 at 16:10

3 Answers3

4

I have just come across http://scalameter.github.io/ which looks appropriate, works in scala and java.

Benedikt Waldvogel
  • 12,406
  • 8
  • 49
  • 61
TimP
  • 925
  • 1
  • 8
  • 13
2

Take a look at Caliper CI, I put out version 2.0 yesterday as a Jenkins plugin.

Brian Harris
  • 2,735
  • 3
  • 22
  • 34
  • This might be close to what I am looking for. Would prefer a standalone application with nearly no configuration. The project's I'm targeting of are not very big and don't require continuous integration tests. – lessthanoptimal Jan 23 '12 at 18:08
0

I don't know any separate tools to handle this, but JUnit has an optional parameter called timeout in the @Test-annotation:

The second optional parameter, timeout, causes a test to fail if it takes longer than a specified amount of clock time (measured in milliseconds). The following test fails:

@Test(timeout=100) public void infinity() {
   while(true);
}

So, you could write additional unit-tests to check that certain parts work "fast enough". Of course, you'd need to somehow first decide what is the maximum amount of time a particular task should take to run.

-

If the second question is relevant, then here are the issues that I see:

  1. Variability depending on the environment it is run on.

There will always be some variability, but to minimize it, I'd use Hudson or similar automated building & testing server to run the tests, so the environment would be the same each time (of course, if the server running Hudson also does all other sorts of tasks, these other tasks still could affect the results). You'd need to take this into account when deciding the maximum running time for tests (leave some "head room", so if the test takes, say, 5% more to run than usually, it still wouldn't fail straight away).

  1. How do detect changes since micro benchmarks in Java have a large variance.

Microbenchmarks in Java are rarely reliable, I'd say test larger chunks with integration tests (such as handling a single http-request or what ever you have) and measure the total time. If the test fails due to taking too much time, isolate the problematic code by profiling, or measure and log out the running time of separate parts of the test during the test run to see which part takes the largest amount of time.

  1. If Caliper collects the results, how to get the results out of caliper so that they can be saved in a custom format. Caliber's documentation is lacking.

Unfortunately, I don't know anything about Caliper.

esaj
  • 15,875
  • 5
  • 38
  • 52
  • The junit time out might work if it read the expected times are read from a file. Still the issue of creating and maintaining all those tests. Leaning towards a combined benchmark/regression test to avoid that problem. I might be stuck with just testing large chucks of code together like you suggested to avoid excessive false positives. – lessthanoptimal Jan 20 '12 at 21:00