How do to perform a good performance comparations test?

Question

To write a good comparations test test you have to run it several thousands (millions) times. It will level (in most cases) other programs' influence.

But if a JVM can influence on the results. For example:

First solution is:

    final StringBuilder stringBuilder = new StringBuilder();
    stringBuilder.append(getStrOne());
    stringBuilder.append(getStrTwo());
    final String result1 = stringBuilder.toString();

And second is:

    final String result2 = getStrOne() + getStrTwo();

I do not know which one is better because JVM can influence on the results. How to know which one is better?

UPDATE: I don't mean exactly that appending comporation test. I'm asking about such a hard to test situation.

The second one will be converted to `final String result2 = "str1str2";` by the compiler... — assylias, Mar 08 '13 at 12:57
@assylias I guess so... It's just an example of such situation. I'll fix that in header. Thank you; — Pavel, Mar 08 '13 at 12:59
@PavelRyzhov what it will cache? It may compile code according to found hot spots (but as pointed out elsewhere you have to make a comparison AFTER that). — Adriano Repetti, Mar 08 '13 at 12:59
http://stackoverflow.com/questions/504103/how-do-i-write-a-correct-micro-benchmark-in-java — assylias, Mar 08 '13 at 13:02
Second is clearly better because it's more readable and concatenation of fixed Strings is optimized anyway (and if it weren't, it's treated like using StringBuilder internally). But trying to measure performance without influence of the JVM really makes no sense - your program will probably never run without a JVM. — Axel, Mar 08 '13 at 13:04

score 2 · Accepted Answer · answered Mar 08 '13 at 12:57

I recently performed some benchmark tests that relied on the excellent IBM article found here: http://www.ibm.com/developerworks/java/library/j-benchmark1/index.html.

The article describes many of the pitfalls that can affect accuracy of results, for example:

Runtime optimisation/re-compilation of your code.
Dead code elimination (i.e. unused results can cause your test code to be removed)
Garbage collection
Caching
...

Finally, the article links to a site where a framework can be downloaded. The framework does a very good job of spooling up a test method, looking for evidence of recompilation and waiting for the execution time to settle.

score 0 · Answer 2 · answered Mar 08 '13 at 12:57

In case of 2 strings, performance differences are negligible, but try the following:

String s = "";
for( int i = 0; i < 10000; i++ ) {
  s += i;
}

vs.

StringBuilder b = new StringBuilder();
for( int i = 0; i < 10000; i++ ) {
  b.append(i);
}

You'll find that the second loop is way faster. Why? Because string concatenation will create a new String object in every iteration, which wastes CPU cycles as well as memory.

I'll cite you:

To write a good comparations test test you have to test it several thousands (millions) times. It will level (in most cases) other programs' influence.

The same applies to tests within a single VM: test your code multiple times, use larger data, larger loops etc. Comparing only small portions doesn't make sense due to timing precision errors and other influences (e.g. garbage collection running in between).

How do to perform a good performance comparations test?

2 Answers2

Linked