How can I be sure that the compiler doesn't optimize away my performance test?

Question

I have a class that does some time-consuming calculations. I'm trying to performance test it:

int numValues = 1000000;
Random random = new Random();
startMeasuringTime();
double result;
for (int i = 0; i < numValues; i++) {
    result = calculatorInstance.doSomeTimeConsumingCalculationsOn(random.nextDouble());
}
stopMeasuringTime();

I'm using random values so the compiler won't optimize the calculations for being a million times the same. But what about the results? Does the compiler see it isn't used any more and leaves out the call (but then, can it see any side effects the method call could have?)

I don't want to put the results somewhere (into a file, array or to System.out), because I think this will slow down the test with work that I don't want to measure. Or produce an OutOfMemoryError.

Thanks in advance.

EDIT: changed the title a bit

Answer 1

But what about the results? Does the compiler see it isn't used any more and leaves out the call (but then, can it see any side effects the method call could have?)

It depends. If the JIT compiler can detect that the method call has no side-effects, then it is entitled to optimize it away. Especially since the result value is not used. In this case, you might just be measuring the calls to random.nextDouble() ... or possibly an empty loop.

To be sure you should it cannot be optimized away, you should probably write it like this:

int numValues = 1000000;
Random random = new Random();
startMeasuringTime();
double result;
for (int i = 0; i < numValues; i++) {
    result = result +
        calculatorInstance.doSomeCalculationsOn(random.nextDouble());
}
stopMeasuringTime();
System.err.println(result);  // Force result to be computed.

(I'm assuming that the time consuming calculation does depend on the argument ...)

---

You also need to take JVM warmup into account; i.e. run that benchmark code multiple times in the JVM until the measured time stabilizes.

---

Saying that the compiler is "over-optimizing" is kind of wrong. The compiler is actually doing its job correctly. If anything, the fault is in your code; i.e. it does "nothing useful".

Answer 2

Make sure that the result is used in some way, for example by summing it and printing it at the end. Summing is a good choice because addition is a very cheap operation.

Answer 3

The compiler actually do not do much optimization (except calculate values of constant expressions), as it has been found that the JVM does a much better job of optimization.

It is the modern JVM's that discover that code can be inlined (i.e. inserted directly in the calling code instead of executing a method call) and that applies especially well to empty code, removing the method call and replacing it with - tada - no code. That runs very fast, but does not measure well.

Above that, the JVM cannot optimize calls away, so your worry about passing different arguments to enforce evaluation is not necessary. If your method is non-trivial it will be called.

But, your concern about micro-benchmarks showing the wrong thing is valid. A much better approach to get insight in the performance is to do a real run with a profiler attached (there is a simple one in jvisualvm in the JDK 6).

What is it you need to know?

Answer 4

If in doubt - have a look at the byte code of this test class. If the compiler has "optimized" that call, then you won't find a call to that method there.

thod is a small disassembler that ships with the jdk. Dump the output to a file, open that file with standard editor and try to find the methodname. You don't have to understand the entire byte code if you just want to check if a certain method is called or used somewhere.

---

Made a quick test:

public static main(String[] args) {
  for (int i = 0; i < 1000000; i++) {
    double result = doSomething(Math.random());
  }
}

public double doSomething(double random) {
  return random * random;
}

For this class, the byte code contains the lines

invokestatic   #22; // Method doSomething:(D)D
dstore_2

which definitly tells, that the method is invoked and the result is stored to the local variable.

But it's still possible that the virtual machine detects the unused local variable and that the just-in-time compiler eliminates the call while compiling real machine code.