Possible unwanted compiler optimization on benchmark test

Question

I've developed a small library that enables declarative argument validation using annotations, something like this:

@Validate({"not null | number"})
public static Integer notNullAnnotated(Integer number) {
    return number++;
}

now I'm benchmarking this against pure java code version:

public static Integer notNullInline(Integer number) {
    if (number != null) {
        return number++;
    } else {
        throw new IllegalArgumentException("argument should not be null");
    }
}

and here is the test:

@Test
public void performanceTest() {
    long time = System.nanoTime();
    for (int i = 0; i < iterationCount; i++) {
        notNullAnnotated(i);
    }
    System.out.println("time annotated : " + (System.nanoTime() - time));

    time = System.nanoTime();
    for (int i = 0; i < iterationCount; i++) {
        notNullInline(i); // TODO does compiler do any optimization here?
    }
    System.out.println("time inline : " + (System.nanoTime() - time));
}

I know that this is not the intended way to do benchmark tests. Right now I'd rather avoid adding any utility libraries for this simple test (as even this way results are good), but I'd like to know if compiler does any optimization here?

Are you trying to benchmark the speed of the annotation? – David Ehrmann Jul 06 '14 at 17:47 — David Ehrmann, Jul 06 '14 at 17:47

score 2 · Answer 1 · answered Jul 06 '14 at 15:18

2

Since you're already this far down the premature optimization path, look into the PrintAssembly and CompileThreshold flags for Hotspot. The first option lets you inspect the assembly Hotspot generates. The second lets you set a threshold for when the JIT kicks in.

answered Jul 06 '14 at 15:18

David Ehrmann

7,366
2
31
40

done, compileThreshold seems had some influence, result is better now. Can I disable any optimization at all (disable JIT for example)? (using java8 jdk) – vach Jul 06 '14 at 15:24
You can disable JIT by added ```-Djava.compiler=NONE``` to the command. But this makes that test very artificial; you'd never run that in production. I usually *want* optimizations in benchmarks so they're at least representative of what's going on. – David Ehrmann Jul 06 '14 at 17:45
Does `-XJava.compiler=NONE` differ from `-Xint`? I always used the latter flag to disable the JITC, didn't know the former existed... – awksp Jul 06 '14 at 19:12
-X flags are JVM-specific. J9 might support setting java.compiler. Beyond that, I'm not sure. – David Ehrmann Jul 06 '14 at 19:24
Oops, meant `-DJava.compiler=NONE`, but I think you got the point. Didn't know `-X` flags were VM-specific; thanks for that!. Looks like it's the `-D` version I'll be using from now on – awksp Jul 06 '14 at 20:19

score 1 · Accepted Answer · edited May 23 '17 at 12:29

Of course the just in time compiler will optimize your code (assuming a reasonably high iteration count), just as it would in an actual programm running that code. Optimization, by itself, is therefore desireable in a benchmark. Of course you'll have a problem if the artificial nature of your code permits optimizations not available to the real code. In your case, the compiler may conclude that notNullInline will never throw, therefore have no effect, and choose to remove the entire loop.

Writing a correct benchmark is already discussed at How do I write a correct micro-benchmark in Java?

Possible unwanted compiler optimization on benchmark test

2 Answers2