Trying to benchmark lambda performance

Question

I've read this post: Performance difference between Java 8 lambdas and anonymous inner classes and provided there article

and it there said:

Lambda invocation behaves exactly as anonymous class invocation

"Ok" I said and decided to write my own benchmark, I've used jmh, here it is below (I've also added benchmark for method reference).

public class MyBenchmark {

    public static final int TESTS_COUNT = 100_000_000;

    @Benchmark
    public void testMethod_lambda() {
        X x = i -> test(i);
        for (long i = 0; i < TESTS_COUNT; i++) {
            x.x(i);
        }
    }
    @Benchmark
    public void testMethod_methodRefernce() {
        X x = this::test;
        for (long i = 0; i < TESTS_COUNT; i++) {
            x.x(i);
        }
    }
    @Benchmark
    public void testMethod_anonymous() {
        X x = new X() {
            @Override
            public void x(Long i) {
                test(i);
            }
        };
        for (long i = 0; i < TESTS_COUNT; i++) {
            x.x(i);
        }
    }

    interface X {
        void x(Long i);
    }

    public void test(Long i) {
        if (i == null) System.out.println("never");
    }
}

And the results (on Intel Core i7 4770k) are:

Benchmark                                     Mode  Samples   Score  Score error  Units
t.j.MyBenchmark.testMethod_anonymous         thrpt      200  16,160        0,044  ops/s
t.j.MyBenchmark.testMethod_lambda            thrpt      200   4,102        0,029  ops/s
t.j.MyBenchmark.testMethod_methodRefernce    thrpt      200   4,149        0,022  ops/s

So, as you can see there is 4x difference between lambda and anonymous method invocation, where lambda is 4x slower.

The question is: what am I doing wrong or I have misunderstanding of performance theory about lambdas?

EDIT:

# VM invoker: C:\Program Files\Java\jre1.8.0_31\bin\java.exe
# VM options: <none>
# Warmup: 20 iterations, 1 s each
# Measurement: 20 iterations, 1 s each

Answer 1

The problem is in your benchmark: you are the victim of dead code elimination.

JIT-compiler is quite smart to understand sometimes that the result of automatic boxing is never null, so for anonymous class it simply removed your check which in turn made the loop body almost empty. Replace it with something less obvious (for JIT) like this:

public void test(Long i) {
    if (i == Long.MAX_VALUE) System.out.println("never");
}

And you will observe the same performace (anonymous class becomes slower, while lambda and method reference perform at the same level).

For lambda/method reference it did not made the same optimization for some reason. But you should not worry: it's unlikely that you will have such method in real code which can be optimized out completely.

In general @apangin is right: use Blackhole instead.

Answer 2

In addition to the issues raised by @TagirValeev, the benchmark approach you are taking is fundamentally flawed, because you are measuring a composite metric (despite your attempts not to.)

The significant costs you want to measure independently are linkage, capture, and invocation. But all your tests smear together some amount of each, poisoning your results. My advice would be to focus only on invocation cost -- this is the most relevant to overall application throughput, and also the easiest to measure (because it is less influenced by caching at multiple levels.)

Bottom line: measuring performance in dynamically compiled environments is really, really hard. Even with JMH.

Answer 3

My question is another example how you shouldn't do benchmarking. I've recreated my test according to the advice in other answers here.

Hope now it is near to correctness, because it shows that there isn't any significant difference between lambda's and anon's method invocation performance. See it below:

@State(Scope.Benchmark)
public class MyBenchmark {

    @Param({"1", "100000", "500000"})
    public int arg;

    @Benchmark
    public void testMethod_lambda(Blackhole bh) {
        X x = (i, bh2) -> test(i, bh2);
        x.x(arg, bh);
    }

    @Benchmark
    public void testMethod_methodRefernce(Blackhole bh) {
        X x = this::test;
        x.x(arg, bh);
    }

    @Benchmark
    public void testMethod_anonymous(Blackhole bh) {
        X x = new X() {
            @Override
            public void x(Integer i, Blackhole bh) {
                test(i, bh);
            }
        };
        x.x(arg, bh);
    }

    interface X {
        void x(Integer i, Blackhole bh);
    }

    public void test(Integer i, Blackhole bh) {
        bh.consume(i);
    }
}
Benchmark                                     (arg)   Mode  Samples          Score  Score error  Units
t.j.MyBenchmark.testMethod_anonymous              1  thrpt      200  415893575,928  1353627,574  ops/s
t.j.MyBenchmark.testMethod_anonymous         100000  thrpt      200  394989882,972  1429490,555  ops/s
t.j.MyBenchmark.testMethod_anonymous         500000  thrpt      200  395707755,557  1325623,340  ops/s
t.j.MyBenchmark.testMethod_lambda                 1  thrpt      200  418597958,944  1098137,844  ops/s
t.j.MyBenchmark.testMethod_lambda            100000  thrpt      200  394672254,859  1593253,378  ops/s
t.j.MyBenchmark.testMethod_lambda            500000  thrpt      200  394407399,819  1373366,572  ops/s
t.j.MyBenchmark.testMethod_methodRefernce         1  thrpt      200  417249323,668  1140804,969  ops/s
t.j.MyBenchmark.testMethod_methodRefernce    100000  thrpt      200  396783159,253  1458935,363  ops/s
t.j.MyBenchmark.testMethod_methodRefernce    500000  thrpt      200  395098696,491  1682126,737  ops/s