Why isn't there measureable difference between new Something() and Something.class.newinstance()?

Question

I am running the following experiment and surprisingly realised that there is no measureable difference between the two runs:

public static void main(String[] args) throws Exception {
    long count = 1_000_000_0L;
    long measureCount = 1000L;


    // Measurement 1
    SpecificRecord tp1 = null;
    List<Long> times = new ArrayList<>();
    for (long j=0; j<measureCount; ++j) {
        long timeStart = System.currentTimeMillis();
        for (long i = 0; i < count; ++i) {
            tp1 = new WebPageView();
        }
        times.add(System.currentTimeMillis() - timeStart);
    }
    Stats st = Stats.of(times);
    double avg = st.mean();
    double stdDev = st.populationStandardDeviation();
    times.sort(Long::compareTo);
    int upper = (int)Math.ceil(times.size()/2.0);
    int lower = (int)Math.floor(times.size()/2.0);
    double median = ((times.get(upper))+(times.get(lower)))/2.0;
    System.out.println("avg: "+avg);
    System.out.println("stdDev: "+stdDev);
    System.out.println("median: "+median);
    System.out.println(tp1);


    // Measurement 2
    SpecificRecord tp2 = null;
    List<Long> times2 = new ArrayList<>();
    for (long j=0; j<measureCount; ++j) {
        long timeStart = System.currentTimeMillis();
        for (long i = 0; i < count; ++i) {
            tp2 = WebPageView.class.newInstance();
        }
        times2.add(System.currentTimeMillis() - timeStart);
    }
    Stats st2 = Stats.of(times2);
    double avg2 = st2.mean();
    double stdDev2 = st2.populationStandardDeviation();
    times2.sort(Long::compareTo);
    int upper2 = (int)Math.ceil(times2.size()/2.0);
    int lower2 = (int)Math.floor(times2.size()/2.0);
    double median2 = ((times2.get(upper2))+(times2.get(lower2)))/2.0;
    System.out.println("avg: "+avg2);
    System.out.println("stdDev: "+stdDev2);
    System.out.println("median: "+median2);
    System.out.println(tp2);
  }
}

The results:

avg: 110.63300000000005
stdDev: 47.07256431298379
median: 100.0
{"aid": 0, "uid": 0, "rid": 0, "sid": 0, "d": null, "p": null, "r": null, "f": null, "q": null, "ts": 0}
avg: 101.0840000000001
stdDev: 7.8092857547921835
median: 99.0
{"aid": 0, "uid": 0, "rid": 0, "sid": 0, "d": null, "p": null, "r": null, "f": null, "q": null, "ts": 0}

Update1:

Many of you pointed out that it is impossible to benchmark the JVM this way due to heavy optimization that masquarades the performance difference between new Something() and Something.class.newinstance().

Update2:

After repeating the test with the suggested methodology the results are kind of surprising to me:

Benchmark                   Mode  Cnt   Score   Error  Units
ReflectionTest.newInstance  avgt    5  12.923 ± 0.801  ns/op
ReflectionTest.newOperator  avgt    5  11.524 ± 0.289  ns/op

Update3: The test code:

import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

import java.util.concurrent.TimeUnit;

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@Warmup(iterations = 5, time = 10, timeUnit = TimeUnit.SECONDS)
@Measurement(iterations = 5, time = 100, timeUnit = TimeUnit.SECONDS)
@State(Scope.Benchmark)
public class ReflectionTest {

    public static void main(String[] args) throws RunnerException {
        Options opt = new OptionsBuilder().include(ReflectionTest.class.getSimpleName()).forks(1).build();
        new Runner(opt).run();
    }

    @Benchmark
    public WebPageView newOperator() {
        return new WebPageView();
    }

    @Benchmark
    public WebPageView newInstance() throws InstantiationException, IllegalAccessException {
        return WebPageView.class.newInstance();
    }
}

This question sill remains unanswered, there is not test that could outline any difference between class.newInstance() vs new for the class we use. This class uses is an implementation of Avro SpecificRecord.

public class WebPageView extends org.apache.avro.specific.SpecificRecordBase implements org.apache.avro.specific.SpecificRecord

You should be aware that your benchmarking may be incorrect due to numerous compiler optimizations. If you want to write proper benchmark, look into this answer: http://stackoverflow.com/a/513259/3449268 — vhula, Mar 13 '17 at 19:38
@Istvan it's the way that you measure (someone will suggest `jmh` soon enough). But they **should be very different** in speed, the `newInstance` does a lot of checks that are expensive. — Eugene, Mar 13 '17 at 19:42
@Eugene thanks! I guess so. I am not sure how to measure how much slower it is. I guess we put this into production and let the profiler run(not sure which one yet) for a longer time period to see the effect on the service. — Istvan, Mar 13 '17 at 19:44
There is a difference between using these techniques for the first time before the class is loaded and initialized, and after. Your benchmark is naïve about timing. I don't know how @Eugene predicts that they "should be very different in speed". How much is "very"? How expensive is "expensive", and in what coin? How many are "a lot"? Some of the checks are the same whether you use `new` or `newInstance`. Which ones differ? — Lew Bloch, Mar 13 '17 at 19:48
Possible duplicate of [How do I write a correct micro-benchmark in Java?](http://stackoverflow.com/questions/504103/how-do-i-write-a-correct-micro-benchmark-in-java) — 4castle, Mar 13 '17 at 19:58
@4castle I've seen this before and don't know if these are duplicates or not... this is indeed about how to write a correct micro-benchmark; but at the same time it is about `why is there a big difference`.. — Eugene, Mar 13 '17 at 20:01
@Istvan you're update 2 makes this indeed interesting. The answer wan run with jdk-9, BUT when I switch to jdk-8, there is still a difference, but much much smaller(like in your findings). I'll look into it more. — Eugene, Mar 14 '17 at 05:54
@Eugene thank you! I was surprised to see the results as well. I was expecting my methodology to be the culprit here. Anyways, thanks for showing me how to do the testing right. — Istvan, Mar 14 '17 at 06:45
@Istvan the truth is that I have no idea why the difference is so big. I really hope someone smarter will be able to shed some light. Here is the link: http://stackoverflow.com/questions/42786629/newinstance-vs-new-in-jdk-9-jdk-8-and-jmh — Eugene, Mar 14 '17 at 12:57

Eugene · Accepted Answer · 2017-03-14T13:19:43.643

EDIT

The test is fine still, BUT it was run on with jdk-9 where is proved to be much slower then jdk-8.

Running with jdk-8, will prove that the JIT is doing lots of optimizations that the real difference is around 2x.

A little bit of tests using jmh that show the big difference (running with jdk-9)

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@Warmup(iterations = 5, time = 2, timeUnit = TimeUnit.SECONDS)
@Measurement(iterations = 5, time = 2, timeUnit = TimeUnit.SECONDS)
@State(Scope.Benchmark)
public class TestNewObject {

    public static void main(String[] args) throws RunnerException {
        Options opt = new OptionsBuilder().include(TestNewObject.class.getSimpleName()).forks(1).build();
        new Runner(opt).run();
    }

    @Benchmark
    public Something newOperator() {
        return new Something();
    }

    @Benchmark
    public Something newInstance() throws InstantiationException, IllegalAccessException {
        return Something.class.newInstance();
    }

    static class Something {

    }
}

And the results, showing a pretty big difference.

Benchmark                  Mode  Cnt    Score    Error  Units
TestNewObject.newInstance  avgt    5  274.070 ± 50.554  ns/op
TestNewObject.newOperator  avgt    5    5.119 ±  3.550  ns/op

That's a 44x in speed difference. The call to newInstance does many (check the source) things that are very expensive (reflection and security checks).

While the new operator, taking TLAB into consideration is an extremely fast way to allocate an Object.

One thing to mention is that newInstance is deprecated in jdk-9.

Excellent answer thank you! Also, what is the way in Jdk-9 to handle a case like this when I do not know the class at compile time that I need to use? This might be worth to ask in a separate question. — Istvan, Mar 13 '17 at 20:30

Why isn't there measureable difference between new Something() and Something.class.newinstance()?

1 Answers1