7

UPDATE

Throughout comments it turned out that the approach for benchmark I taken was incorrect, therefore results were misleading. After correcting my approach (as in accepted answer) results are as one would expect - JDK 13 performance is just as good as with JDK 11. See the answer for more details.

Original question

I was doing some performance benchmarks on HashSet under Windows 10, using following test code for JMH:

@Benchmark
@BenchmarkMode(Mode.AverageTime)
@Fork(value = 1, warmups = 1)
public void init() {
    HashSet<String> s = new HashSet<>();
    for (int i = 0; i < 1000000; i++) {
        s.add(Math.random() + "");
    }
    s.size();
}

I compiled and run it under different JDK versions and here is what results I got:

enter image description here

I tested it with different heap sizes too (thus 3 different colors for each JDK). JDK 14 is of course a pre-release snapshot from today - just to see ZGC performing under Windows.

I wonder - what happened after JDK 11? (note, for JDK 12 it already starts growing, even though it's not present on the chart above)

Googie
  • 5,742
  • 2
  • 19
  • 31
  • 1
    Are the results similar if you use `Integer.toString(i)` instead of `Math.random() + ""`? I'm curious if the regression is due to generating random numbers or the `HashSet` itself. – Jacob G. Feb 10 '20 at 16:52
  • 3
    wait... your `init` does not return _anything_? neither do you have `Blackhole::consume`? I wonder what will happen if you remove that `warmups` too. It seems to me that if you run this with `C2` compiler only, all of them should be close to zero, as that method can be treated as NOOP – Eugene Feb 10 '20 at 17:03
  • 2
    @Eugene no, because of `Math.random()` which will advance the state of a globally visible `Random` seed. Basically, this code is benchmarking the efficiency of `Math.random()`… – Holger Feb 10 '20 at 17:53
  • @Holger but this is soooo not obvious when you look at it with a "naked" eye. – Eugene Feb 10 '20 at 22:04
  • 5
    There was a *huge* amount of changes in G1 GC between JDK 11 and 13. Not sure which one is to blame, but the difference is clearly caused by garbage collection - you can see this by adding `-prof gc` JMH option, or just by switching to Parallel GC, which performs almost equally in both JDK versions. Note that your benchmark does *not* measure HashSet performance. According to async-profiler, about 50% CPU time is spent in GC, and ~25% converting double to String. – apangin Feb 10 '20 at 22:38
  • 3
    @Eugene wrong use of random in benchmarks is a popular mistake, though in this specific case, it saves the benchmark from dead code elimination. There were versions of Java 7 where the sole creation of a `HashMap` suffered from updating a global random seed (that awful alt-hashing feature). – Holger Feb 11 '20 at 08:31

1 Answers1

1

Thank you all for suggestions in comments.

The answer was most likely Math.random() or HashSet, or missing Blackhole::consume or combination of all. I changed the test to simply do i + "aaaaaaaaa" and replaced HashSet with ArrayList pre-initialized with appropriate size to accommodate for all values to be populated. I also added Blackhole::consume at the end to exclude unwanted JIT optimizations.

After all of that, timing drops from JDK 8 to 11 gradually and then stays around the same among JDK 11-13. In JDK 14 it raises slightly, but well - it's not released yet.

Googie
  • 5,742
  • 2
  • 19
  • 31
  • Just an open question remains in my head - even if benchmark code was not ideal, why it would give consistent results (despite multiple runs) proving decreased performance of `Math.random()` or `HashSet`? It's a bit worrying to me. Also results presented on the chart are aligned with how long it takes for Eclipse to start (from splash screen to main window appearing). Surprisingly Eclipse takes way longer to start with JDK 12-13 (14 secs) than with JDK 9-11 (10 secs). Should that be worrying? – Googie Feb 12 '20 at 08:08
  • 3
    Please edit your question to reflect the fact that you've realized that the problem was likely to do with an extraneous factor (random) or benchmarking methodology. Otherwise people who read the question and don't read down to the bottom answer will come away with a mistaken "Java 14 is slow!". – Brian Goetz Feb 16 '20 at 16:52