Warm up JVM for running Scalabench

Question

I'd like to have a java application, which runs the different benchmarks of the Scalabench. I'd like to ensure that the JVM is warmed up before I start to measure how long the benchmark runs. How should I run these benchmarks from my java app?

The first thing which came to my mind is to execute the Scalabench's jar inside a for loop using

Process proc = Runtime.getRuntime().exec("java -jar scalabench.jar testname");

But I don't think if it helps the JVM to warm up, because I think the JVM will load the jar in every iteration of the for loop.

The next possibility is to add the scalabench.jar as a dependency to my project, and call its main() method in the for loop to run the test. I think that in this case warming up the JVM should not be a problem, because it runs inside my application and it won't be killed after each iteration.

Bonus: can I use JMH to measure the tests from the Scalabench?

Answer 1

The analogy of "warming up the JVM" is incorrect in so far that it is not a single entity you're bringing up to a uniform temperature. It's called hotspot JVM for a reason.

What warm-up means is bringing the application from transient start-up behavior into a steady-state mode. This involves many things such as garbage during initialization being collected, the heap settling into its (hopefully) steady-state size, JITs emitting optimized code in the highest optimization tiers for the hottest code paths without profiles polluted from startup, IO caches filled, external resources fetched etc. etc., depending on what exactly you're benchmarking.

The very general recipe for this is to run warm-up for a in a loop, then run the benchmark with the same code and data, also in a loop to gather multiple samples. How many iterations it takes can be derived from statistical properties of the samples, such as variance.

For microbenchmarks this can be insufficient (explained in various other questions), due to additional complications such as benchmark order with in the JVM influencing optimization behavior, thus confounding the results. JMH does aim to solve them.