1

I want to benchmark 2 Libraries to evaluate them, with nearly Identical code.

But the issue is, that Java needs some Time to warm up.

Do you have any Idea how to properly set up a Benchmark, especially for "read write" operations with Java?

Its hard for me to grasp, how Java "warms up" or "cashes" Input data via streams.

My Usecase:

I read a template file. And fill it.

What I did:

Mesuring the Time of each Library to read a template and fill a document.

The issue I stumbled upon

In the first iteration the first library is significantly faster than the second one. But when I do like 1000+ iterations, they come out really close. I read the same tamplate file multiple times, that could be a issue.

Do you have any suggestions to create a realistic benchmark?

Because there is no Usecase, where 1000 Documents will be generated at once. But one per "workflow". But I need to take into account, that the JVM optimizes read data not only initialisation at runtime, and "is warmed up". And for how long it is warmed up, because since the usecase never includes multiple files at once, a "1000 iterations szenario" is not realistic.

Luxusproblem
  • 1,913
  • 11
  • 23
  • Potential duplicate: [How do I write a correct micro-benchmark in Java?](https://stackoverflow.com/questions/504103/how-do-i-write-a-correct-micro-benchmark-in-java) – Lino Apr 27 '20 at 06:49
  • Related: [Idiomatic way of performance evaluation?](https://stackoverflow.com/q/60291987) - microbenchmarking is *hard* - warm-up iterations are necessary, especially in code running in a multi-stage optimizing JIT VM which doesn't decide to fully optimize a method until it's been running for a while. Also the OS you're running it on will cache disk contents in RAM, and your CPU frequency is variable, and fresh memory allocations from the OS page-fault the first time you touch them (lazy allocation)... So many complications from over 50 years of advances in HW and SW. – Peter Cordes Apr 27 '20 at 06:51
  • @Lino I already read something like this. Also the Microbenchmarking Tutorial on Baeldung. But there is no information on how to deal with "read write" operations for files. Since the warm up is initalization, but does not inclode how "java deals with already read data of files" or potential "chaching" Thought, there would be some adivice – Luxusproblem Apr 27 '20 at 06:53
  • @PeterCordes I oriented myself on (Microbenchmarking)[https://www.baeldung.com/java-microbenchmark-harness], but the read wirte use case for files is never picked up. Is there any information, on under wich circumstances the JVM cashes already read files ? Or how to test it in realistic situation? Because multiple iterations in a short period will never happen for my usecase – Luxusproblem Apr 27 '20 at 06:55
  • Sounds like less of a JVM issue and more of a CPU data-cache issue, especially if your file is smaller than 8MiB or whatever your CPU's L3 cache size is. Or an OS issue with the pages from disk being hot in RAM. Unclear what kind of file writes you're doing or how you're timing anything. – Peter Cordes Apr 27 '20 at 07:01
  • @PeterCordes is there any way how to "properly" test this scenario ? like flushing the cache or fill it up with "data" so the cashed data will be gone? This is a server application wich does a lot of stuff, and every once in a while a "File will be generated". How can simulate a environmonet like this? – Luxusproblem Apr 27 '20 at 07:12
  • IDK what aspect of performance you're trying to isolate. Are you trying to optimize the file-write part, and need to test it in a caches-cold situation where no file has been created recently in that directory? The first step in measuring something is to be clear about exactly what you want to measure. – Peter Cordes Apr 27 '20 at 07:25
  • @PeterCordes I already tested it in the production environment manually with the same result. From a template a new file will be generated and then provided as download. Now I need to document it and test it via benchmark to explain the issue. It reads from already existing templates, fills them and generates a new file each time. – Luxusproblem Apr 27 '20 at 09:19

0 Answers0