5

We have a bad performing application and therefore we used the flight recorder's method profiling to see where the time is spent. It's basically working, but the number of taken samples is way below 100 for a minute of recording. (using the "profiling" preset)

I used a simple sample application (summing up random numbers) for comparison, which yields about 6000 samples for a minute or recording - this seems correct to me.

There are several warnings on the front page, like: High memory consumption, loooooots of Exceptions. But the application is basically working so this might be a red herring.

I already fixed the "stack depth truncated" problem by increasing the number.

My guess: either the flight recorder is somehow misconfigured or the time isn't actually spent in the code but on other tasks. The cpu is pretty busy during the run so I don't assume all threads are waiting.

Please let me know what information might be of importance, so I can add them.

(It's a web framework library written in Scala using Jetty as Webserver; Oracle JDK 8)

Kai Giebeler
  • 517
  • 6
  • 12

1 Answers1

5

Java Flight Recorder method sampling is very specific. There two types of method samples

  • "Method Profiling Sample" - sample is taken only if thread is executing Java code (application code, not JNI, no part of JVM)
  • "Method Profiling Sample Native" - sample is taken only if thread is in JNI call

These same are taken separately and only former are visualized by Mission Control. Following execution state are omitted in both types of samples.

  • Threads suspended via JVM builtin facilites (BLOCKED, WAITING, SLEEPING states)
  • Threads execution JVM specific code e.g. raising exception

These omitted states lead to reduced sample count. High CPU utilization may also reduce sampling frequency by JFR due to CPU starvation.

I wouldn't recommend to use JFR method sampling as a first line performance diagnostic. Visual VM with thread dump based sampling usually gives more consistent picture. JFR is a powerful tools, but you need to combine information from multiple type of events to build overall performance picture.

"Looots of exceptions" is a one of cases there Visual VM tends to show hot spot, while JFR silently omits samples.

Below more reading about quirks of sampling profiling in JVM and Mission Control usage:

Suma
  • 33,181
  • 16
  • 123
  • 191
Alexey Ragozin
  • 8,081
  • 1
  • 23
  • 23
  • 1
    Thanks a lot! "[exceptions] is a one of cases there Visual VM tends to show hot spot, while JFR silently omits samples." - I had a closer look. They actually get sampled when activating all Exception options in JMC. After showing the _internal_ Exceptions as well, I was able to see that over 80 % of all processing time landed there (those huge stack traces took a lot of time). Now we just have to fix the MySQL-Connector, JOOQ and the Lift framework to quadruple our performance. Will accept as answer because of the "Exception omitted" hint. – Kai Giebeler Aug 21 '20 at 18:47