How to profile the IO activities triggered by the java process?

Question

Currently, I know there are some tools for profiling the CPU/memory for the java process.

When I wanted to track how the java process interacts with IO, I didn't find any tool for checking the details, especially for which threads/call-stacks, usages, and files.

The tools I can use are some OS-level commands like dstat.

Is there any tool for profiling DISK IO for java?

Mike Dunlavey · Answer 1 · 2021-12-01T15:12:20.537

It's unconventional but easy. Just do this.

It's hard to find a profiler that a) works on wall-clock time rather than CPU time, b) samples the call stack, not just the IP, c) takes a small number of samples rather than a large number.

Remember, your overriding goal is locating the problem, so you can fix it, not just timing it but wondering what it is. The bigger the problem is, the more likely a random halt will land in it and tell you exactly what it's doing and why.

How big are problems? If a problem takes X% of time, X% is the most you could save by fixing it. So if X is 10% or less, why bother? Supposing X% is bigger than 10%, then every sample has that probability of landing in it. So if you take N samples, the number of times it lands in the problem is N times X% (average).

How many samples do you need? Well, if you see the program doing something wasteful on one sample, it's doubtful that you've found it. But if you see it doing the same wasteful thing on two samples, statistics say you've found the problem! And the smaller N is before you see the problem, the bigger it is! Do you know exactly how big it is? No, and you don't care, because you know it's big enough to fix. (If you want more information, take more samples. I never take more than 20.) (You can always measure the speedup afterwards, with a simple timer.)

Is there more than one problem? You bet there is! If you fix one problem, it makes the remaining ones bigger! Example, there could be two problems, problem A costing 50%, and problem B costing 25%. The first time you look, you find problem A, fix it, and now problem B costs 50%, not 25%. Fix both of them, and you're 4 times faster. (This is the magnification effect.) (If you happen to fix B first, no problem. Now A is 67% of the time.)
Moral: Don't stop looking and fixing until you fail to find things you can fix.

What typically happens with profilers is they find things that don't amount to much and/or you can't fix anyway. Like 5% in some obscure library routine. Does this leave programmers disappointed? NO. It makes them happy, because it seemingly tells them their code is optimal! Is that a paradox? Yes. Profilers are popular precisely because they don't work!

score 1 · Accepted Answer · answered Nov 28 '21 at 11:22

1

Is there any tool for profiling DISK IO for java?

Some JVM profilers have probes that deal with file access. JProfiler for example has a file probe that measures read and write operations for both java.io and java.nio.

Measurements are resolved by file:

For performance considerations, the hot spots view of the file probe shows which files are responsible for most of the time spent in disk I/O and gives you expandable back traces for each file.

Disclaimer: My company develops JProfiler

answered Nov 28 '21 at 11:22

Ingo Kegel

46,523
10
71
102

It's the right answer, but JProfiler is too expensive for individuals. – Steven Chou Mar 02 '23 at 13:07
1

@StevenChou FYI, we offer academic licenses for personal use as well as open source licenses at no cost – Ingo Kegel Mar 03 '23 at 10:19

score 1 · Answer 3 · answered Feb 23 '23 at 23:11

If your goal is find the bottleneck in your Java application (including IO), I would also suggest to try the "poor man's profile" technique pointed out already by Mike Dunlavey in his answer:

Run the application with a realistic workload and take thread dumps.
Repeat it a few times until you spot a pattern. (If there is an obvious bottleneck, it is likely that the program gets interrupted during expensive operations.)

How can you get a stack trace in Java? If you have the PID of the process, you have two options:

Run kill -3 PID to make the process print the stack traces on stdout
JDK includes a tool called jstack to get the thread dump: jstack PID

In my experience, this simple technique is surprisingly powerful and requires no setup. The overhead is also low, so you can use it safely in production.

If you are on Linux, an alternative is to use perf top for live profiling: sudo perf top

perf tends to work better with natively compiled languages such as C++, but if you cannot get anything out of Java thread dumps, it is worth a try. Since perf works on the Kernel level, it can also reveal bottlenecks in IO. Again, the overhead is low, so you can run it in production (with a realistic workload).

Actually, the async-profiler is much better for tracking CPU usage than the built-in jstack. It aggregates and sorts the most costly function and its call stacks in your application. — Steven Chou, Feb 24 '23 at 16:49

score 0 · Answer 4 · answered Mar 03 '23 at 16:23

0

I've had good results with Linux perf. It can measure all IO activity for time and take stack traces.

I haven't used it with java, but I'm reading that if you compile with -XX:+PreserveFramePointer the stack traces and symbols will work.

answered Mar 03 '23 at 16:23

dspeyer

2,904
1
18
24

How to profile the IO activities triggered by the java process?

4 Answers4