How can I analyse large size heap dump around of 35-40 GB

Question

I have to analyse java heap dump of size 35-40GB, which can't be loaded on local machine except of remote servers of large memory.

I found Tool for analyzing large Java heap dumps as the best link till now. But after configuring all the things and properly executing all the command lines, I was not able to get any report file.

My ParseHeapDump.sh file looks as

#!/bin/sh
#
# This script parses a heap dump.
#
# Usage: ParseHeapDump.sh <path/to/dump.hprof> [report]*
#
# The leak report has the id org.eclipse.mat.api:suspects
# The top component report has the id org.eclipse.mat.api:top_components
#
./MemoryAnalyzer -consolelog -application org.eclipse.mat.api.parse "$@" -vmargs -Xms8g -Xmx10g -XX:-UseGCOverheadLimit

and MemoryAnalyzer.ini file looks as

-startup
plugins/org.eclipse.equinox.launcher_1.5.0.v20180512-1130.jar
--launcher.library
plugins/org.eclipse.equinox.launcher.gtk.linux.x86_64_1.1.700.v20180518-1200
java -Xmx8g -Xms10g -jar plugins/org.eclipse.equinox.launcher_1.5.0.v20180512-1130.jar -consoleLog -consolelog -application org.eclipse.mat.api.parse "$@"
-vmargs
-Xms8g
-Xmx10g

Please tell me If I'm doing any mistake in configuration or suggest me any other tool available in the market.

I feel your pain -- the Eclipse heap analyser can use a truly colossal amount of heap. I've not found a better way to use it, than to set a massive -Xmx value, and allocate a lot of swap to the system. It's very slow, though. I'd also like to know if there's a better way. — Kevin Boone, Sep 21 '20 at 18:39

score 3 · Answer 1 · answered Sep 22 '20 at 06:03

3

Processing large heap dump is a challenge. Both VisualVM and Eclipse Memory Analyzer required too much memory to process heap dumps in order of few dozen of GiB.

Commercial profilers show better result (YourKit in particular) though I not sure of their practical limit.

To routinely process 100+ GiB, I came up you with headless solution heaplib, which based on code base from VisualVM (Netbeans actually).

Heaplib is neigther graphical, nor interactive. It is oriented to automated reporting. Tool allows you to write code for heap analysis in OQL/JavaScript (or Java if you wish), though capabilities are limited to accommodate memory requirements. Processing of 100GiB could take hour, but for non interactive workflow it is acceptable.

answered Sep 22 '20 at 06:03

Alexey Ragozin

8,081
1
23
23

Hey Aragozin, Please tell me how to use https://github.com/aragozin/heaplib ? – Prince Gupta Sep 22 '20 at 06:30
1+ I was not aware of this beauty, will try as soon as I get a chance. – Eugene Sep 22 '20 at 14:50
@PrinceGupta I have put some minimal info to README.md – Alexey Ragozin Sep 22 '20 at 21:19

score 0 · Answer 2 · answered May 21 '23 at 09:22

Challenge is that RAM should be larger than the heap dump hprof file. And usually our laptop hosting windows system has RAM <=16 GB. Thus analyzing heap dump of 35-40 gb in local system is almost impossible.

Configure MAT in your remote unix server having enough RAM ( > 35-40GB). And run it using command line. (GUI is always slower than command line anyway.)

Assigning 8GB-10GB wont work. So Better to increase the heap assigned to java process.

-vmargs -Xms40g -Xmx40g

Check by descriptive answer here. https://stackoverflow.com/a/76298700/5140851

How can I analyse large size heap dump around of 35-40 GB

2 Answers2