0

I have a handful of heap dumps that I am analyzing after the JVM has thrown OutOfMemory exceptions. I'm using Hotspot JDK 1.7 (64bit) on a Windows 2008R2 platform. The application server is a JBoss 4.2.1GA, launched via the Tanuki Java Service Wrapper.

It is launched with the following arguments:

wrapper.java.additional.2=-XX:MaxPermSize=256m
wrapper.java.initmemory=1498
wrapper.java.maxmemory=3000
wrapper.java.additional.19=-XX:+HeapDumpOnOutOfMemoryError

which translate to:

-Xms1498m -Xmx3000m -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError

There are some other GC & JMX configuration parameters as well.

My issue is when I analyze a heap dump created due to an OutOfMemoryException using the Eclipse Memory Analyzer, invariably, MAT shows me heap sizes of 2.3G or 2.4G. I have already enable the option in MAT to Keep Unreachable Objects, so I don't believe that MAT is trimming the heap.

java.lang.RuntimeException: java.lang.OutOfMemoryError: GC overhead limit exceeded

or

java.lang.OutOfMemoryError: Java heap space

Summary in MAT:

Size: 2.3 GB Classes: 21.7k Objects: 47.6m Class Loader: 5.2k

My actual heap file sizes are roughly 3300KB, so they are in line with my 3000m max heap size setting.

So where is the missing 500-600M of memory in MAT? Why does MAT only show my heap size as 2.4G?

Other posts on SO tend to indicate that it is the JVM doing some GC prior to dumping the heap, but if the missing 500M is due to a GC, why is it even throwing the OOM in the first place? If a GC could actually clear up 500M (or nearly 25% of my heap), is the JVM really out of memory?

Are there ways to tune the heap dumps so I can get a full/complete picture of the heap (including the missing 500M)?

If not, I find I'm really struggling to find how/why I'm encountering these OOMs in the first place.

As requested by someone, I am attaching the output of a jstat -gc <PID> 1000 from a live node: http://pastebin.com/07KMG1tr.

Eric B.
  • 23,425
  • 50
  • 169
  • 316
  • MAT just prints a number that is stored in the header of the heap dump file. That number doesn’t say anything about the completeness of the object graph in the dump file. – Holger Jun 07 '16 at 18:09

2 Answers2

1

Which GC are you using? You are probably missing Eden, try to use jstat - Java Virtual Machine Statistics Monitoring Tool

ebanouz
  • 51
  • 2
  • 4
  • As far as I can tell, it is using the default GC, which, to my knowledge, is the ParallelGC. – Eric B. Jun 06 '16 at 14:59
  • Can you please share your jstat output? – ebanouz Jun 06 '16 at 15:09
  • Any particular options/config that I should use for `jstat` that you would like to see? – Eric B. Jun 06 '16 at 15:33
  • jstat -gc PID 1000 should be enough – ebanouz Jun 06 '16 at 15:37
  • Apparently getting access to the production JVM to run jstat is not as simple as I thought it would be and requires several different levels of intervention. How critical are the stats? – Eric B. Jun 06 '16 at 15:57
  • I might be able to get some stats via the VisualGC plugin to VisualVM more easily though; let me see what I can do. – Eric B. Jun 06 '16 at 16:05
  • That would be helpful. For now enable unreachable objects in MAT, see: https://wiki.eclipse.org/MemoryAnalyzer/FAQ#Problems_Interpreting_Results to enable the option: Preferences-> Memory Analyzer-> Keep Unreachable Objects and reload the heap – ebanouz Jun 06 '16 at 16:11
  • I have already enabled that (view initial post). And it made no difference when viewing the heap. I even deleted all the index files just in case and reloaded the hprof file altogether and had MAT reindex the entire thing. – Eric B. Jun 06 '16 at 16:12
  • As requested, here is the dump from a `jstat` output. http://pastebin.com/07KMG1tr – Eric B. Jun 06 '16 at 17:54
  • thanks, i don't know what your application is doing but from the jstat i see that there were no major gc during these 303 seconds. my best suggestion would be: 1.use jmap to analyze memory footprint jmap -histo – ebanouz Jun 07 '16 at 13:18
  • 2.use Concurrent Mark and Sweep GC: -XX:ConcMarkSweepGC 3. try to change generations size: Sizing the Generations- https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/sizing.html – ebanouz Jun 07 '16 at 13:22
1

java.lang.OutOfMemoryError: GC overhead limit exceeded

this does not necessarily mean your heap is full, see this Q&A

java.lang.OutOfMemoryError: Java heap space

and this does not mean that your heap has 0 bytes left, it means it that an allocation request could not be satisfied. If something tries to allocate 600MB and there are only 500MB left then that will throw an OOME.

If not, I find I'm really struggling to find how/why I'm encountering these OOMs in the first place.

Obtaining a stack trace to see if the callsite doing the allocation in question does anything suspicious would be a start. Or you could just try bumping the heap size and see if the problem goes away.

Community
  • 1
  • 1
the8472
  • 40,999
  • 5
  • 70
  • 122
  • I realize that that `GC overhead limit exceeded` doesn't mean that it is full, but rather that it cannot reclaim enough space. It seems odd, however, that it would throw this type of exception if it doesn't `need` to reclaim additional space. And I can't imagine any way in java to allocate a 500MB block of memory in a single call. Maybe with an initial allocation to an array or hashtable/etc with a massive size, but other than that, I can't imagine anything that would allocate that size of memory chunk. Am I missing something obvious? – Eric B. Jun 06 '16 at 17:02