How to reliably take JVM core dump if application is in GC?

Question

I'm trying to find the cause of memory leaks in my java application. I need to get a heap dump for a process that is in a long GC-cycle. Jmap isn't working in this case both because app is hanged and because heap is very large.

Unfortunately, jmap throws UnknownOopException on the core dump I took. I suppose that it isn't correct to take core dump during GC. Is there any way to suspend java process at the point where taking core dump will be correct?

Or am I totally wrong and got broken core dump because of some other problem?

score 2 · Accepted Answer · answered Dec 18 '12 at 14:05

2

You cannot take a heap dump while a GC is being performed. You need to take a heap dump before or after the GC. If you want to know why it is taking so long it is usueful to determine which pahse is taking so long. To see this to add -verbosegc This will indicate if it is taking a long time to reach a safe point, copy objects, scan the tenrured space, check references or something else.

It could be taking along time because you have lots of objects to clean up. As a guessimate it can take about a worst case 1 second per 2 GB of heap objects.

answered Dec 18 '12 at 14:05

Peter Lawrey

525,659
79
751
1,130

2

Is it possible to forcibly suspend JVM at a safepoint to safely take core dump? – Sergej Zagursky Dec 18 '12 at 14:19
1

For people that come here: in practice, I've seen it be _much_ worse than 1 second per 2 GB of heap objects when using CMS with 30 GB heaps (it can take minutes as it struggles to survive). – pickypg Dec 18 '15 at 02:29
1

@pickypg You are right that you can get much worse timings depending on your usage. i.e. do you have a lot of references from old to new generation, lots of References, or lots of finalise()able objects. If you avoid the features which are slow however you can get pause times of 4 seconds worst case for an 80 GB heap (an app I am trying to tune right now ;) – Peter Lawrey Dec 18 '15 at 15:42

score 0 · Answer 2 · answered Dec 18 '12 at 14:04

0

What you need to do is take the heap dump before the heap is so close to full that the GC locks up the application.

answered Dec 18 '12 at 14:04

Stephen C

698,415
94
811
1,216

Even if I get core dump before GC locks up the app I can't be sure that core dump will be consistent. – Sergej Zagursky Dec 18 '12 at 14:08
I think you can. My understanding is that the JVM suspends all threads while the heap is being dumped. – Stephen C Dec 18 '12 at 14:44
2

Maybe it does so when using jmap directly on JVM process. But jmap is performing very bad on huge heaps so I have to use gcore. – Sergej Zagursky Dec 19 '12 at 04:14
1

Unfortunately it isn't possible – Sergej Zagursky Dec 19 '12 at 09:28

score 0 · Answer 3 · answered Dec 18 '12 at 14:06

In my experience, an OutOfMemory exception or long GC cycles do not indicate a memory leak for certain.

In order to search for a memory leak, take 2 separate heap dumps some time apart (I've used jvisualvm, nowadays a version is bundled in with the JDK) and analyze them. Hint: Inspecting retain size of objects helps.

Depending on what your application does and if an apparent memory leak does not turn up, tweaking JVM GC options is your best bet. Look for generation ratios, generations after a new object is tenured etc.

Hope this helps a bit.

How to reliably take JVM core dump if application is in GC?

3 Answers3

Linked