13

In my work, we are running into a difficult to reproduce OOM issue. Or, more accurately, it is very easy to reproduce on one system, making that system unusable, but difficult to reproduce anywhere else, given the same inputs.

The application is being run as a service using a service wrapper. We did manage to get the configuration changed to launch it with the option of outputting a heap dump file on OOM but, unfortunately, they were truncated, most likely due to the service wrapper timing out and killing the process as it wrote the file. This is readily apparent, since the max memory is set to 1GB, and the hprof files are as small as 700MB, which is too small to be the entire heap upon OOM.

It would take a lot of jumping through hoops to additionally configure the wrapper to give the java process a longer time to write out the heap, but we are pursuing this using these 2 options:

wrapper.jvm_exit.timeout=600
wrapper.shutdown.timeout=600

The question is, is there anything useful I can do with the truncated hprof files I have? Eclipse MAT chokes on them. Jhat appears to load them, but then only shows 3 instances of Java.Object of size 0 and nothing else. I tried YourKit and it couldn't write its oids file.

It seems to me like these files should have some useful, accessible information in them. Is there a tool that can read what's there?

Thank you for your time!

user1410910
  • 1,050
  • 6
  • 12
  • 2
    For what it's worth here's a [link to the "spec", described in `hprof_b_spec.h`](http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/6ff36c6e0cf4/src/share/demo/jvmti/hprof/hprof_b_spec.h). An `hprof` file has a small header followed by many `record`s, so on a truncated file you could perhaps remove the last (incomplete) record, and salvage the rest, but even with that you'd get many records that would contain object IDs that would be declared in absent records... so probably unexploitable. – Hugues M. Jun 21 '17 at 12:11
  • 1
    Have you checked whether the hprof file is really truncated? The fact that Xmx=1g and the file is 700mb means nothing. – Oleg Estekhin Jun 27 '17 at 07:01
  • I've found the yourkit folks super responsive. have you posted anything on their forum? – Martin Serrano Jul 22 '17 at 00:34
  • @user1410910 were you able to find a suitable tool for the job? – Andrew Nov 20 '17 at 21:19
  • No, I was not able to. We just got another dump file from a different environment that was not truncated. – user1410910 Jan 03 '18 at 18:33
  • @OlegEstekhin The fact that we finally managed to reproduce the error on another system, with a longer wrapper timeout, and got a larger hprof file which could be analyzed, seems to imply it was. But, genuine curiosity, may I ask why Xmx=1g and the file being 700MB doesn't mean it's truncated? The heap dump should include the whole heap, and we knew it was an OOM error, so the heap should be 1GB... – user1410910 Jan 26 '18 at 11:38

2 Answers2

2

Best option for analyzing the dump file which i came across till date, is text editors like vim.

Amit Mahajan
  • 895
  • 6
  • 34
-2

Use Jpofiler(https://www.ejtechnologies.com/products/jprofiler/overview.html). It's not free, but, it has a trial period.

The live memory and CPU view options are your best bet to isolate your issues. It generally runs reasonably well even on large dumps.

abhijeet
  • 109
  • 4
  • 4
    How does this answer the question? Does JProfiler support loading truncated or otherwise corrupt heap dumps? The author states he already tried YourKit, which offers a "live" mode pretty similar to JProfiler. So this answer does not really help a lot in my opinion. – cello Jun 27 '17 at 07:40
  • That did not work in my case: ```java.io.IOException: Could not read heap dump. It seems that the data is corrupted. at com.jprofiler.core.e.b.az(ejt:1452) at com.jprofiler.core.e.b.J(ejt:1029) at com.jprofiler.core.e.e.a.a(ejt:249) at com.jprofiler.core.e.b.i(ejt:500) at com.jprofiler.core.e.b.a(ejt:235) at com.jprofiler.core.e.a.run(ejt:108) at com.jprofiler.core.e.b.aE(ejt:250) ``` – pyb Mar 22 '21 at 17:12