21

I am looking into a potential memory leak (or at least memory waste) in a largish Java based system. The JVM is running with a maximum heap size of 5 GB and 2-3GB heap usage is an expected base line for the application. (There can be peaks that are higher)

In an overload scenario which I am investigating the heap gets filled up. Analyzing the a heap-dump with the "Eclipse MemoryAnalyzer Tool" shows (no surprise) that the heap is entirely used up.

MAT shows 2 potential leak candidates, both roughly retaining 2.5GB: java.lang.Thread and a domain object from the system which is used extensively during transaction processing in the system. All these domain objects are however (no surprise) reachable from the Thread instances. Those threads are processing the transactions, after all. Thus, the 2.5 GB attributed to java.lang.Thread is almost entirely caused by those domain objects. No surprise here.

Listing the object tree of all java.lang.Thread instances and summing up the retained heap of all threads results in 2.5 GB of retained heap.

Where should I look for the other 2.5 GB that are needed to fill up the heap, if they are not reachable from an instance of java.lang.Thread? - There is nothing in the finalizer queue - There is not a significant amount of unreachable objects pending GC

I think another way to put this question is: "How do I find all objects that are not reachable from an instance of java.lang.Thread? Maybe an OQL query?, and the other question: "What kind of Objects are there that are not reachable from an instance of java.lang.Thread other then Objects in the Finalizer Queue and unreferenced objects pending GC?"

VoidPointer
  • 17,651
  • 15
  • 54
  • 58
  • Would you mind providing some more data. Like maybe the graph you are speaking of and the Domain objects themselves. How they are used. Do they make a large use of oddly sized arrays? Is there an allocation which fails and causes an OOM to be thrown? – Greg Giacovelli Dec 19 '13 at 22:25
  • Please also include whether or not you are using GC flags (like: -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:+UseConcMarkSweepGC) – JoeG Dec 19 '13 at 22:40
  • Are you sure the 2.5GB from the thread is also including the domain objects? If they are two separate counts then that is going to total your 5GB. – Tim B Dec 20 '13 at 09:49
  • Java objects don't "leak" (when the GC is working correctly, which one must assume). You *can* sort of "leak" open file instances and the like (if your code does not properly close them), since these include non-Java structures. And, of course, you can have objects that are reachable but "lost", because you don't remember where you stashed the reference to it. – Hot Licks Dec 20 '13 at 17:53
  • 4
    And keep in mind that a lot of objects are "anchored" outside of threads -- everything having to do with class loaders and classes (including all static storage), plus stuff that the system tracks such as the system P-list. – Hot Licks Dec 20 '13 at 17:55
  • @TimB the 2.5GB of domain objects are all referenced from the threads. – VoidPointer Dec 20 '13 at 20:07
  • @HotLicks I'll try to check those out. Do you know how to list all static storage in Eclipse MAT (or VisualVM) or some other tool using a hprof heapdump? – VoidPointer Dec 20 '13 at 20:09
  • I'm not really familiar with the current Java tools. – Hot Licks Dec 20 '13 at 20:14

5 Answers5

4

I too faced the problem with memory leaks at our site,
Use yourkit java profiler which provide lots of information and with its ability you can have a wider image where all the memory is being utilized.
You can find a great tutorial Find Java Memory Leaks with the above tool.

Your question,

"What kind of Objects are there that are not reachable from an instance of java.lang.Thread other then Objects in the Finalizer Queue and unreferenced objects pending GC?"

There are four kinds of object,

  1. Strong reachable, objects that can be reached directly via references from live objects
  2. Weak/Soft reachable, objects that are having weak/Soft reference associated with them
  3. Pending Finalization, objects that are pending for finalization and whose reference can be reached through finalizer queue
  4. Unreachable these are objects that are unreachable from GC roots, but not yet collected

Besides these JVM also uses Native memory whose information you can find on IBM Heap and native memory use by the JVM and Thanks for the memory and according to YourKit the JVM Memory Structure has Non-Heap Memory whose definition according to them is

Also, the JVM has memory other than the heap, referred to as non-heap memory. It is created at the JVM startup and stores per-class structures such as runtime constant pool, field and method data, and the code for methods and constructors, as well as interned Strings.

Deepak Bhatia
  • 6,230
  • 2
  • 24
  • 58
  • 1
    wouldn't the JVM remove any Weak/Soft references before throwing an OutOfMemory error in a last ditch effort to free up as much memory as possible? – dkatzel Dec 26 '13 at 17:05
  • 1
    yes it is true that JVM will try to free all the possible memory before throwing `OutOfMemory` exception but this is not related to question and i also did not mentioned any – Deepak Bhatia Dec 27 '13 at 04:51
  • 1
    I'll check out yourkit – VoidPointer Dec 27 '13 at 13:23
3

Since the extra memory is not showing in MAT it's hard to know what to suggest. My apologies if some (or even most) of this is things you already know, I've just tried to pull together everything I could think of.

FindBugs

FindBugs is a static analysis tool that will scan your code looking for common anti-patterns and problems and giving you a nice report on them. It does pick up on a lot of causes of potential memory and resource leaks.

Manual dump

You could try using something like jmap or visualvm to take a heap dump for analysis manually and see if you get different results from letting eclipse do it:

http://docs.oracle.com/javase/1.5.0/docs/tooldocs/share/jmap.html

http://java.dzone.com/articles/java-heap-dump-are-you-task

Analyzer Quirks

The memory analyzer FAQ:

http://wiki.eclipse.org/MemoryAnalyzer/FAQ

says:

Symptom: When monitoring the memory usage interactively, the used heap size is much bigger than what MAT reports.

During the index creation, the Memory Analyzer removes unreachable objects because the various garbage collector algorithms tend to leave some garbage behind (if the object is too small, moving and re-assigning addresses is to expensive). This should, however, be no more than 3 to 4 percent. If you want to know what objects are removed, enable debug output as explained here: MemoryAnalyzer/FAQ#Enable_Debug_Output

Another reason could be that the heap dump was not written properly. Especially older VM (1.4, 1.5) can have problems if the heap dump is written via jmap.

Enabling debug output will allow you to see what is going on there and confirm there is nothing odd in that area.

Some of these tips may be relevant

http://eclipsesource.com/blogs/2013/01/21/10-tips-for-using-the-eclipse-memory-analyzer/

Community
  • 1
  • 1
Tim B
  • 40,716
  • 16
  • 83
  • 128
2

Use JProfiler and break the heap object count down by class - find which class has lots of instances and start your hunt there.

You can also take a couple of snapshots a short time apart and compare the two heap dumps to see what objects were created during that time. This is particularly handy if you know that a certain action is causing the problem and you want to ignore all the background JVM object noise and just examine the delta.

I have used it with great success to find a memory leak. It isn't free, but it's worth the licence fee.

FYI: I have no affiliation with JProfiler.

Bohemian
  • 412,405
  • 93
  • 575
  • 722
2

Since the extra memory is not showing in MAT it's hard to know what to suggest. It isn't true. MAT show unreachable objects. Just go to de Preferences and select check box enabling this options. After MAT restart you will see these objects with details. Of course roots to GC will be not available.

Krashan Brahmanjara
  • 226
  • 2
  • 5
0

Maybe you should look for memory leaks in database connector code or maybe ORM. Because if you are using raw connection library when you don't close cursor you can get potentially memory leak. Also my second thought is also related to database connector. Because some of them (may be not yours) uses native code beneath and this is source of this leak. Due to heavy concurrent usage that makes sens for me. You can check that if you want.

bkowalikpl
  • 817
  • 5
  • 11