7

I have a java program which keeps calling java.util.zip to compress/decompress data. It runs out of memory within a few seconds. I had a memory dump with jmap and I'm viewing it with jhat.

Finalizer summary shows Total instances pending finalization: 0. If I understand it correctly, I don't have any objects which (1) have finalize() method, (2) have been marked by GC and (3) are waiting to be finalized. This seems good.

When I look at a particular object, the only reference to this object is a java.lang.ref.Finalizer. The Finalizer object is created for each object which has a finalize() method no matter if the object is GC'ed or not. So it looks like nothing is preventing this this Deflater object to be GC'ed.

Object at 0x7f4aeb7a35d0

instance of java.util.zip.Deflater@0x7f4aeb7a35d0 (51 bytes)

References to this object:

java.lang.ref.Finalizer@0x7f4aeb8607c8 (64 bytes) : field referent

The program is paused in the run by System.in.read(). The memory usage doesn't go down after a while.

UPDATE:

I should make it clear. The memory dump shows many objects were not GC'ed but no other objects (except Finalizer objects) were referencing them. I'm trying to find out why they were not GC'ed.

Community
  • 1
  • 1
woodings
  • 7,503
  • 5
  • 34
  • 52
  • @dashrb I don't work with Deflater in my code. I'm using Apache Thrift which uses Deflater. Deflater is not in scope as `Show all members of the rootset` shows only 1 reference to Deflater class but to no instances. – woodings Nov 21 '12 at 19:00
  • 1
    Actually, the fact that you are running out of memory and have no objects that have been marked by GC is _bad_. That means your code (or some library code you use) is keeping references to objects so the GC can't collect them. – Ted Hopp Nov 21 '12 at 19:00
  • @TedHopp that's what I'm going to find out. I don't know how to find those references as memory dump shows there were not any references. – woodings Nov 21 '12 at 19:02
  • If you use jmap to print a histogram of object allocations, that might show what kind of objects are chewing up memory and give you a clue as to where to look next. There's a nice article [here](http://www.infoq.com/articles/java-profiling-with-open-source) on tools for tracking down memory leaks. – Ted Hopp Nov 21 '12 at 19:15
  • Why not use a memory profile to see what exactly is alive and who is referencing whom? – NPE Nov 25 '12 at 08:12
  • You could try something more user-friendly to understand your memory dump... Eclipse MAT is an option: http://www.eclipse.org/mat/ – Flavio Nov 25 '12 at 08:21
  • OutOfMemory error is usually contains a hint whether there is a problem with heap, PermGen, fragmentation, GC, etc. Are you sure your heap is exhausted? – mazaneicha Nov 26 '12 at 02:32
  • Make a for loop that will call System.gc() 5 times. Report back. – Shark Nov 29 '12 at 15:28
  • Unzip the file manually and take a look at how much data is inside. All of the Java Zip implementations I have found have made the assumption that the entire file can be read into memory and decompressed there. – Devon_C_Miller Nov 29 '12 at 22:15

5 Answers5

5

Are you positive you closed the stream and invoked end on the deflater? I'm sorry if this is a simplistic suggestion you've already tried, but there were a lot of complaints about a memory leak when using Deflater when not immediately invoking end on it, for example:

The root cause is apparently that the collector is not able to keep up with the application when memory used by native elements is involved. That also explains the behavior you are seeing when profiling: the memory is ready to be reclaimed, but just isn't reclaimed quickly enough.

Since you wrote you don't use a Deflator directly but through Apache Thrift, try to identify which method in that library is responsible for ending the deflator, and made sure you invoked that method.

Community
  • 1
  • 1
Oak
  • 26,231
  • 8
  • 93
  • 152
1

The base problem is that you aren't freeing objects from the C heap. Many classes in java.util.zip use Deflater. Deflater maintains a reference to data in the C heap. Your code is likely not calling close() on ZipOutputStream or DeflaterOutputStream, or it's not calling end() on Deflater.

(If you are passing your own Deflater to ZipOutputStream or DeflaterOutputStream, you are responsible for calling end() on the Deflater.)

GC is of limited help in your case because the stream(s) must be dereferenced and eventually finalized. This may take multiple GC sweeps. I experienced a similar problem with Jetty and proposed a fix for it.

mikeslattery
  • 4,039
  • 1
  • 19
  • 14
1

Adding to Marian-Daniel Craciunescu's answer

I did have this kind of problem with another type of use.

The solution I found is the following :

  • Put the object into a map or somewhere you can easily drop its reference
  • Put also the object into a WeakReference for using it
  • Use your reference as normal
  • Delete the reference from the map where no more used

Be careful of inherited references

It is greatly possible your object is "hidden-inner-referenced" in another object, use the debugger and "expand" all objects using it.

You will never know when the GC is running, nor what it will handle (except objects in use).

Useful link

You should read (if it is not already done) this post about WeakReferences : Understanding Java's Reference classes: SoftReference, WeakReference, and PhantomReference

Community
  • 1
  • 1
Benj
  • 1,184
  • 7
  • 26
  • 57
0

Firstly before thinking about memory leak we should try to understand if the memory requirement is normal for the processing you are doing. What is the size of memory available to JVM i.e -XmX. If you increase the size of memory or decrease the size of file being processed does work?

Ashwinee K Jha
  • 9,187
  • 2
  • 25
  • 19
-1

Maybe you should use a WeakReference for your object.In this way GC may finalize () your object much quickly

http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/ref/WeakReference.html http://weblogs.java.net/blog/2006/05/04/understanding-weak-references

Marian-Daniel Craciunescu
  • 164
  • 6