3

I have an application that uses a lot of memory diff'ing the contents of two potentially huge (100k+) directories. It makes sense to me that such an operation would use a lot of memory, but once my diff'ing operation is done, the heap remains the same size.

I basically have code that instantiates a class to store the filename, file size, path, and modification date for each file on the source and target. I save the additions, deletions, and updates in other arrays. I then clear() my source and target arrays (which could be 100k+ each by now), leaving relatively small additions, deletions, and updates arrays left.

After I clear() my target and source arrays though, the memory usage (as visible via VirtualVM and Windows Task Manager) doesn't drop. I'm not experienced enough with VirtualVM (or any profiler for that matter) to figure out what is taking up all this memory. VirtualVM's heap dump lists the top few objects with a retained size of a few megabytes.

Anything to help point me in the right direction?

tau
  • 6,499
  • 10
  • 37
  • 60
  • 1
    are you sure the garbage collector has already run when you look at the memory usage? Memory usage will only go down once a (probably major) GC has run. – cello Oct 27 '14 at 08:03
  • according to virtualvm, the heap stays about the same while the used heap will go down (to about where i would expect it, which is way less than half the total heap). the heap size roughly aligns with what windows task manager is telling me java is using. i see the usual GC staircase collection pattern when profiling the app, i just dont know why the total heap is so large still. – tau Oct 27 '14 at 08:14

3 Answers3

2

If the used heap goes down after a Garbage Collection, than it likely works as expected. Java increases its heap when it needs more memory, but does not free it -- it prefers to keep it in case the application uses more memory again. See Is there a way to lower Java heap when not in use? for this topic on why the heap is not reduced after the used heap amount lowers.

Community
  • 1
  • 1
cello
  • 5,356
  • 3
  • 23
  • 28
  • i just glanced at that thread and it was already highly informative. thank you! – tau Oct 27 '14 at 08:27
2

The VM grows or shrinks the heap based on the command-line parameters -XX:MinHeapFreeRatio and -XX:MaxHeapFreeRatio. It will shrink the heap when the free percentage hits -XX:MaxHeapFreeRatio, whose default is 70.

There is a short discussion of this in Oracle's bug #6498735.

RealSkeptic
  • 33,993
  • 7
  • 53
  • 79
1

Depending on your code you might be generating memory leaks and the Garbage collector just can't free them up.

I would suggest to instrument your code in order to find potential memory leaks. Once this is ruled out or fixed, I would start to look at the code itself for possible improvement.

Note that for instance if you use the try/catch/finally block. The finally block might not be called at all (or at least not immediately). If you do some resource freeing in a finally block this might be the answer.

Nevertheless read up on the subject, for instance here: http://www.toptal.com/java/hunting-memory-leaks-in-java

Drejc
  • 14,196
  • 16
  • 71
  • 106