22

This is yet another "please tell me how to force the Java garbage collector to run" question. In our application, I believe we have good reasons for doing this.

This is a server application, which typically has around 5M live objects. Once every 5 minutes, we perform an analysis task which takes ~60 seconds. If a full GC is triggered while the analysis is running, there will be around 40M live objects. The extra 35M objects become garbage when the analysis completes. The server must remain responsive to requests at all times (even while the analysis is running).

We've found that a full GC takes around 1.5 seconds if invoked when the analysis is not running, but around 15 seconds while the analysis is running. Unfortunately, our allocation pattern is such that full GCs usually trigger during the analysis, even though the analysis is only running 20% of the time. (Every third or fourth analysis run triggers a full GC.)

I added code to call the much-scorned System.gc() just before beginning an analysis run, if free space in the old generation is below a certain threshold (5GB). The benefit was very substantial: we're getting 1.5 second pause times instead of 15 second pause times, and we free more garbage into the bargain. However, sometimes the System.gc() call is ignored, and we wind up with a 15-second pause a few minutes later when the GC is triggered automatically.

My question, then: is there something we can do to more strongly convince the garbage collector to run? We're running 1.7.0_09-icedtea and using the Parallel GC. I'd like either (a) a reliable way to manually force garbage collection, or (b) some way to tune the collector so that it makes a more intelligent automatic decision. (b) seems hard, as it's not clear to me how the collector could detect that our working set varies in this dramatic fashion.

I'm willing to resort to substantial hackery if need be; this is a serious issue for us. (We might look into the CMS or G1 compactors as alternatives, but I'm leery of the throughput impact of CMS, and G1 is reputed to behave poorly in the face of large byte arrays, which we use.)

addendum: In production, our experience so far has been that System.gc() usually does trigger a full garbage collection; at least, under the situations where we're calling it. (We only call it once every 10 to 30 minutes, with the heap somewhat but not completely filled with garbage.) It would be nice to be able to trigger garbage collection more reliably, but it is helping us most of the time.

Steve
  • 1,033
  • 1
  • 10
  • 19
  • You've clearly done your research! I hope that you get a good answer; I'm curious to learn more about this as well! – templatetypedef Oct 17 '13 at 00:35
  • I've often wondered about this -- there *are* legitimate reasons for forcing garbage collection, though they seem to be very, very rare. Perhaps what one needs is a special JVM with a GC hook in it. – arcy Oct 17 '13 at 00:46
  • Have you tried just making the heap smaller? – Hot Licks Oct 17 '13 at 01:00
  • @HotLicks: We have not tried shrinking the heap. Do you have a reason for thinking that would help? My understanding is that GC cost is mostly determined by the amount of live (non-garbage) data on the heap. Shrinking the heap would force GC to trigger with less garbage, but there would be the same amount of live data; and on top of that, the GCs would run more often. – Steve Oct 17 '13 at 01:05
  • Have you thought of using off-heap based data structures like mapdb or mmap backed structures like journal etc? Would be faster than rolling your own from scratch. And if you go for a mmap based approach - you would actually move closer to a multi-jvm solution. – quixver Oct 03 '14 at 09:34
  • See also http://stackoverflow.com/questions/2448941/force-full-garbage-collection-when-memory-occupation-goes-beyond-a-certain-thres? – Raedwald Mar 31 '16 at 16:10

4 Answers4

6

Your problem is that you're running two applications with entirely different requirements and memory profiles in the same JVM.

Run the data analysis separately, in a non-user-facing process, so that the user-facing server remains constantly responsive. I assume the periodic analysis generates a summary or result data of some kind; make that available to end users by shipping it across to the user-facing server so it can be served from there, or else let your front end fetch it separately from the analysis server.

DPM
  • 1,571
  • 11
  • 8
  • That's not a bad point. Unfortunately, the two parts are deeply entwined, sharing a significant amount of state. It would probably be possible to split them apart but it would be a significant project. We might undertake this as a last resort. (In our case, moving the bulkiest data structures off-heap, as maaartinus suggests, would be some work but probably not as difficult as shipping the analysis to a separate process.) – Steve Oct 17 '13 at 14:56
2

Consider using non-managed memory, i.e., ByteBuffers in place of the byte arrays.

I can only offer a hack which will need some tuning and then might or might not work. I'd first try the more sane solutions. When you want to force the GC, do it by allocating a lot of memory. Do this so that the memory can be immediately reclaimed, but so that the whole allocation can't be optimized away (something like sum += new byte[123456].hashCode() should do). You'll need to find a reliable method for determining when to stop. An object with a finalizer might tell you or maybe watching runtime.getFreeMemory could help.

maaartinus
  • 44,714
  • 32
  • 161
  • 320
  • 1
    Forcing a GC by allocating a lot of memory scares me. I could easily see this causing more problems than it solves. We'd have to get it just right, allocating enough memory to trigger a GC of the tenured generation, but not so much that the allocations themselves crush the system (or create so much live data as to cause the precise problem we were hoping to avoid). Non-managed memory might be a solution; it would be a nontrivial project, but doable. We already use non-managed memory for our largest data structures, we'd have to extend this to cover some of the analysis data structures. – Steve Oct 17 '13 at 15:00
1

Rather than answer your question directly (I can't), I'd like to offer a possible alternative.

It sounds like you are allocating a large number of large byte arrays during your analysis run, and then allowing them to be garbage collected at the end of the run (or attempting to force them to be garbage collected just before the next run).

Instead, if possible, try managing your own pool of byte arrays, so that, in the best case, you allocate all of the needed arrays once when the application is first started, and then they live for the lifetime of the application, and don't need to garbage collected.

This idea can, of course, be extended to more complex data structures and object instances.

This is all quite a bit more work than just allocating memory when you need it, and 'freeing' it when you don't, but should cut down considerably on the work that the garbage collector needs to do.

GreyBeardedGeek
  • 29,460
  • 2
  • 47
  • 67
-1

I found that java GC deals very poorly with a large number of objects (20-100m objects). your situation would have been worse if those objects would actually had remain alive, because GC would be horrible even if there was nothing to actually collect.

the solution is to reduce the number of objects (not the total memory you are using). I would dare to guess that you analysis phase is using collections and many primitive wrappers (Integer, Long etc). if this is the case, one solution is to switch to a primitive collections library. one such library is what I created to solve a similar problem I encountered where I ran a simulation with 100m live objects for a long time. That library is called Banana, see the wiki for details.

Omry Yadan
  • 31,280
  • 18
  • 64
  • 87
  • Just to add, there are also HPPC, Trove and FastUtil which do primitive collections - every one has a slightly different reasoning and approach. You might want to take a look at all of them. – Petr Janeček Oct 17 '13 at 17:13