Explain observed JVM Garbage Collection on JBoss Server

Question

With VisualVM I am observing the following heap usage on a JBoss server:

The server is started with the following (relevant) JVM options:

-Xrs -Xms3072m -Xmx3072m -XX:MaxPermSize=512m -XX:+UseParallelOldGC -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000

And we currently also have enabled GC logging:

-XX:+PrintGC -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xloggc:log\gc.log

Basically I am happy with the observed pattern, since it looks like we don't have any memory leaks (the pattern repeats itself over days).

However I am wondering if there is room for optimization?

First of all, I don't understand why the garbage collection already kicks in when the heap usage reaches about 2GB? It looks to me like it could kick in later since the heap would have 3GB available?

Further more I would be interested in tips regarding the observed heap usage pattern and the used JVM options:

• Does the observed pattern allow me to draw conclusions about the used GC strategy (UseParallelOldGC)? Ist this strategy the right one, or should I try to use another one given the observed heap usage?

• Can I optimize the GC process, so that the full heap size (3GB) is used?

• Right now it looks like the full 3GB are never used, should I reduce the Xms/Xmx to 2.5GB?

• Are there any obvious GC optimizations that I am missing? Like tuning -XX:NewSize or -XX:NewRatio?

• Any other tips that come to mind?

Thanks!

Answer 1

I'd say the GC behaviour in your screen-shot looks 'normal'.

You'd usually want major collections to trigger before the heap space gets too full or it would be very easy to encounter OutOfMemoryError's, based on a number of scenarios.

Also, are you aware that Java's heap space is divided into distinct areas for new (eden), current (survivor) and old (tenured) objects?

This answer provides some excellent information on the subject, so I won't repeat it here:

How is the java memory pool divided?

Very basically, each area of the heap triggers its own collections. The eden space is normally collected often and 'quickly' the survivor and tenured spaces are usually larger and take longer to collect.

Could you reduce your heap size based on the above graph?

Yes. However, your current configuration allows your application some breathing room, if it's ever likely to encounter busier periods or spikes in load.

Can you optimize GC?

Yes, but there are no magic settings. The first question is do you really need to? If your application is just a non-interactive 'processor', I really wouldn't bother. If you have a genuine need for a low pause application, then there are some tweaks available. The trade off is generally that you'll need more resources to achieve the same result.

My experience is that low-pause JVM configurations have a very noticeable fall-off point when load increases. If your application is usually fairly idle, but you expect a 'quick' response when it is called, low pause may be appropriate. On a busier system, with peaks in traffic / load, you may prefer a more traditional approach.

Summary

In any case, don't be tempted to make arbitrary changes to 'improve' your configuration. Be scientific and professional about your approach.

If you don't have production metrics available, consider using tools like Apache JMeter to build load test scenarios to simulate the typical live load on you application, increased load (by say, 10%, 20% or 50% etc.) and intermittent peak load.

Use metrics for both the GC and the application, measuring at least:

• Average throughput.
• Peak throughput.
• Average load (CPU and memory).
• Peak load.
• Application pause times (total and individual pauses).
• Time spent performing collections.
• Reliability (OOME's etc.).

Once you're happy that you've recorded an accurate benchmark on the performance of you application with its current configuration, only then should you start making any changes.

Obviously, record you configuration and its metrics. Document any changes and then perform the same benchmark tests. Then you'll be able to see any performance gain (or loss) and any trade-off that may be applicable.

Here's the some further reading from Oracle on the subject to get you started:

Java SE 6 Virtual Machine Garbage Collection Tuning