7

I'm trying to tune my application mainly to reduce the "spread" of response times. The average is fine, but the range is too wide.

Dynatrace showed that the higher response times are associated with a larger amount of time spent in suspension. This points to GC.

I've tried changing some JVM GC values based on reading online, to little success.

Based on GC logs I've figured the Allocation Rate to be about 324 MB/s, and the Promotion Rate to be only 0.85 MB/s. To me this seems that it has a very high allocation rate, so I've tried to increase the size of the Young Generation.

The first screenshot is with default Java 8 settings, 1024MB Xmx.

The second screenshot is setting NewRatio=1.

Any suggestions on what to try next would be really appreciated.

Things I've already tried: changing to G1GC, setting NewRatio=1, setting NewRatio=1 and increasing Xmx to 2048, setting NewSize=1600m and Xmx=2048, setting MetaspaceSize=100

Edit: Adding GC logs: http://pastebin.com/VhJwSuxv

Note: These logs are from the 10min test with the change: NewRatio=1

enter image description here enter image description here

Bobby
  • 1,666
  • 3
  • 16
  • 27
  • 1
    *"Based on GC logs"*, post them? – the8472 May 03 '16 at 15:26
  • Added link to pastebin with GC logs. Note these are with the change: NewRatio=1 – Bobby May 03 '16 at 15:55
  • The young collections deliver a fairly consistent 70-80ms pause. Old collections take longer, but they are rare and that should be expected when you're using the throughput collector. So you probably should get more concrete on the responsiveness measures to verify that they're actually GC-related. – the8472 May 03 '16 at 16:38
  • I see your point. My trouble there is the customer sees high "Suspension Time" in their Dynatrace tool. But I see these pretty reasonable times in the GC logs (I don't have Dynatrace). Maybe it's an issue with their tool that we don't understand. – Bobby May 04 '16 at 11:58
  • Then you should figure out what precisely they are measuring. – the8472 May 04 '16 at 13:36

1 Answers1

4

Before tuning the GC, you should use a memory profiler to try to reduce the allocation rate in the first place.

You can try increasing the young generation further, e.g. you can set -Xmn2g -Xmx3g and not set the NewRatio.

I've figured the Allocation Rate to be about 324 MB/s,

This is moderate rate for a web application. It could be lower but I wouldn't expect significant problems at this level.

I would start with a much larger heap depending on how memory your machine has e.g. -Xmn24g -Xmx32g and look at the pause times. Then reduce the heap size until it appears to be impacting the GC times (they might even get shorter).

Another way to look at it; You might find it is acceptable to have a minor GC every 2 to 10 seconds. This means you want an Eden space of 650 MB to 3.2 GB.

Peter Lawrey
  • 525,659
  • 79
  • 751
  • 1,130
  • Thanks for the reply. Can you give me an example of a "memory profiler" you suggest? I've been using VisualVM. Haven't been able to get jstatd working on remote server. I'll try with much larger Xmx and Xmn as you suggest. I did try with NewSize=1600/Xmx=2048 and this reduced the minor gc frequency from every 1s to ever 4s, however the duration of those gc's took about 3x as long each, having an overall negative impact on perf. – Bobby May 03 '16 at 15:37
  • @Bobby I would try FlightRecorder. This can be triggered from the command line and give a lot more detail as to what you application is doing. – Peter Lawrey May 03 '16 at 16:13
  • 1
    OK I tried your suggestion with much bigger heap and different New sizes and came to a result that gives some increase in performance and improved consistency. Next will be FlightRecorder. Thanks again for the replies! – Bobby May 04 '16 at 17:54