4

Environment:

sun JDK 1.6.0_16

vm settings:
-XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -Xms1024 -Xmx1024M -XX:MaxNewSize=448m -XX:NewSize=448m -XX:SurvivorRatio=4(6 also checked) -XX:MaxPermSize=128M

OS: windows server 2003

processor: 4 cores of INTEL XEON 5130,  2000 Hz

my application description: high intensity of concurrent(java 5 concurrency used) operations completed each time by commit to oracle. it's about 20-30 threads run non stop, doing tasks.

application runs in JBOSS web container.

My GC starts work normally, I see a lot of small GCs and all that time CPU shows good load, like all 4 cores loaded to 40-50%, CPU graph is stable.

Then , after 1 min of good work, CPU starts drop to 0% on 2 cores from 4, it's graph becomes unstable, goes up and down("teeth"). I see, that my threads work slower(I have monitoring), I see that GC starts produce a lot of FULL GC during that time and next 4-5 minutes this situation remains as is, then for short period of time, like 1 minute, it gets back to normal situation, but shortly after that all bad thing repeats.

Question: Why I have so frequent full GC??? How to prevent that?

I played with SurvivorRatio - does not help.

I noticed, that application behaves normally until first FULL GC occurs, while I have enough memory. Then it runs badly.

my GC LOG:

  1. starts good
  2. then long period of FULL GCs(many of them)
1027.861: [GC 942200K->623526K(991232K), 0.0887588 secs]
1029.333: [GC 803279K(991232K), 0.0927470 secs]
1030.551: [GC 967485K->625549K(991232K), 0.0823024 secs]
1030.634: [GC 625957K(991232K), 0.0763656 secs]
1033.126: [GC 969613K->632963K(991232K), 0.0850611 secs]
1033.281: [GC 649899K(991232K), 0.0378358 secs]
1035.910: [GC 813948K(991232K), 0.3540375 secs]
1037.994: [GC 967729K->637198K(991232K), 0.0826042 secs]
1038.435: [GC 710309K(991232K), 0.1370703 secs]
1039.665: [GC 980494K->972462K(991232K), 0.6398589 secs]
1040.306: [Full GC 972462K->619643K(991232K), 3.7780597 secs]
1044.093: [GC 620103K(991232K), 0.0695221 secs]
1047.870: [Full GC 991231K->626514K(991232K), 3.8732457 secs]
1053.739: [GC 942140K(991232K), 0.5410483 secs]
1056.343: [Full GC 991232K->634157K(991232K), 3.9071443 secs]
1061.257: [GC 786274K(991232K), 0.3106603 secs]
1065.229: [Full GC 991232K->641617K(991232K), 3.9565638 secs]
1071.192: [GC 945999K(991232K), 0.5401515 secs]
1073.793: [Full GC 991231K->648045K(991232K), 3.9627814 secs]
1079.754: [GC 936641K(991232K), 0.5321197 secs]
Andreas Dolk
  • 113,398
  • 19
  • 180
  • 268

2 Answers2

3

From the last two lines of your log (before you edited your question):

397.245: [Full GC 660160K->443379K(660160K), 2.7433121 secs]
401.793: [Full GC 660160K->446464K(660160K), 2.7697340 secs]

You definitly hit a memory limit, which is 660160K. This limit is the total available space, not counting the space in the permanent generation, which is the total heap minus one of the survivor spaces. (Ref)

Every four seconds you create ~220M of new, gc'able objects, the total memory usage seems to increase at a rate of 1 MB/sec. So after a while, the JVM will do nothing but Full GC's until running out of heap space.

In this scenario, I stongly doubt, that tweaking memory setting will help for a long time, just because you hit the limit after 6 minutes already.

Looks like you have to look for memory leaks or stored reference to large, unused objects (result sets, DOM objects, ...)

With you new values: Same situation. You still hit the same upper memory limit, just that it occures some minutes later, because you assigned more memory. Really smells like a memory leak. You're still producing gc'able content at a rate of ~60 MB/s.

Andreas Dolk
  • 113,398
  • 19
  • 180
  • 268
  • my gc output was wrong . it was for lesser memory settings, right one is about Full GC 991231K->648045K(991232K). I updated my question. I checked 12 hours of running system there were no OOM exceptions. It managed to run. Thanks anyway. I have JProfiler, but still did not find the problem. –  Jan 04 '11 at 08:57
2

This doesn't look like a case of memory leaks; it's quite possible that all that memory actually is being used by the JVM which might explain the frequent full GC's. Is there any chance you can spawn multiple processes? i.e. instead of a single process having 20-30 threads, having 5 processes running 4-5 threads each?

Also, is there any reason why you have those NewSize, MaxNewSize and SurvivorRatio JVM parameters in place? Have you observed any significant performance gains by having those things there? My first approach when tuning any application is to run it with minimum tunings to the JVM and only make changes if my newly added parameters do have an impact on the situation at hand.

Sanjay T. Sharma
  • 22,857
  • 4
  • 59
  • 71
  • 1
    you were right. I left only Xmx and Xms, it became more stable. thanks a lot!!! defaults help. –  Jan 04 '11 at 14:58
  • @Peter: Glad it worked out for you. If these defaults end up solving your full GC issue, please either update your original question or mark this question as solved. – Sanjay T. Sharma Jan 04 '11 at 15:23
  • The question resolved by defaults. But what are defaults in my case? this code allows get those "hidden" defaults, which I use if I do not provide my parameters http://stackoverflow.com/questions/1880299/what-gc-parameters-is-a-jvm-running-with im my case it is: NewSize=16777216 MaxNewSize is 67108864, SurvivorRatio=8. I see real improvements in GC, less Full GCs, but I also reduced test load to that we have in production, so resolution in my case not only in GC parameters, but in reduced throughput. This vm is not able support my initial load no matter what GC parameters used. –  Jan 05 '11 at 12:27
  • @Peter: Interesting link. Regarding your question, like I mentioned before, it is very much possible that your VM actually requires all that memory and hence is not able to perform to its optimal. I've personally observed that giving the VM more mem. than required for usual processing results in better performance. You still haven't mentioned whether you are running a 32bit or 64bit VM (I guess it's 32bit). Because, if it is 64 bit VM, increasing the `Xmx` values might be an interesting exercise. Also, you still haven't relied to my query as to whether multiple processes are a possibility? – Sanjay T. Sharma Jan 05 '11 at 13:10
  • "Is there any chance you can spawn multiple processes?" , how can I do it ? Is there any JVM parameter available for this. I have 16 core CPU but never gave any such parameter. – Vipin Mar 15 '14 at 09:27
  • @Vipin: Maybe I was a bit unclear. What I meant was that instead of having a single JVM process with a huge heap, you can spawn multiple processes explicitly (something like a cluster) assuming your use case supports it. For e.g. I once worked on a compute engine which was almost stateless and hence could be easily configured in cluster mode. – Sanjay T. Sharma Mar 15 '14 at 11:08