0

hello frnds we have migrated our java web application on java 1.6 , tomcat 6, oracle 11g but now we are facing issues after some time site is going down : we are getting java heap out of memory error and many threads are in waiting state. tomcat is getting hanged and our site is getting down

INFO: Maximum number of threads (200) created for connector with address null and port 80 Feb 21, 2013 9:56:04 PM here is thread log:

logs 
"main" prio=10 tid=0x09f67c00 nid=0x2d51 runnable [0xf7622000]
       java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
        - locked <0xc5078c60> (a java.net.SocksSocketImpl)          
    "VM Thread" prio=10 tid=0x09f8ac00 nid=0x2d52 runnable     
    "VM Periodic Task Thread" prio=10 tid=0x09f9cc00 nid=0x2d58 waiting on condition     
    JNI global references: 1074    
    Heap
     def new generation   total 78656K, used 78533K [0xbf760000, 0xc4cb0000, 0xc4cb0000)
      eden space 69952K, 100% used [0xbf760000, 0xc3bb0000, 0xc3bb0000)
      from space 8704K,  98% used [0xc4430000, 0xc4c91428, 0xc4cb0000)
      to   space 8704K,   0% used [0xc3bb0000, 0xc3bb0000, 0xc4430000)
     tenured generation   total 174784K, used 174783K [0xc4cb0000, 0xcf760000, 0xcf760000)
       the space 174784K,  99% used [0xc4cb0000, 0xcf75fff8, 0xcf760000, 0xcf760000)
     compacting perm gen  total 33792K, used 33647K [0xcf760000, 0xd1860000, 0xd3760000)
       the space 33792K,  99% used [0xcf760000, 0xd183be68, 0xd183c000, 0xd1860000)
        ro space 10240K,  61% used [0xd3760000, 0xd3d86298, 0xd3d86400, 0xd4160000)
        rw space 12288K,  60% used [0xd4160000, 0xd4896cb8, 0xd4896e00, 0xd4d60000)
end

Please guide.

Thanks, Aru

aru
  • 59
  • 1
  • 7
  • 16
  • Threads in WAITING state and OutOfMemoryErrors are completely different things. The first must not be a problem at all (but can be) and must not be related to the latter. I would face the OOM errors first. Is the heap and permgen size set to the same than before the migration? – MRalwasser Feb 28 '13 at 17:30
  • @MRalwasser: we are using these options JAVA_OPTIONS="-Xms512m -Xmx2048m -XX:MaxPermSize=512m -verbose:gc";export JAVA_OPTIONS – aru Feb 28 '13 at 17:32
  • Try JAVA_OPTS instead JAVA_OPTIONS. – Szymon Jednac Feb 28 '13 at 18:05

2 Answers2

1

According to the log you posted, your jvm settings (defined in JAVA_OPTIONS) mentioned in the comment above are not respected/used at all.
(e.g. 32m for permgen is used instead of the specified 512m).

My guess is that you just used the wrong name for the environment variable:
The correct environment variable for specifying JVM settings in tomcat is named JAVA_OPTS.

MRalwasser
  • 15,605
  • 15
  • 101
  • 147
  • could you guide how i can investigate i am new to tomcat one thing i noticed in the start tomcat script whereever i hv written export it's giving me error " : invalid identifier" – aru Feb 28 '13 at 17:41
  • You should not change the tomcat scripts at all. Just set the environment variable (on unix, invoke: export JAVA_OPTS="-Xms512m -Xmx2048m -XX:MaxPermSize=512m" ) before you execute startup.sh – MRalwasser Feb 28 '13 at 17:44
  • if i hv user java_options then it means it is using some default values it is not all reading java_options – aru Feb 28 '13 at 17:53
  • as I stated in my answer, the correct name is JAVA_OPTS - not java_options. Revert all changes you did to the tomcat scripts and invoke the command stated in my previous comment - this will most probably resolve your issue. – MRalwasser Feb 28 '13 at 18:25
  • thanks for your help i will use ur sugessted changes could you tell me command in unix to see the default jvm heap size – aru Feb 28 '13 at 18:55
  • look here: http://stackoverflow.com/questions/2915276/what-is-the-default-maximum-heap-size-for-suns-jvm-from-java-se-6 - according to your log file your defaults are 256m (heap) and 32m (permgen) – MRalwasser Feb 28 '13 at 21:22
  • thanks for all your help yes the culprit was JAva_options application has started working fine . i am worried about one thing tanured gen memory keep on increasing , shoud i worry about this or it would be empty itself Time:  2013-03-01 22:48:17 Used:     203,013 kbytes Committed:     349,568 kbytes Max:   1,398,144 kbytes GC time:  10.388 seconds on Copy (216 collections)  1.754 seconds on MarkSweepCompact (5 collections) – aru Mar 01 '13 at 17:17
  • I'm glad that I could help (and I would be happy if you would upvote and/or accept my answer). Increasing tanured must not be a problem, don't be worried, especially if the same application ran fine on earlier tomcat releases. Monitor your application - if an OutOfMemoryError occurs again then somethings need to get fixed (most probably the application itself). But until then - relax ;) – MRalwasser Mar 01 '13 at 17:57
  • i am new to this community could you tell me what is upvote and how can i accept your answer – aru Mar 01 '13 at 18:11
  • Left to my answer, click on the UP-Arrow to upvote and click on the check (below to the DOWN-Arrow) to accept the answer. – MRalwasser Mar 01 '13 at 18:15
0

Looks like the perm gen space is all used. Have you tried increasing it with -XX:MaxPermSize=128M ?

jontro
  • 10,241
  • 6
  • 46
  • 71