Segmentation Fault in Java

Question

Recently my Java application has started to hang. It has been running fine for over a year. After the program runs for a few hours or overnight, I come in the next day and the application is basically frozen. Rebooting the server makes it work again, but after some time goes by... the problem comes back.

When looking at "top" from the command line I find that Java is at 100% CPU usage! I read about this in other threads and forums and was able to use the JSTACK function to find the problem:

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0xf72e8791, pid=2197, tid=2979408784

JRE version: 6.0_39-b04 Java VM: Java HotSpot (TM) Server VM (20.14-b01 mixed mode linux-x86 ) Problematic frame: C [libsaproc.so+0x1791] long double restric_0x1d

Here is the major problem... I am not a Java coder. I maintain and support the overall system, but this is just one part. The vendor of the product has laid off the software guy that wrote all this, so they have not been a very big help at all. I really want to get this fixed so I don't have to reboot my server every single morning when I arrive to work.

I am a fairly competent, but novice programmer (learning ASP.NET). So I think that I may be able to handle this issue on my own, I just don't really know WHERE to begin. I understand that it is likely a problem in the code, but it seems odd to me that it is happening all of the sudden and never in the past.

There is a lot more information in the hs_err_pidxxxxx.log file, will that help me track down the EXACT line of code in error? Sadly, I don't even know where the code is stored in the folder hierarchy.

What more information can I supply to "help you help me"?

Edit 1: (Adding all of the jstack info - OS info is included in the system section) (removed to make room for edit 2)

System is CentOS 5.8

Edit 2 - (adding Cataline Out file from Tomcat)

Jul 3, 2015 9:49:47 AM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/amd64/server:/usr/lib/jvm/java-1$
Jul 3, 2015 9:49:48 AM org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on http-8080
Jul 3, 2015 9:49:48 AM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 1270 ms
Jul 3, 2015 9:49:48 AM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Jul 3, 2015 9:49:48 AM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.35
Jul 3, 2015 9:49:48 AM org.apache.catalina.startup.HostConfig deployDescriptor
INFO: Deploying configuration descriptor manager.xml
Jul 3, 2015 9:49:48 AM org.apache.catalina.session.StandardManager doLoad
SEVERE: IOException while loading persisted sessions: java.io.EOFException
java.io.EOFException
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2297)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2766)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:797)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:297)
        at org.apache.catalina.util.CustomObjectInputStream.<init>(CustomObjectInputStream.java:58)
        at org.apache.catalina.session.StandardManager.doLoad(StandardManager.java:362)
        at org.apache.catalina.session.StandardManager.load(StandardManager.java:321)
        at org.apache.catalina.session.StandardManager.start(StandardManager.java:648)
        at org.apache.catalina.core.StandardContext.start(StandardContext.java:4714)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
        at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:675)
        at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:601)
        at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
        at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
        at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
        at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
        at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
        at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
        at org.apache.catalina.core.StandardService.start(StandardService.java:525)
        at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Jul 3, 2015 9:49:48 AM org.apache.catalina.session.StandardManager start
SEVERE: Exception loading sessions from persistent storage
java.io.EOFException
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2297)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2766)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:797)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:297)
        at org.apache.catalina.util.CustomObjectInputStream.<init>(CustomObjectInputStream.java:58)
        at org.apache.catalina.session.StandardManager.doLoad(StandardManager.java:362)
        at org.apache.catalina.session.StandardManager.load(StandardManager.java:321)
        at org.apache.catalina.session.StandardManager.start(StandardManager.java:648)
        at org.apache.catalina.core.StandardContext.start(StandardContext.java:4714)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
        at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:675)
        at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:601)
        at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
        at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
        at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
        at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
        at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
        at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
        at org.apache.catalina.core.StandardService.start(StandardService.java:525)
        at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Jul 3, 2015 9:49:49 AM org.apache.catalina.startup.HostConfig deployDescriptor
INFO: Deploying configuration descriptor host-manager.xml
Jul 3, 2015 9:49:49 AM org.apache.catalina.session.StandardManager doLoad
SEVERE: IOException while loading persisted sessions: java.io.EOFException
java.io.EOFException
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2297)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2766)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:797)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:297)
        at org.apache.catalina.util.CustomObjectInputStream.<init>(CustomObjectInputStream.java:58)
        at org.apache.catalina.session.StandardManager.doLoad(StandardManager.java:362)
        at org.apache.catalina.session.StandardManager.load(StandardManager.java:321)
        at org.apache.catalina.session.StandardManager.start(StandardManager.java:648)
        at org.apache.catalina.core.StandardContext.start(StandardContext.java:4714)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
        at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:675)
        at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:601)
        at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
        at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
        at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
        at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
        at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
        at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
        at org.apache.catalina.core.StandardService.start(StandardService.java:525)
        at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Jul 3, 2015 9:49:49 AM org.apache.catalina.session.StandardManager start
SEVERE: Exception loading sessions from persistent storage
java.io.EOFException
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2297)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2766)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:797)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:297)
        at org.apache.catalina.util.CustomObjectInputStream.<init>(CustomObjectInputStream.java:58)
        at org.apache.catalina.session.StandardManager.doLoad(StandardManager.java:362)
        at org.apache.catalina.session.StandardManager.load(StandardManager.java:321)
        at org.apache.catalina.session.StandardManager.start(StandardManager.java:648)
        at org.apache.catalina.core.StandardContext.start(StandardContext.java:4714)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
        at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:675)
        at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:601)
        at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
        at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
        at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
        at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
        at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
        at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
        at org.apache.catalina.core.StandardService.start(StandardService.java:525)
        at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Jul 3, 2015 9:49:49 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory nms
Jul 3, 2015 9:49:59 AM org.apache.catalina.session.StandardManager doLoad
SEVERE: IOException while loading persisted sessions: java.io.EOFException
java.io.EOFException
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2297)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2766)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:797)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:297)
        at org.apache.catalina.util.CustomObjectInputStream.<init>(CustomObjectInputStream.java:58)
        at org.apache.catalina.session.StandardManager.doLoad(StandardManager.va:36275)
        at org.apache.catalina.session.StandardManager.load(StandardManager.java:321)
        at org.apache.catalina.session.StandardManager.start(StandardManager.java:648)
        at org.apache.catalina.core.StandardContext.start(StandardContext.java:4714)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
        at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1079)
        at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1002)
        at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:506)
        at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
        at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
        at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
        at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
        at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
        at org.apache.catalina.core.StandardService.start(StandardService.java:525)
        at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Jul 3, 2015 9:49:59 AM org.apache.catalina.session.StandardManager start
SEVERE: Exception loading sessions from persistent storage
java.io.EOFException
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2297)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2766)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:797)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:297)
        at org.apache.catalina.util.CustomObjectInputStream.<init>(CustomObjectInputStream.java:58)
        at org.apache.catalina.session.StandardManager.doLoad(StandardManager.java:362)
        at org.apache.catalina.session.StandardManager.load(StandardManager.java:321)
        at org.apache.catalina.session.StandardManager.start(StandardManager.a:648)
        at org.apache.catalina.core.StandardContext.start(StandardContext.java:4714)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
        at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1079)
        at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1002)
        at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:506)
        at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
        at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
        at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
        at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
        at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
        at org.apache.catalina.core.StandardService.start(StandardService.java:525)
        at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Jul 3, 2015 9:49:59 AM com.vaadin.terminal.gwt.server.AbstractApplicationServlet checkProductionMode
WARNING:
=================================================================
Vaadin is running in DEBUG MODE.
Add productionMode=true to web.xml to disable debug features.
To show debug window, add ?debug to your application URL.
=================================================================
Jul 3, 2015 9:50:34 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory ROOT
Jul 3, 2015 9:50:34 AM org.apache.catalina.session.StandardManager doLoad
SEVERE: IOException while loading persisted sessions: java.io.EOFException
java.io.EOFException
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2297)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2766)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:797)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:297)
        at org.apache.catalina.util.CustomObjectInputStream.<init>(CustomObjectInputStream.java:58)
        at org.apache.catalina.session.StandardManager.doLoad(StandardManager.java:362)
        at org.apache.catalina.session.StandardManager.load(StandardManager.java:321)
        at org.apache.catalina.session.StandardManager.start(StandardManager.java:648)
        at org.apache.catalina.core.StandardContext.start(StandardContext.java:4714)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
        at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1079)
        at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1002)
        at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:506)
        at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
        at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
        at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
        at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
        at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
        at org.apache.catalina.core.StandardService.start(StandardService.java:525)
        at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Jul 3, 2015 9:50:34 AM org.apache.catalina.session.StandardManager start
SEVERE: Exception loading sessions from persistent storage
java.io.EOFException
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2297)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2766)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:797)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:297)
        at org.apache.catalina.util.CustomObjectInputStream.<init>(CustomObjectInputStream.java:58)
        at org.apache.catalina.session.StandardManager.doLoad(StandardManager.java:362)
        at org.apache.catalina.session.StandardManager.load(StandardManager.java:321)
        at org.apache.catalina.session.StandardManager.start(StandardManager.java:648)
        at org.apache.catalina.core.StandardContext.start(StandardContext.java:4714)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
        at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1079)
        at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1002)
        at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:506)
        at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
        at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
        at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
        at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
        at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
        at org.apache.catalina.core.StandardService.start(StandardService.java:525)
        at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Jul 3, 2015 9:50:34 AM org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-8080
Jul 3, 2015 9:50:34 AM org.apache.jk.common.ChannelSocket init
INFO: JK: ajp13 listening on /0.0.0.0:8009
Jul 3, 2015 9:4 AM org.apache.jk.server.JkMain start
INFO: Jk running ID=0 time=0/111  config=null
Jul 3, 2015 9:50:34 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 45960 ms
Jul 3, 2015 7:46:00 PM org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor processChildren
SEVERE: Exception invoking periodic operation:
java.lang.OutOfMemoryError: Java heap space

Edit 3 - JSTAT info This is the JSTAT information from the process. It is not currently hanging, so whenever it hangs again I will get another JSTAT.

[root@localhost bin]# ./jstat -gc 3678
 S0C    S1C    S0U    S1U      EC       EU        OC         OU       PC     PU    YGC     YGCT    FGC    FGCT     GCT
7488.0 7360.0 6288.1  0.0   334656.0  3737.8  1594816.0  1401457.2  99968.0 61991.7   1068   21.244   7      5.625   26.869

Could you tell us if there has been any recent kernel update or more specifically in the shared library that looks to be firing the error? — Jorge_B, Jul 06 '15 at 14:40
The only thing that I can think of is on the MySQL side where I had to delete a unit out of the system. I replaced a unit in the field that reports back to the system, in order to get it to work again I had to delete the old unit and add a new one in so the serial addresses, etc were correct. I haven't done any changes to the Kernel or updates to the OS or anything like that. — Jason Goss, Jul 06 '15 at 14:42
*"JRE version: 6.0_39-b04"* <- that stuff is old. try upgrading and see if it goes away. Also, you should mention your operating system. and paste the whole crash log. update your question with that information. — the8472, Jul 06 '15 at 14:43
Keep in mind that executing jstack and having a segmentation fault while executing it (and I hope you're saying that the segmentation fault was server side and not in the jstack execution) may be totally misleading of what the actual problem is. — pabrantes, Jul 06 '15 at 14:44
Ok, I will update my question in a moment. Thanks for the information. — Jason Goss, Jul 06 '15 at 14:46
Java distributables don't necessarily come with their source code. Before going any further, I suggest you check that you actually have access to the source code. If you don't, then you should try to get in touch with the vendor. It's going to be a lot harder if you don't have the source. — Giulio Franco, Jul 06 '15 at 14:50
Where is that source code typically located? I have been searching through the different Java folders but don't know exactly what I am looking for. — Jason Goss, Jul 06 '15 at 14:58
@JasonGoss: This was what I was calling your attention to. What you're showing is a segmentation fault while trying to execute jstack, which is a **diferente process** that will attach to your server process (that is hanging). I've already seen jstacks failing when the server JVM is totally screwed and couldn't even connect. Running jstack with the -F switch might help you retrieve the actual jstack information. But bottom line beware because that problem might be totally misleading to the real problem. — pabrantes, Jul 06 '15 at 15:02
@pabrantes I see what you are saying. I did have to use -F just to get this jstack log. Is there another way I can hunt out problems or debug to find this while the system is running. If I reboot everything will be working fine and then maybe it can create a log when an error occurs during normal operation. — Jason Goss, Jul 06 '15 at 15:06
It may be more reliable to obtain a stack trace by killing the JVM with `kill -3 pid`. At the very least it's worth a try to see if you're stuck in the same library as with `jstack` — Dmitry Grigoryev, Jul 06 '15 at 15:09
@JasonGoss: Even if you run multiple times jstack you'll always get the segmentation fault? if you're running your application inside a container (tomcat, glassfish, jetty, to name a feew) it's expected to exist application logs that might have more information, you might find some helpful info there — pabrantes, Jul 06 '15 at 15:10
@DmitryGrigoryev: +1 on that comment :) Didn't know about `kill -3 pid` — pabrantes, Jul 06 '15 at 15:12
@pabrantes I am using tomcat. I have a catalina file. Adding to question. — Jason Goss, Jul 06 '15 at 15:13
@JasonGoss: check catalina.out file, you might find relevant information there. — pabrantes, Jul 06 '15 at 15:24
@DmitryGrigoryev I performed the kill -3, but I have no idea where the core dump is. Sorry. — Jason Goss, Jul 06 '15 at 15:24
@JasonGoss: the output of kill -3 it goes into the catalina log... but the catalina log you posted already shows the problem. You're having a OutOfMemoryError, you still have to find what's causing it..but for now if it's possible you can just increase the server's memory using the -Xms -Xmx parameters — pabrantes, Jul 06 '15 at 15:26
@pabrantes wouldn't the `OutOfMemory` terminate the JVM? (I don't know how it's handled in a container). I'm more prone to think the 100% usage is due to that `EOFException` during session loading. — Giulio Franco, Jul 06 '15 at 15:34
@GiulioFranco; The EOFExceptions in tomcat while trying to materialize sessions in the startup is "normal" when you're sessions cannot be properly serialized. Usually that isn't a problem for tomcat, the only thing is that sessions cannot be rested after a Tomcat restart. That could increase the cpu to 100% but after a while it should stop because there are only a finite number of persisted sessions. Regarding the OOM terminating the JVM, I've already had my share of zombie tomcats due to OOM, it's not 100% sure that the JVM will be terminate, sometimes it just hangs there doing nothing. — pabrantes, Jul 06 '15 at 15:39
Ok, where can I find the name of my java class? I tried to use java -Xmx????m pid but it didn't like that. I also used JSTAT to find the current values but it just spent out a bunch of gibberish. — Jason Goss, Jul 06 '15 at 15:40
@JasonGoss: the -Xmx is a parameter that you give to tomcat in the CATALINA_OPTS variable for example. Check this question to know how: http://stackoverflow.com/questions/6897476/tomcat-7-how-to-set-initial-heap-size-correctly. If you have information from jstat post as well, it might give more clues to the problem. — pabrantes, Jul 06 '15 at 15:58

Segmentation Fault in Java

0 Answers0