0

I'm using Tomcat 7 to deploy my app.

Tomcat goes down randomly and requires me to restart the server time and again to keep it up and running.

Sometimes when i restart ,i check the logs to see waiting for instances to be deallocated.

Other times when i restart ,i get the ERROR 1053.

I tried deleting my temp/work folders.I checked my database connections through hibernate ,i made sure that i have all the required jars in lib under tomcat.

The way i deploy is copy my war file and its respective Applicationfolder into Webapps under Tomcat.Please direct me as to what needs to be done to keep the server stable.

Another important note is that the exception in the logs below,changes from Nullpointer to Illegal state Exception

Any help to identify the root cause will be truly appreciated as this has been an impending and critical issue that has been haunting,inspite of trying different solutions.

My Logs are as follows

    INFO: Waiting for 3 instance(s) to be deallocated
    Jan 27, 2014 2:17:27 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
    SEVERE: The web application [/cep4] registered the JDBC driver       [oracle.jdbc.OracleDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Jan 27, 2014 2:17:27 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/cep4] appears to have started a thread named [Timer-0] but has failed to stop it. This is very likely to create a memory leak.
Jan 27, 2014 2:17:27 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/cep4] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation.
Jan 27, 2014 2:17:27 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/cep4] created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [Locale context]) and a value of type [org.springframework.web.servlet.DispatcherServlet$1] (value [en_US]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 27, 2014 2:17:27 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/cep4] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1655d52d]) and a value of type [org.springframework.security.core.context.SecurityContextImpl] (value [org.springframework.security.core.context.SecurityContextImpl@388100c: Authentication: org.springframework.security.cas.authentication.CasAuthenticationToken@388100c: Principal: com.java.service.MyUserDetails@9154e84; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@ffff4c9c: RemoteIpAddress: 128.32.111.117; SessionId: CF331A8C27CEA68FAC797ED7E0E654B5; Granted Authorities: com.java.service.MyUserDetails$1@1aca2c25 Assertion: org.jasig.cas.client.validation.AssertionImpl@1a6d280b Credentials (Service/Proxy Ticket): ST-89206-bOamLKnOoLTLDJoIxYqs-cas-p3]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 27, 2014 2:17:27 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/cep4] created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [Request attributes]) and a value of type [org.springframework.web.context.request.ServletRequestAttributes] (value [SecurityContextHolderAwareRequestWrapper[ org.springframework.security.web.savedrequest.SavedRequestAwareWrapper@5d1e88cf]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 27, 2014 2:17:27 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-80"]
Jan 27, 2014 2:17:27 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-8443"]
Jan 27, 2014 2:17:27 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-8009"]
Jan 27, 2014 2:17:27 PM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already.  Could not load org.hibernate.cfg.ImprovedNamingStrategy.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1600)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
    at org.springframework.util.ClassUtils.forName(ClassUtils.java:258)
    at org.springframework.beans.factory.support.AbstractBeanDefinition.resolveBeanClass(AbstractBeanDefinition.java:417)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doResolveBeanClass(AbstractBeanFactory.java:1283)
    at org.springframework.beans.factory.support.AbstractBeanFactory.resolveBeanClass(AbstractBeanFactory.java:1254)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:433)
    at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:270)
    at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:120)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1360)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1118)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
    at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

Jan 27, 2014 2:17:29 PM org.apache.catalina.startup.SetAllPropertiesRule begin
WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'maxSpareThreads' to '75' did not find a matching property.
Jan 27, 2014 2:17:29 PM org.apache.tomcat.util.digester.SetPropertiesRule begin
WARNING: [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'source' to 'org.eclipse.jst.jee.server:cep4' did not find a matching property.
Jan 27, 2014 2:17:29 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-80"]
Jan 27, 2014 2:17:29 PM org.apache.coyote.AbstractProtocol init
Abby
  • 11
  • 4
  • Similar questions: http://stackoverflow.com/questions/5292349/is-this-very-likely-to-create-a-memory-leak-in-tomcat http://stackoverflow.com/questions/5239146/tomcat-6-going-down-after-reaching-its-maximum-number-of-threads http://www.coderanch.com/t/534444/Tomcat/Tomcat-causing-application-hang – Evgeni Dimitrov Jan 29 '14 at 20:33
  • Those logs look like the end of a standard shutdown. What might be interesting is what happens before that. You need to show normal operation followed by the start of the shutdown sequence. – Mark Thomas Jan 29 '14 at 23:25
  • @MarkThomas:Here are the Catalina.out logs under Normal operation. – Abby Apr 01 '14 at 23:03
  • Can you guys please give me some directions on this.Need some serious help on this.Its still unresolved – Abby Apr 01 '14 at 23:07
  • @Evgeni Couldnt get much help from the links.Please respond with specific solutions to help fix ma issue – Abby Apr 03 '14 at 19:54
  • Assuming your situation, this is what I'd do to find the root cause. First increase the logging level of all my app and library specific settings to at least info, better debug if I could afford that on production and trace the application logs as well to see if I can find any pointers. If nothing is found, I'd have assumed the application is calling System.exit() somewhere which may be from the bug in my own app or from a third party library. I'd use security manager as described [here](http://stackoverflow.com/a/5401402/2641683) to rule out that possibility. – Avnish Apr 03 '14 at 21:13

1 Answers1

0

Really my colleagues and I faced with a similar issue with nearly the same error logs when using Apache Tomcat 7 as application server. Finally we could solve the problem using helps of contents at Stack Overflow and Apache Tomcat websites. Here I would like to share our experiences in a step by step form of debugging:

  1. Try to find out crashing time due to server time.
  2. Goto log folder of Tomcat and check errors just before the crashing time to the end.
  3. Try to find out the reason of each error and try to solve the problems which caused the errors.
  4. Start to monitor Tomcat behaviour using available tools. We used jConsole, but do not forget to link jConsole to Tomcat.
  5. Try to control important parameters such as changes in Heap Memory and Threads especially in peak time of using your system. Maybe you have to increase specified or default values of Heap Memory or Threads.
    P.S.: Do not panic on PS Survivor space! That's full most of time. ;-)

Final Word

Finally in our case by monitoring Tomcat with jConsole, we found out the problem was insufficient Heap Memory in peak time. The solution was increasing the amount of allocated heap memory of Tomcat.

Community
  • 1
  • 1
Kiamehr
  • 1
  • 1