0

I have a web application hosted on tomcat 8.0. Application uses ADFS for single signon. Over the past few weeks we are experiencing a weird issue. After we deploy a fresh war or restart the application, there are no issues and application runs fine for 3 days. On the third day/fourth day (approx), application is not able to respond to any request and needs a restart. On analysing, I found that web application sends a request to Spring rest controller. Rest controller receives the request and then gets the credentials from SAML. With loggers, I found that we get the credentials from SAML properly however, the request does not proceed further in the same method and gets stuck. Tomcat logs does not show any memory issues or out of memory errors. Ultimately we have to restart the application to get things working. After restart, every thing works fine for 3 more days. Running out of ideas.

java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(‌​GenericObjectPool.ja‌​va:1104) - locked <0x000000071d714410> (a org.apache.commons.pool.impl.GenericObjectPool$Latch) at org.apache.commons.dbcp.PoolingDataSource.getConnection(Pool‌​ingDataSource.java:1‌​06) at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicD‌​ataSource.java:1044) at org.hibernate.engine.jdbc.connections.internal.DatasourceCon‌​nectionProviderImpl.‌​getConnection

aksappy
  • 3,400
  • 3
  • 23
  • 49
pawanhb
  • 1
  • 1
    Did you try getting a thread dump of your JVM? There could me a deadlock in your app somewhere that's causing it to hang. A thread dump analysis would help you diagnose that. – Mustafa Aug 07 '17 at 08:23
  • 1
    Is there anything in your application which can happen every 3 days? Or rather, is there a deadlock or a leak that is causing the application to crash? Install NewRelic or JavaMelody or VisualVM to see whether there are any anomalies – aksappy Aug 07 '17 at 09:04
  • @Mustafa thank you for your responses. I shall check for memroy leaks or deadlocks in my application. Can you please provide me some more insight into how to do this or use the tools which you specify. Any link to demonstrate the same? – pawanhb Aug 07 '17 at 09:41
  • @aksappy thank you for your responses. I shall check for memroy leaks or deadlocks in my application. Can you please provide me some more insight into how to do this or use the tools which you specify. Any link to demonstrate the same? – pawanhb Aug 07 '17 at 09:41
  • Search for them in Google. They are simple to be used. There is no link as such other then their home websites. Any questions, ask here again. – aksappy Aug 07 '17 at 09:43
  • @aksappy Same issue resurfaced, this time I checked the heap size and found that heap memory was pretty unused. I further got the thread dump for the tomcat application server. First impression I get after looking at this dump is that there could be some threads hanging for database connection. Please find portion of thread dump log in other comment – pawanhb Aug 17 '17 at 07:30
  • java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1104) - locked <0x000000071d714410> (a org.apache.commons.pool.impl.GenericObjectPool$Latch) at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106) at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044) at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection – pawanhb Aug 17 '17 at 07:36
  • (DatasourceConnectionProviderImpl.java:139) – pawanhb Aug 17 '17 at 07:36
  • @aksappy Please let me know how to handle this – pawanhb Aug 17 '17 at 07:37
  • Are you closing your database connections properly? Also what is the max connections at application and database? Are you running any sql queries which is taking a long time? – aksappy Aug 17 '17 at 07:46
  • https://stackoverflow.com/questions/5714511/deadlock-issue-in-dbcp-deployed-on-tomcat – aksappy Aug 17 '17 at 07:47
  • @aksappy it is set to 300 [ds.setMaxActive(300);] Also, we are closing the connections properly everytime. The javadoc comment says we can provide a negative number for infinite number of connections. Wonder if that may have any sideeffect – pawanhb Aug 17 '17 at 08:44

0 Answers0