We started recently to have a problem on our production server, the application suddenly starts failing all outbound requests (calls to public APIs/local database) for a period of time and then requests start completing successfully again, and then after a while again they start failing. Every-time i restart Tomcat the issue goes away for a while and then re-appear again, and with every restart tomcat the issues start appearing within a less time until i have to restart the whole server to guarantee a lengthy period of stability.
I've looked all around through stackoverflow and google search results and not one of the suggested solutions or investigation paths has led me to solve the problem.
My server setup is : - Windows server 2008 R2 - Tomcat 7.0.59 - JRE 7
I have used "netstat -ano" as suggested in one the answers on stackoverflow.com - i am missing the reference, sorry about that - to check if my application is leaking connections, i looked for CLOSE_WAIT status connections and there appears to be no leak, connections with CLOSE_WAIT status don't live for long they are 0 most of the times, 1 or 2 at maximum.
I have set the preferIPv4Stack java option setx _JAVA_OPTIONS -Djava.net.preferIPv4Stack=true successfully, but it didn't work. I guess its not related to it anyway, because my server wouldn't be able to send any successful requests if it was, right ? In my case, after Tomcat is restarted the server sends requests successfully for x hours, and then starts failing randomly until all requests are failing.
Can someone point me to a certain direction in investigating this issue ? What should i be looking for ?
Any help is appreciated.
EDIT :
Log :
org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://MYURL.COM": Permission denied: connect; nested exception is java.net.SocketException: Permission denied: connect
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:524) ~[spring-web-3.2.8.RELEASE.jar:3.2.8.RELEASE]
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:472) ~[spring-web-3.2.8.RELEASE.jar:3.2.8.RELEASE]
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:420) ~[spring-web-3.2.8.RELEASE.jar:3.2.8.RELEASE]
at model.RestTemplateClient.AbstractRestClient.doRequest(AbstractRestClient.java:50) [OutswitchCore.jar:?]
at clients.KigoRestClient.doRequest(KigoRestClient.java:57) [KigoAdapter.jar:?]
at controller.KigoController.getProperty(KigoController.java:673) [KigoAdapter.jar:?]
at controller.KigoController.getProperty(KigoController.java:1) [KigoAdapter.jar:?]
at KigoAdapter.getProperty(KigoAdapter.java:108) [KigoAdapter.jar:?]
at com.codefish.pcms.channel.processors.PmsActionExecutor.execute(PmsActionExecutor.java:73) [OutswitchCore.jar:?]
at com.codefish.hibernate.HibernateThread.run(HibernateThread.java:27) [FishTankJava.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.7.0_76]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.7.0_76]
at java.lang.Thread.run(Unknown Source) [?:1.7.0_76]
Caused by: java.net.SocketException: Permission denied: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method) ~[?:1.7.0_76]
at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source) ~[?:1.7.0_76]
at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) ~[?:1.7.0_76]
at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) ~[?:1.7.0_76]
at java.net.AbstractPlainSocketImpl.connect(Unknown Source) ~[?:1.7.0_76]
at java.net.PlainSocketImpl.connect(Unknown Source) ~[?:1.7.0_76]
at java.net.SocksSocketImpl.connect(Unknown Source) ~[?:1.7.0_76]
at java.net.Socket.connect(Unknown Source) ~[?:1.7.0_76]
at sun.security.ssl.SSLSocketImpl.connect(Unknown Source) ~[?:1.7.0_76]
at sun.security.ssl.BaseSSLSocketImpl.connect(Unknown Source) ~[?:1.7.0_76]
at sun.net.NetworkClient.doConnect(Unknown Source) ~[?:1.7.0_76]
at sun.net.www.http.HttpClient.openServer(Unknown Source) ~[?:1.7.0_76]
at sun.net.www.http.HttpClient.openServer(Unknown Source) ~[?:1.7.0_76]
at sun.net.www.protocol.https.HttpsClient.<init>(Unknown Source) ~[?:1.7.0_76]
at sun.net.www.protocol.https.HttpsClient.New(Unknown Source) ~[?:1.7.0_76]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(Unknown Source) ~[?:1.7.0_76]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source) ~[?:1.7.0_76]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source) ~[?:1.7.0_76]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(Unknown Source) ~[?:1.7.0_76]
at org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:76) ~[spring-web-3.2.8.RELEASE.jar:3.2.8.RELEASE]
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:46) ~[spring-web-3.2.8.RELEASE.jar:3.2.8.RELEASE]
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:49) ~[spring-web-3.2.8.RELEASE.jar:3.2.8.RELEASE]
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:509)
ANOTHER LOG PART :
ERROR org.quartz.core.ErrorLogger - An error occurred while scanning for the next triggers to fire.
org.quartz.JobPersistenceException: Failed to obtain DB connection from data source 'springNonTxDataSource.quartzJob': com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:168) ~[quartz-2.2.0.jar:?]
at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3774) ~[quartz-2.2.0.jar:?]
at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTriggers(JobStoreSupport.java:2759) ~[quartz-2.2.0.jar:?]
at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:264) [quartz-2.2.0.jar:?]
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.7.0_76]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) ~[?:1.7.0_76]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[?:1.7.0_76]
at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:1.7.0_76]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:377) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1036) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:338) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2232) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2265) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2064) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:790) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:44) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
at sun.reflect.GeneratedConstructorAccessor54.newInstance(Unknown Source) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[?:1.7.0_76]
at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:1.7.0_76]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:377) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:395) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:325) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
at org.springframework.jdbc.datasource.SimpleDriverDataSource.getConnectionFromDriver(SimpleDriverDataSource.java:140) ~[spring-jdbc-3.2.8.RELEASE.jar:3.2.8.RELEASE]
at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnectionFromDriver(AbstractDriverBasedDataSource.java:153) ~[spring-jdbc-3.2.8.RELEASE.jar:3.2.8.RELEASE]
at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnection(AbstractDriverBasedDataSource.java:119) ~[spring-jdbc-3.2.8.RELEASE.jar:3.2.8.RELEASE]
at org.springframework.scheduling.quartz.LocalDataSourceJobStore$2.getConnection(LocalDataSourceJobStore.java:131) ~[spring-context-support-3.2.8.RELEASE.jar:3.2.8.RELEASE]
at org.quartz.utils.DBConnectionManager.getConnection(DBConnectionManager.java:108) ~[quartz-2.2.0.jar:?]
at org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:165) ~[quartz-2.2.0.jar:?]
... 3 more
Caused by: java.net.SocketException: Permission denied: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method) ~[?:1.7.0_76]
at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source) ~[?:1.7.0_76]
at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) ~[?:1.7.0_76]
at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) ~[?:1.7.0_76]
at java.net.AbstractPlainSocketImpl.connect(Unknown Source) ~[?:1.7.0_76]
at java.net.PlainSocketImpl.connect(Unknown Source) ~[?:1.7.0_76]
at java.net.SocksSocketImpl.connect(Unknown Source) ~[?:1.7.0_76]
at java.net.Socket.connect(Unknown Source) ~[?:1.7.0_76]
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:213) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:297) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2232) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2265) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2064) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:790) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:44) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
at sun.reflect.GeneratedConstructorAccessor54.newInstance(Unknown Source) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[?:1.7.0_76]
at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:1.7.0_76]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:377) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:395) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:325) ~[mysql-connector-java-5.1.34-bin.jar:5.1.34]
at org.springframework.jdbc.datasource.SimpleDriverDataSource.getConnectionFromDriver(SimpleDriverDataSource.java:140) ~[spring-jdbc-3.2.8.RELEASE.jar:3.2.8.RELEASE]
at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnectionFromDriver(AbstractDriverBasedDataSource.java:153) ~[spring-jdbc-3.2.8.RELEASE.jar:3.2.8.RELEASE]
at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnection(AbstractDriverBasedDataSource.java:119) ~[spring-jdbc-3.2.8.RELEASE.jar:3.2.8.RELEASE]
at org.springframework.scheduling.quartz.LocalDataSourceJobStore$2.getConnection(LocalDataSourceJobStore.java:131) ~[spring-context-support-3.2.8.RELEASE.jar:3.2.8.RELEASE]
at org.quartz.utils.DBConnectionManager.getConnection(DBConnectionManager.java:108) ~[quartz-2.2.0.jar:?]
at org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:165) ~[quartz-2.2.0.jar:?]
EDIT (NETSTAT -ANO RESULT In pastebin.com link) :
