Tomcat stopped working suddenly on production server

Question

Tomcat suddenly stopped. I restarted it and tomcat starts running again but give (code=exited, status=1/failure). After checking the catalalina.log I found:

The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

Also, the error was:

Feb 08, 2018 1:48:31 AM org.apache.catalina.startup.Catalina stopServer
SEVERE: Could not contact localhost:8005. Tomcat may not be running.
Feb 08, 2018 1:48:31 AM org.apache.catalina.startup.Catalina stopServer
SEVERE: Catalina.stop:
java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at java.net.Socket.connect(Socket.java:538)
        at java.net.Socket.<init>(Socket.java:434)
        at java.net.Socket.<init>(Socket.java:211)
        at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:498)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:370)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:457)

The tomcat has started running, but I am afraid it can happen again. What could be the reason as there was no update or installation done recently on server? Also, what could be the possible permanent solution for this?

Update: Only log around that time and before the above logs mentioned is localhost.2018-02-08.log.

Feb 08, 2018 3:26:56 AM org.apache.catalina.core.ApplicationContext log
INFO: No Spring WebApplicationInitializer types detected on classpath
Feb 08, 2018 3:26:57 AM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
Feb 08, 2018 3:28:06 AM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring FrameworkServlet 'ptsw'

You have to look in the logfile for the real reason. The log that you posted is only the error from the restart script, because obviously it couldn't stop Tomcat because it was already stopped. But it's not the reason why Tomcat originally stopped. — dunni, Feb 08 '18 at 10:18
There is no catalina log maintained before this...the correct ti me of server downtime is not known but around that time only these logs are stored at /var/log/tomcat: -rw-r--r--. 1 tomcat tomcat 0 Feb 8 01:48 manager.2018-02-08.log -rw-r--r--. 1 tomcat tomcat 0 Feb 8 01:48 host-manager.2018-02-08.log -rw-r--r--. 1 tomcat tomcat 389 Feb 8 03:28 localhost.2018-02-08.log -rw-r--r--. 1 tomcat tomcat 5.5K Feb 8 03:28 catalina.2018-02-08.log — Aastha Gupta, Feb 08 '18 at 10:49
If you have no logs, then you should check your log configuration and make sure, that logs are properly created. Otherwise you don't have a chance to do troubleshooting. — dunni, Feb 08 '18 at 10:51
I have updated the question and included the log created before it, can you please suggest something that could be done as the tomcat status is still showing (code=exited, status=1/failure) — Aastha Gupta, Feb 08 '18 at 11:19
If the server died with no logs, then it is likely to be the [Linux OOM killer](https://stackoverflow.com/questions/624857/finding-which-process-was-killed-by-linux-oom-killer). It could also have been some fatal JVM error, but there is often some kind of core dump from the JVM when that happens. Either way, the likely cause was something external to Tomcat and not something you can prepare for, in general terms. It is best to make sure you have robust monitoring and alerting to let you know when this happens as quickly as possible. — Mike, Feb 09 '18 at 12:48
and for now tomcat status is running, but still showing (code=exited, status=1/failure) on running 'service tomcat status' cmd. Should I worry about that? Is that could be a warning? — Aastha Gupta, Feb 12 '18 at 05:17

score 1 · Answer 1 · answered Feb 08 '18 at 11:00

The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path

Refer this link for the above mentioned issue.

java.net.ConnectException: Connection refused

This can be due to improper shutdown of Apache tomcat. The server may have been down but the process might have been still running. So you should kill the existing process before starting the server. You can identify and kill the running process using

kill -9 $(ps auxxx|grep Tomcat|awk '{print $2}')

Kill -9 is a force shutdown command to kill any running process.

score -1 · Answer 2 · answered Feb 08 '18 at 10:37

Occurs in cases where the Tomcat port is used by some other process. You can check using the command:

    netstat -tulpn | grep {Tomcat port}

In your case:

    netstat -tulpn | grep 8005

Then, kill the process and restart Tomcat.

Please check /{CATALINA_HOME}/conf/server.xml for the ports your Tomcat uses.

Tomcat stopped working suddenly on production server

2 Answers2