10

I have tomcat web app on VPS and the tomcat sometimes (about once a month) crashes with the following error in catalina.out:

Java HotSpot(TM) 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGTERM to handler- the VM may need to be forcibly terminated.

Here are some details about my configuration:

  • VPS: debian-5.0-x86_64

  • RAM: 2.5 gb,

  • virtual processors: 8

  • HDD: 60gb hdd - 70% free

  • Tomcat 7.0

  • java -version:

    java version "1.6.0_18"
OpenJDK Runtime Environment (IcedTea6 1.8.13) (6b18-1.8.13-0+squeeze1)
OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode)

  • Java params: -Xms512m -Xmx1024m

I have also Apache-PHP on that server.

I'm monitoring server load with Munin and it shows me that memory and CPU usage is always stable and there were no any increases before crash.

I'm also logging java memory usage via java.lang.Runtime class, and it shows that jvm always uses max200Mb memory and there were no increase before crash. The last log before crash was 40 seconds ago and that time used memory was: 152Mb.

My web app also runs 6-7 threads that collecting data from different public APIs. These threads start when tomcat is starting, and they always running with periodic sleeps.

Can you please tell me why it crashes? How can I find the reason?

valiano
  • 16,433
  • 7
  • 64
  • 79
Arshak
  • 716
  • 2
  • 7
  • 15

4 Answers4

2

Lets unpick this:

Exception java.lang.OutOfMemoryError occurred dispatching signal SIGTERM to handler- the VM may need to be forcibly terminated.

First of all, it looks like something sent the JVM (Tomcat) process a SIGTERM signal. It has to be something external to the JVM that did that. A JVM doesn't send signals to itself1.

So you need to figure out what is doing that. My guess first guess would be the OOM killer ... but the OOM killer uses SIGKILL not SIGTERM. And the JVM never sees the SIGKILL coming!

(You can confirm that it is not the OOM killer by looking in "/var/log/messages" ... or wherever your system logs kernel messages. See How to Configure the Linux Out-of-Memory Killer)

If it is not the OOM killer, then there are a few ways to find the source of a signal:

And once you have the source of the signal, you will have clues as to why it was sent.

The other noteworthy thing is that the OutOfMemoryError occurred while handling the SIGTERM. This strongly suggests (to me) that the root cause is that something has detected that Tomcat is using too much memory, and has sent it a SIGTERM to make it go away (cleanly). I surmise that what happens then is that the JVM goes to the OS to ask for a little bit more memory (to handle the SIGTERM) and the OS says "No", and the JVM throws an OutOfMemoryError. Unfortunately, the JVM is now in a state where it cannot either exit cleanly or recover. Hence it says "the VM may need to be forcibly terminated".

Anyhow. This looks to me like a rather unusual manifestation of a common Java problem. You most likely have a bug in the webapps running in your Tomcat that is leaking memory. If that is the case, the only real solution is to find and fix the bug. (Increasing the heap size ... if possible ... only puts the problem off. It may decrease the interval between crashes, but it is unlikely to prevent them.)

Assuming that you are ready to bite the bullet:

1 - Unless something is doing something nutty in native code ...

Stephen C
  • 698,415
  • 94
  • 811
  • 1,216
0

  • Run top from the shell and see how much memory Linux claims the Java process is taking up.

  • Check the overall memory usage in the machine. It could be that other processes are consuming all the memory, forcing Linux to kill the JVM: A database such as Mysql or Postgress is a likely suspect.

  • Check /var/log/messages in the time periods around the crashes for unusual events.

Adi Dembak
  • 2,433
  • 2
  • 18
  • 26
0

You can edit the service.bat or service.sh file(found in bin folder of deistribution) and put the below two jvm parameters

XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<PATH_TO_WRITE_DUMP>

The PATH_TO_WRITE_DUMP should have necessary permissions, Once you get the heap dump file which will be generated in case of an OutOfMemory issue, analyze it either by hand or by some third party tools. You can load this file in jvisualvm and do some analysis or you can use Memory-Analyzer-Plugin of eclipse , it's very useful and gives some potential memory leak suspects and dominator trees (i.e. which object dominates the heap) This can be a good starting point.

TruckDriver
  • 1,383
  • 13
  • 28
0

increase you tomcat java startup memory, like: set JAVA_OPTS=-server -Xms256m -Xmx512m

Wei Kleeff
  • 188
  • 1
  • 11