8

I want to create a nagios watchdog for JVM that looks when the JVM runs out of memory and restarts it.

Currently I was able to setup the JVM be allow JMX but I don't know how to detect OutOfMemory condition and restart it.

 /check_jmx -U service:jmx:rmi:///jndi/rmi://127.0.0.1:1100/jmxrmi -O "java.lang:type=Memory" -A "HeapMemoryUsage" -K used -I HeapMemoryUsage -J used -vvvv
 JMX OK HeapMemoryUsage.used=957414288{committed=2415984640;init=2147483648;max=2863333376;used=957414288}

https://github.com/tcurdt/nagios-check-jmx

sorin
  • 161,544
  • 178
  • 535
  • 806
  • 1
    unfortunately, the standard jdk has _no_ way to detect that a jvm has hit OOM. went through this myself in our product. i ended up installing a logging handler which checks for LogRecords which contain an OOME. works as long as no code swallows the Error without reporting it. – jtahlborn Jul 25 '12 at 16:42
  • i did find some promising API's through the java tooling API, but i concluded that any solution would involve implementing native code in a tooling agent, which was a "no go" for us. – jtahlborn Jul 25 '12 at 16:45

2 Answers2

12

If you're using Java 1.4.2 or later, this option allows you to execute a user defined command when the first OutOfMemeory exception occurs: -XX:OnOutOfMemoryError="<cmd args>;<cmd args>"

That should give you some decent options. e.g. you could kick off a passive check to nagios to tell it that the server is being restarted, and then kick off a shell script to stop/start your errant JVM.

Nicholas
  • 15,916
  • 4
  • 42
  • 66
1

I don't think you are going to be able to detect an out-of-memory condition using JMX. If the JVM is really at the end of it's life, JMX connections will most likely throw OOM exceptions themselves when you try to connect.

We detect high memory conditions instead of OOM. We alarm when our systems' free-memory goes below some water mark for a certain amount of time. We also have threads that run to dump per-serve metric files. Since the thread is already allocated, it can reliably dump system memory information after the JVM runs out.

We log:

// free memory
Runtime.getRuntime().freeMemory()
// current heap usage
Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory()
Gray
  • 115,027
  • 24
  • 293
  • 354
  • 3
    If you wanted to know if you where starting to run out of available memory though, wouldn't you need to use `Runtime.getRuntime().maxMemory()`? Since `totalMemory()` is based on what the JVM has in the JVM at that moment (i.e. it could grow). Essentially available memory would be `Runtime.getRuntime().maxMemory() - Runtime.getRuntime().totalMemory() + Runtime.getRuntime().freeMemory() `. – Jacob Schoen Jul 25 '12 at 17:29