0

I know it's more complex than "you hit 100% usage and the service goes down". I know that as you approach 100% usage, garbage collection starts running more aggressively, for example, so you see an uptick in GC and CPU usage when you get near that.

But what happens as it keeps going? Will it thrash more and more until I run out of CPU? Start dropping calls? Or just hit 100% eventually and crash entirely?

Is there a good way to find out how close to the edge of that cliff I actuallyam?

StolenKitten
  • 337
  • 1
  • 10

1 Answers1

0

This really depends on your application and which VM you're running on (eg. Oracle JVM, IBM JVM, OpenJDK JVM, Azul Systems JVM, etc). On a Oracle VM you may see one or more of the following before the VM exits:

  • Frequent GC cycles that pushes the VM's CPU usage very high
  • Long running GC cycles with the VM using just 1 CPU at 100% while all other JVM threads are blocked. This can happen when using the CMS Collector and it needs to run a "Stop The World" GC to compact the old generation.
  • The thread or threads that were unable to allocate memory after the GC completes will throw OutOfMemoryError. What happens next depends on the application, ie. If these were the only threads doing anything useful in your application and these threads did not catch the exception or were not restarted after they died then your JVM may continue running but your application will stop responding or stop making progress.
  • If the GC consumes a lot of CPU time and recovers only a small amount of memory after each collection then the JVM may throw a OutOfMemoryError with the message "GC overhead limit exceeded" See Error java.lang.OutOfMemoryError: GC overhead limit exceeded
  • If your application caches data using soft/weak references then you may see increased load on your system resources due to your application not being able to use the cached data and having to regenerate them by computing them or loading them from disk or some other external service.
  • If your application has sockets open then the GC pauses may cause OS buffers to fill forcing the OS to drop packets. Network clients/servers on the other end of your connection may drop the connection because you're application has become a slow consumer or because it missed too many heartbeats or timed out your application.

You can monitor your applications memory usage and what the GC is doing by enabling detailed GC logging (see GC logging flags) and/or through JMX.

You can also access the data JMX is publishing using a JVM API, see MemoryPoolMXBean Documentation

Nam San
  • 1,145
  • 9
  • 13