How does -XX:OnOutOfMemoryError on OOM?

Question

How does the -XX:OnOutOfMemoryError flag work in Java? When an OOM occurs that means there's no space to allocate for new objects? So if I run a script on OOM, how are the objects created by the script handled?

Even -XX:+HeapDumpOnOutOfMemoryError, how does the system have enough memory to generate a heap dump and save it, even though the system just OOM'ed?

I tried looking online for documentation but haven't really found anything. One guess I have is that if a system has 100MB of space, it actually reserves like 1MB for the memory taken by the -XX:OnOutOfMemoryError flag, so even though you have 100MB of memory, you would OOM on 99MB.

Answer 1

Out of memory means the java heap is full and garbage collecting it doesn't release a significant amount of that.

The java heap is its own, more or less constant sized batch of memory. It's, at its largest, equal to whatever you passed with the -Xmx option (e.g. if you run java -Xmx1g it's 1 gigabyte worth of memory). If you don't use the -Xmx option it depends on your platform and the mode the JVM is in, but, oversimplifying quite a bit, it's still not quite 'as much as your machine can handle'.

More generally most OS platforms (notably, not iOS) use swap: You have huge swaths of apparent memory because the system will shift pages to disk and back as needed. A JVM doesn't take that much memory (JVMs would perform very very badly) - instead of getting stuck swapping memory pages between disk and main memory, it just aborts with an OutOfMemory.

The point is: The system is just peachy fine and can run whatever you want.

The -XX:OnOutOfMemoryError runs a system command. Not java code. Even if you use it to run e.g. java -jar myjar.jar OutOfMemoryTrigger, that starts an entirely new JVM completely separate from the OOMing JVM. Your host system might not be able to do that (for example, if you have no swap and you gave every inch of RAM to the OOMing JVM). In that case, on some OSes your OOMing JVM gets hardkilled by the OS first which is a bit awkward - it's also possible the JVM will wipe out its heap (it's dying/dead - there is no point to it now) and then run the OnOOM command.

OnOutOfMemoryError's purpose is to trigger e.g. a watchdog - something that logs a problem or sends a notification to sysadmins that the system has gone down and needs urgent attention.

The heap dump thing is easy - the code that does that isn't java code (it's the JVM core code), that code is already (and always) in memory (it's basically java.exe itself, not any code from jars/jmods/class files), and only uses memory the system always reserves for core ops. Using this option does not mean that your JVM has a few megs less vs. not using this option.

Answer 2

Adding to rzwitserloot's answer ...

It's important to understand that there are different types of memory being used. There's the OS memory where everything lives, but then the JVM has different areas of memory, and the Heap is just one part of it.

Some of these links explain that better than I could:

• https://www.tutorialspoint.com/Java-JVM-Memory-Types
• https://www.baeldung.com/java-jvm-memory-types
• https://spring.io/blog/2019/03/11/memory-footprint-of-the-jvm

We mostly think about the Heap because that's where the Java objects are created. Garbage Collection is mostly about recovering space there after they are no longer in use, so we can create more objects later. The Heap is likely the largest area of memory in use by the JVM, however there are other areas that get used for other purposes as described in those links.

Since the Heap is just a portion of the memory used by the JVM, and all of that is just a portion of the OS/system memory, there will likely be enough system memory to run your command in the -XX:OnOutOfMemoryError, because (as rzwitserloot pointed out) what runs is actually a different process. Hopefully it's not a huge program; in my work experience, we just killed the java process (e.g. kill -9 %p) after dumping heap.

It's also interesting that if you get an OutOfMemoryError, it does not necessarily mean that there is absolutely zero heap left. It could mean that a threshold was exceeded where the JVM has decided that it's not making enough progress... that it's having to do more Garbage Collection than is reasonable to keep continuing, and that it might as well give up. See Error java.lang.OutOfMemoryError: GC overhead limit exceeded for more discussion on that.