For some time we've had a problem in our production environment about one of our JVM pods having seemingly random spikes of memory usage. This sometimes leads to the JVM process being OOMKilled by the Kernel, as it would otherwise go over the memory limit set through kubernetes. This also leads to no OOM Error being thrown, and therefor, no JVM Heapdump being made.
Initially, all we had was -Xmx900m to limit the maximum heap. The pod itself was limited to 2000Mi, which is corresponds to 2097.15MB. So, in theory this shouldn't be a problem, right? Well it still was, as different parts like direct memory etc. are not limited by that, and lead to over usage of memory.
So, I've been doing some reading and added some flags, enabled some other things, with the hopes of making the JVM kill itself before the kernel, so it could do a heapdump for us to analyse.
At the same time we added a new node to be able to just increase the pod limits (hey, maybe it's just more users? Would be okay...)
So after some changes, this is our new setup. The pod is now limited to 2500Mi or 2621.44MB, and the JVM has the following flags applied:
-XX:MaxDirectMemorySize=900m
-Xmx1200m
-XX:NativeMemoryTracking=detail
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/pod-dump.bin
-Xlog:gc*:file=/tmp/gc.log:time,level,tags:filecount=4,filesize=50M
Long story short: It's still dying through Kernel induced OOMKilled. Therefor, no Heapdump is being created.
Using -XX:NativeMemoryTracking=detail, I can get further details on the JVM, and the first thing I realised is the first line:
Total: reserved=2448034KB, committed=750586KB
From what I understand is that reserved resembles the total memory space the JVM thinks it can use, but isn't actually committed to any real memory addresses(This article goes above and beyond to explain this stuff https://blog.arkey.fr/2020/11/30/off-heap-reconnaissance/). If we convert it, we get 2390.65MB, which is above our pod limits! So, in theory this seems to be the reason why the kernel still has to OOMKill the process.
The next step was to check the JVM flags. We do set some, but others are set automatically, so maybe we can find something there? (For clarity sake I only show the flags which seem interesting:
-XX:G1HeapRegionSize=1048576
-XX:MarkStackSize=4194304
-XX:MaxDirectMemorySize=943718400
-XX:MaxHeapSize=1258291200
-XX:MaxMetaspaceSize=536870912
-XX:MaxNewSize=754974720
-XX:SoftMaxHeapSize=1258291200
So as we set Xmx1200m, MaxHeapSize and SoftMaxHeapSize are the same, namely 1200 MB. The next one is MaxDirectMemorySize, which is also correct and evaluates to 700MB. Now the question is, which of these other flags contribute to the total JVM memory usage, or the Total reserved` amount?
My intuition tells me that to calculate the possible maximum amount of memory the JVM could use before going OOM (itself, not kernel, also ignoring that Heap can go OOM without the complete JVM going OOM) I can do the sum of some of these flags:
MarkStackSize + MaxDirectMemorySize + MaxHeapSize + MaxMetaspaceSize + MaxNewSize, which results in a total of 3336MB, which is way above the pod limits!
What might also be interesting is that MaxRAM changed after the pod limits were adapted, and is now MaxRAM=2621440000 or 2500MB, which perfectly reflects the pod limits. So here is another question: If the JVM can correctly figure out the possible MaxRAM, why does it not set flags that aren't manually set to accommodate for that?
Quite possibly I'm absolutely in the wrong here and misunderstood quite a bit about JVM + Memory usage, but from what I understood we either lack some flags, or are forced to manually set everything to make sure it does not go over it.
Additional information:
We are using temurin_17 image, and use Spring + GRPC-java for our application (which uses netty, which is where most native memory usage is coming from), and we are using G1GC as garbage collector
