0

We have a kubernetes pod going out of memory very frequently, but the heapdump file that gets generated during OOM is only 200 MB while Xmx and Xms are defined at 2400 MB. So it looks like GC is able to clean the objects and bring down the heap when OOM is signalled. If that is the case I am not able to understand why jvm still goes and kills the pod even though heap usage went down after GC (as per the heap dump).

Here are the jvm parameters:

-XX:HeapDumpPath=xxx -Xmx2400m -Xms2400m -XX:+HeapDumpOnOutOfMemoryError -XX:+UseG1GC -XX:CICompilerCount=4 -XX:MaxGCPauseMillis=1000 -XX:+DisableExplicitGC -XX:ParallelGCThreads=4 -XX:OnOutOfMemoryError=kill -9 %p

I expected that the pod does not get killed since GC is probably able to clean the heap objects

user5797176
  • 13
  • 2
  • Not all OOMs are caused by the shortage of free memory, check that code for example: `int[] a = new int[Integer.MAX_VALUE];`. Another possible case is when JVM understands that it is theoretically not possible to store an object in heap, for example for you sizing (`-Xmx2400m`) `new int[Integer.MAX_VALUE - 2]` will trigger OOM as well. – Andrey B. Panfilov Jul 31 '23 at 20:17
  • understand, I don't think we have any such data that can result in this, but I will double check – user5797176 Jul 31 '23 at 23:06

2 Answers2

0

Im guessing you did configure also resoucce limits (memory) on the pod right?

OOM usually happens when jvm (actually any process in the container) allocates more memory then is allowed.

Older but related thread https://stackoverflow.com/a/53278161/4459920

  • if you can, fiddle just with maxrampercentage and resource limits
Josef Ludvíček
  • 36
  • 2
0

Just to doublecheck - what does your kubectl describe pod say?

Is the exit reason is OOMKilled (k8s killed the pod for using more than memory limit) or something else? For me the source of java memory issues in k8s is usually misconfiguration of jvm flags and k8s resouce limits which results to OOMKilled. Usually the problem was somewhere else then too much heap used.

If you're actually hitting out of memory error in JVM heap and your kill options take place, then Im sorry for spam.

Josef Ludvíček
  • 36
  • 2
  • yes, it is due to OOM only, I see these messages in the logs for the pod that got killed: # java.lang.OutOfMemoryError: Java heap space – user5797176 Jul 31 '23 at 23:00