K8 container orchestration: JVM not releasing memory

Question

We have java based microservices. When these microservices are not serving any requests, each pod of the microservice is using about 270MB. We have kept the memory limit as 512Mi for each microservice.

I did some load testing on some of these microservices. For one microservice, I made 20 API requests per second. For another , 30 requests per second.
A consistent pattern that I have observed is that: the memory utilization goes up as I increase the load and memory does not get released after the load testing (unless of course that container gets terminated).
Once I complete the load testing, I wait for an hour or so. Yet, the memory utilization does not come down. The pod continues occupying same amount of memory as before.
I also got a shell inside the container of such a pod, I can see only the java process and the sh process running. And indeed, I see that memory usage of the pod has increased to about 360MB.

So, a pod which usually takes 270MB (in absence of any load testing), takes 360MB under load test (of 30 API requests per second), and continues to use 360MB long after the load testing has completed.

I do not understand why the pod usage is not coming down.

Could it be:

JVM heap usage problem, probably garbage collection not happening?
Something related to Kubernetes pod requests and limits settings?

Here are my requests and limits for Kubernetes pods:

  resources:
    limits:
      memory: 512Mi
      cpu: '0.5'
    requests:
      memory: 256Mi
      cpu: '0.1'

By the way, I am using Apache JMeter for API load testing. The Docker image being used is : openjdk:8-alpine
I also downloaded the heap dump and opened it in Eclipse Memory analyzer tool.
But, I do not see any application classes (from our code) using a lot of heap.
I see classes related to JDK and Spring framework only, and they do not show any memory leak.

So, the mystery for me is : why does memory usage not come down even 2-3 hours after load testing?
Is it the Kubernetes config of requests and limits OR is it the JVM GC settings tuning?
Is it possible that JVM GC settings are not playing nicely with Kubernetes requests and memory limits?

adding JVM details for reference:

openjdk version "1.8.0_212"
OpenJDK Runtime Environment (IcedTea 3.12.0) (Alpine 8.212.04-r0)
OpenJDK 64-Bit Server VM (build 25.212-b04, mixed mode)

Please add information about the specific JVM you are using. — Thorbjørn Ravn Andersen, Sep 24 '20 at 17:41
@ThorbjørnRavnAndersen, have edited my question and added JVM details — Chetan Yewale, Sep 24 '20 at 17:50
This is a focus area on newer versions of Java. I would suggest you try migrating to Java 11. — Thorbjørn Ravn Andersen, Sep 25 '20 at 12:28
I did more research on this. Indeed, OpenJDK 12 and Eclipse Open J9 VMs resolve above issue. Will write a detailed answer to my own question sometime later. — Chetan Yewale, Oct 17 '20 at 18:09
@ChetanYewale - did you wrote what's causing the issue anywhere? I am facing same issue. — user2585494, Oct 19 '20 at 22:36
@ChetanYewale can you write an answer to this question. I am facing this same issue — sankycse, Mar 08 '21 at 11:48
I believe this [this link](https://stackoverflow.com/questions/61506136/kubernetes-pod-memory-java-gc-logs) has some explanations about this problem. — elmodeer, Jul 12 '21 at 14:22

K8 container orchestration: JVM not releasing memory

0 Answers0