Can Java Garbage Collector disturb HPA scaling down?

Question

I have a Spring API with a heavy use of memory deployed on kubernetes cluster.

I configured the auto scale (HPA) to look at the memory consumption as a scale criterion, and running a load test everything works well at the time of scale up, however at the time of scale down the memory does not go down and consequently the pods created are not removed. If I run the tests again, new pods will be created, but never removed.

Doing an local analysis using visual VM, I believe the problem is correlated to the GC. Locally the GC works correctly during the test, but at the end of the requests it stops running leaving garbage behind, and it only runs again after a long time. So I believe that this garbage left behind is preventing the HPA from scale down.

Does anyone have any tips on what may be causing this effect or something that I can try?

PS. In the profiler I have no indications of any memory leak, and when I run the GC manually, the garbage left is removed

Here are some additional details:

• Java Version: 11
• Spring Version: 2.3
• Kubernetes Version: 1.17
• Docker Image: openjdk:11-jre-slim
• HPA Requests Memory: 1Gi
• HPA Limits Memory: 2Gi
• HPA Memory Utilization Metrics: 80%
• HPA Min Pods: 2
• HPA Max Pods: 8
• JVM OPS: -Xms256m -Xmx1G

Visual VM After Load test

New Relic Memory Resident After Load Test

Answer 1

There most likely isn't a memory leak.

The JVM requests a memory from the operating system up to the limit set by the -Xmx... command-line option. After each major GC run, the JVM looks at the ratio of heap memory in use to the (current) heap size:

• If the ratio is too close to 1 (i.e. the heap is too full), the JVM requests memory from the OS to make the heap larger. It does this "eagerly".

• If the ration is too close to 0 (i.e. the heap is too large), the JVM may shrink the heap and return some memory to the OS. It does this "reluctantly". Specifically, it may take a number of full GC runs before the JVM decides to release memory.

I think that what you are seeing is the effects of the JVM's heap sizing policy. If the JVM is idle, there won't be enough full GC to trigger the JVM to shrink the heap, and memory won't be given back to the OS.

You could try to encourage the JVM to give memory back by calling System.gc() a few times. But running a full GC is CPU intensive. And if you do manage to get the JVM to shrink the heap, then expanding the heap again (for the next big request) will entail more full GCs.

So my advice would be: don't try that. Use some other criteria to triggering your autoscaling ... if it makes any sense.

---

The other thing to note is that a JVM + application may use a significant amount of non-heap memory; e.g. the executable and shared native libraries, the native (C++) heap, Java thread stack, Java metaspace, and so on. None of that usage is constrained by the -Xmx option.