4

I have 23 Java processes running on one machine with 32GB. No process specifies JVM memory params such as Xmx. java -XX:+PrintFlagsFinal -version | grep MaxHeapSize reports that max default heap size is 8GB as expected.

Every process runs embedded Tomcat (Spring Boot apps (most at v 2.3.4)) except one is a standalone tomcat 9 instance running three WARs. These apps have low usage (usually one user and 10 minutes use a day). They are not memory or CPU intensive. One of them is Spring Boot admin and another is Spring Cloud's Eureka service registry. For these two, I have only a main method that simply bootstraps the Spring Boot application.

Yet, RES memory as shown in top for every process keeps gradually increasing. For example, Spring Boot service registry has increased from 1.1GB to 1.5GB in the last 12 hours. All processes show a similar small increase in RES but the total increase has reduced free memory by 2 GB in that same 12 hour period. This was the same in the previous 12 hours (and so on)until current free memory is now only 4.7GB.

My concern is that I continue to see this trend (even without app usage). Memory is never freed from the apps so total free memory continues to decrease. Is this normal since perhaps each JVM sees that memory is still available in the OS and that 8GB heap space is available to it? Will the JVMs stop taking memory at some point say once an OS free memory threshold is reached? Or will it continue until all free memory is used?

Update

The heap used for most apps is under 200MB but the heap size is 1.5 - 2.8GB. Heap max is 8GB.

James
  • 2,876
  • 18
  • 72
  • 116
  • it seems that your problem is that a GC (I assume you use `G1GC`) does not release memory back to the OS... in such a case you might benefit from [this](https://openjdk.java.net/jeps/346), but only since java-12, or switch to a GC that can do that. may be more details can be found [here](https://stackoverflow.com/questions/61506136/kubernetes-pod-memory-java-gc-logs/61512521#61512521) - it talks about docker, but touches the general idea of releasing memory – Eugene Oct 29 '20 at 21:42

2 Answers2

2

Resident memory as reported by the OS doesn't tell you what component is consuming it. You'll have to gather additional data to figure out which part of the process is growing

You'll have to track

  • java heap and metaspace use - you can monitor this with JMC, gc logging and many other java monitoring tools
  • jvm off-heap use - NMT
  • direct byte buffer use - MX beans, also available via JMC
  • use by mapped files - pmap -x <pid>
  • use by native libraries e.g. used via JNI - difficult to monitor
the8472
  • 40,999
  • 5
  • 70
  • 122
0

I also faced this situation and after a long time of research I found the solution here. Basically, for my case, it was just a matter of setting xms and xmx parameters on the jar invocation, forcing the GC to act constantly.

Digao
  • 520
  • 8
  • 22