0

We have a spring cloud system running in K8S enviroment, and I do stress testing on one of service for finding whethere memory leak exist in the service.

There are 3 tools we use to monitor memory change in the server's process:
1 Prometheus and Grafana
2 "top" (top -p )
3 java NMT tool (jcmd VM.native_memory summary)

During the testing, found that there is a confused report when RSS memory changed apparently:
From report of Grafana, the RSS increased 1% about 10MB, that coninstance with the "top" output basicly.
BUT from the NMT output, there seems total committed memory decrease 1MB instead of increasing with RSS.

For example:

"top" output is:

2022/10/19 16:00
RSS     VSZ
**855276**  7108367

2022/10/19 16:05
RSS     VSZ
**862876**  7108367

NMT output is：
2022/10/19 16:00
Total: reserved=2363158KB, committed=**869232KB**

2022/10/19 16:05
Total: reserved=2362056KB, committed=**868365KB**

I have already known that committed memory in java might not equals to RSS in Linux. But the confused point is when committed memory decreasing in JVM, the RSS is incresing conversely. Where is the increasing memory got from "top", because NMT do not give the answer, seems the incereasing memory not produced by JVM?

The other messages of the system:
spring boot 2.1.3.RELEASE
spring cloud 2.1.3.RELEASE
JDK 1.8.0_202
Linux 4.19
GCC 7.3.0
GLibc 2.17

Alvin
  • 133
  • 1
  • 1
  • 7
  • 1
    NMT can report [less](https://stackoverflow.com/questions/44734359/jvm-is-using-more-memory-than-native-memory-tracking-says-how-do-i-locate-where) or [more](https://stackoverflow.com/questions/31173374/why-does-a-jvm-report-more-committed-memory-than-the-linux-process-resident-set) memory than is actually used. For one memory area NMT numbers can be bigger, while smaller for another area. When both areas change in size, the overall difference can go in either direction. – apangin Oct 19 '22 at 16:14
  • ok, I found that RSS not always have the same change direction with the change of commited memory got from NMT, during the stress testing, they keep the same direction when the concurrence request increase to some extent. – Alvin Oct 21 '22 at 07:26

0 Answers0