I am working on identifying memory leak, the java process process data uses IO, database connection and runs continuously.
Initially i suspected the memory leak was in heap area, latter with the heap dump and Eclipe MAT tool, i observed GC happens as expected. The process is running in a headless server where jconsole/ visual vm/ remote doesn't work.
OS and java version: Centos 7+ and jdk 1.8.0_131.
Included -XX:+UnlockDiagnosticVMOptions -XX:+PrintNMTStatistics -XX:NativeMemoryTracking=detail to the process, and used jcmd pid VM.native_memory summary.diff the NMT info.
Below is the summary of NMT with ~12 hrs difference. Noticing that all areas except heap is growing over time.
The code uses reflection for main logic, and trying out jemalloc to get better insights.
When using the top command, noticing VIRT and RES grows gradually to an alarming 15+ GB over few days.
I understand Garbage collection doesn't impact Native memory, but how to release the Native memory allocations for a long running java process. Like best practices in this area.
Any inputs helps.
NMT at Date: Fri Jun 19 23:13:44 GMT 2020Total: reserved=2598254KB +1648KB, committed=1364950KB +1648KB
- Java Heap (reserved=1048576KB, committed=1048576KB)
(mmap: reserved=1048576KB, committed=1048576KB)
- Class (reserved=1107415KB, committed=65111KB)
(classes #9265)
(malloc=1495KB #18444 +12)
(mmap: reserved=1105920KB, committed=63616KB)
- Thread (reserved=58150KB -1032KB, committed=58150KB -1032KB)
(thread #57 -1)
(stack: reserved=57460KB -1028KB, committed=57460KB -1028KB)
(malloc=177KB -3KB #292 -5)
(arena=513KB -1 #112 -2)
- Code (reserved=261777KB +10KB, committed=72825KB +10KB)
(malloc=12177KB +10KB #12126 +5)
(mmap: reserved=249600KB, committed=60648KB)
- GC (reserved=92604KB +1KB, committed=92604KB +1KB)
(malloc=20924KB +1KB #22893 +22)
(mmap: reserved=71680KB, committed=71680KB)
- Compiler (reserved=214KB, committed=214KB)
(malloc=83KB #268)
(arena=131KB #3)
- Internal (reserved=8284KB +2602KB, committed=8284KB +2602KB)
(malloc=8252KB +2602KB #16089 +11)
(mmap: reserved=32KB, committed=32KB)
- Symbol (reserved=15601KB, committed=15601KB)
(malloc=13739KB #124634)
(arena=1862KB #1)
- Native Memory Tracking (reserved=3386KB +65KB, committed=3386KB +65KB)
(malloc=273KB +52KB #4277 +790)
(tracking overhead=3113KB +13KB)
- Arena Chunk (reserved=200KB +1KB, committed=200KB +1KB)
(malloc=200KB +1KB)
- Unknown (reserved=2048KB, committed=0KB)
(mmap: reserved=2048KB, committed=0KB)
NMT at Date: Sat Jun 20 10:13:46 GMT 2020
Total: reserved=2627114KB +30507KB, committed=1397306KB +34003KB
- Java Heap (reserved=1048576KB, committed=1048576KB)
(mmap: reserved=1048576KB, committed=1048576KB)
- Class (reserved=1107424KB +9KB, committed=66016KB +905KB)
(classes #9403 +138)
(malloc=1504KB +9KB #18774 +342)
(mmap: reserved=1105920KB, committed=64512KB +896KB)
- Thread (reserved=86059KB +26877KB, committed=86059KB +26877KB)
(thread #84 +26)
(stack: reserved=85216KB +26728KB, committed=85216KB +26728KB)
(malloc=267KB +86KB #427 +130)
(arena=577KB +62 #166 +52)
- Code (reserved=262195KB +428KB, committed=75843KB +3028KB)
(malloc=12595KB +428KB #12339 +218)
(mmap: reserved=249600KB, committed=63248KB +2600KB)
- GC (reserved=92660KB +58KB, committed=92660KB +58KB)
(malloc=20980KB +58KB #23431 +560)
(mmap: reserved=71680KB, committed=71680KB)
- Compiler (reserved=214KB +1KB, committed=214KB +1KB)
(malloc=84KB +1KB #289 +21)
(arena=131KB #3)
- Internal (reserved=8474KB +2792KB, committed=8474KB +2792KB)
(malloc=8442KB +2792KB #16671 +593)
(mmap: reserved=32KB, committed=32KB)
- Symbol (reserved=15820KB +219KB, committed=15820KB +219KB)
(malloc=13958KB +219KB #126277 +1643)
(arena=1862KB #1)
- Native Memory Tracking (reserved=3446KB +125KB, committed=3446KB +125KB)
(malloc=277KB +56KB #4330 +843)
(tracking overhead=3169KB +69KB)
- Arena Chunk (reserved=197KB -2KB, committed=197KB -2KB)
(malloc=197KB -2KB)
- Unknown (reserved=2048KB, committed=0KB)
(mmap: reserved=2048KB, committed=0KB)
