Why would a Native Memory Tracking report of the JVM report such high memory usage of the arena area within threads

Question

I am having an issue where I have a java application that occasionally gets killed by the oom killer in linux after having run out of memory. I have monitored the heap and it does not grow. In fact, it never even reaches the maximum allowed via Xmx which was 768MB.

Therefore I enabled native memory tracking via -XX:NativeMemoryTracking=summary. I then ran my application, took a baseline and then waited till the point where it started to eat memory, at which point I used jcmd <pid> VM.native_memory detail.diff to produce the following report.

Native Memory Tracking:

Total: reserved=16894180KB +14703341KB, committed=15330936KB +14896985KB

-                 Java Heap (reserved=786432KB, committed=450048KB +129536KB)
                            (mmap: reserved=786432KB, committed=450048KB +129536KB)

-                     Class (reserved=1185329KB +110708KB, committed=156657KB +128180KB)
                            (classes #23288 +19829)
                            (malloc=11825KB +2164KB #27117 +23990)
                            (mmap: reserved=1173504KB +108544KB, committed=144832KB +126016KB)

-                    Thread (reserved=16825159KB +16803246KB, committed=16825159KB +16803246KB)
                            (thread #47 +24)
                            (stack: reserved=47288KB +25472KB, committed=47288KB +25472KB)
                            (malloc=153KB +82KB #240 +120)
                            (arena=16777717KB +16777692 #92 +48)

-                      Code (reserved=260509KB +9756KB, committed=63625KB +56100KB)
                            (malloc=10909KB +9756KB #14012 +11850)
                            (mmap: reserved=249600KB, committed=52716KB +46344KB)

-                        GC (reserved=39135KB +15KB, committed=37831KB +307KB)
                            (malloc=10399KB +15KB #629 +478)
                            (mmap: reserved=28736KB, committed=27432KB +292KB)

-                  Compiler (reserved=890284KB +890139KB, committed=890284KB +890139KB)
                            (malloc=55KB +41KB #334 +258)
                            (arena=890229KB +890098 #8 +5)

-                  Internal (reserved=13299KB +3377KB, committed=13299KB +3377KB)
                            (malloc=13267KB +3377KB #26649 +21418)
                            (mmap: reserved=32KB, committed=32KB)

-                    Symbol (reserved=32729KB +27117KB, committed=32729KB +27117KB)
                            (malloc=28565KB +24400KB #285695 +250801)
                            (arena=4163KB +2717 #1)

-    Native Memory Tracking (reserved=13011KB +12136KB, committed=13011KB +12136KB)
                            (malloc=367KB +241KB #5803 +3803)
                            (tracking overhead=12644KB +11895KB)

-               Arena Chunk (reserved=18014398506330278KB -3153154KB, committed=18014398506330278KB -3153154KB)
                            (malloc=18014398506330278KB -3153154KB)

From the report I can see that the amount of memory used has increased significantly committed=15330936KB +14896985KB, but this is not in the heap.

It appears that this has occurred in the memory used by the threads. The stack itself appears normal in that each thread stack should be 1024KB so 47288KB would be reasonable for 47 threads.

Thread (reserved=16825159KB +16803246KB, committed=16825159KB +16803246KB)
                            (thread #47 +24)
                            (stack: reserved=47288KB +25472KB, committed=47288KB +25472KB)
                            (malloc=153KB +82KB #240 +120)
                            (arena=16777717KB +16777692 #92 +48) 

In particular, it reports the arena as having increased by +16777692. What could have caused this and why? Or is this not an issue? From other reports I have seen online (including other stackoverflow questions) the arena memory usage has never been as high as this.

Also, the Arena Chunk itself in terms of the size reserved is massive. Is this normal and if not what could cause this abnormality?

The following open JDK bug JDK-8164293 appears to report a memory leak in the JVM itself, so could I be experiencing this bug?

Answer 1

We had similar problem which is open and still under investigation. You can see below link private bytes increase for a javaw process in java 8

But still I would share my analysis here.

1) This arena increase is not mapped to heap. So it is clearly not the java code problem. Looking at this, it looks like JNI code issue or the native leak internally by java api. So track your JNI code and see if you are closing all your malloc calls and using deleteLocalRef and deleteGlobalRef for all your native java objects.

2) Check if you get this problem only after java update. And if yes,then check which jni.h and related files you are using in your code. Is it from java8.

3) If still no luck, then use tools like Jemalloc which can tell you the method name which is causing the max leak. It seems this is the best tool for linux.

Check out this link : https://gdstechnology.blog.gov.uk/2015/12/11/using-jemalloc-to-get-to-the-bottom-of-a-memory-leak/

4) you can use tool like VADump to see which dll consumes most memory.

Our investigation is still under progress, I will update you once if get any results. Please you also update here regarding your progress.

Answer 2

Just an observation - the absurd 'arena chunk' works out at 2^63/2^64. It might be an overflow error, but equally it might just be some deep mechanism that has been told it has the whole of the address space to play in.