An application I've been working on has been exhibiting large amounts of what appears to be off-heap memory usage.
I've configured the application to use an Xms and Xmx of 4096m. However, running top on the PID reveals that the process is consuming 9.6% of the machine's memory (64gb machine -> ~ 6.14g application footprint). This seems to continues to grow without bounds the longer the instance is alive. The resident size is approximately 5.9g. Inspecting the process using jvisualvm I see a heap usage of approximately 1.5gb, with healthy GC patterns. I've pulled heap dumps and thread dumps and don't see anything abnormal.
I pulled a pmap for the process. Heap size appears consistent with the application Xms and Xmx paramters. Stack is roughly 130Kb. and I noticed large amounts of anon memory:
00000006c0000000 4195968K 1437248K 1437248K 1437248K 0K rw-p [anon]
00000007c01a0000 1046912K 0K 0K 0K 0K ---p [anon]
00007ff712acd000 230464K 0K 0K 0K 0K ---p [anon]
00007ff7077cb000 148452K 10452K 10452K 10452K 0K rw-p [anon]
00007ff711bdd000 15296K 15224K 15224K 15224K 0K rwxp [anon]
00007ff7109c6000 14924K 14692K 14692K 14692K 0K rw-p [anon]
Some example gc behavior:
Java HotSpot(TM) 64-Bit Server VM (25.40-b25) for linux-amd64 JRE (1.8.0_40-b25), built on Feb 10 2015 21:29:53 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)
Memory: 4k page, physical 65938804k(26225716k free), swap 4200444k(3654604k free)
CommandLine flags: -XX:InitialHeapSize=4294967296 -XX:+ManagementServer -XX:MaxHeapSize=4294967296 -XX:ParallelGCThreads=2 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseParallelGC
2016-06-12T17:24:21.451-0400: 1339.579: [GC (Allocation Failure) [PSYoungGen: 1048576K->33662K(1223168K)] 1048576K->33670K(4019712K), 0.0644541 secs] [Times: user=0.12 sys=0.01, real=0.07 secs]
2016-06-12T18:00:24.301-0400: 3502.429: [GC (Allocation Failure) [PSYoungGen: 1082238K->34800K(1223168K)] 1082246K->34808K(4019712K), 0.0422048 secs] [Times: user=0.08 sys=0.00, real=0.04 secs]
2016-06-12T18:33:27.654-0400: 5485.782: [GC (Allocation Failure) [PSYoungGen: 1083376K->34960K(1223168K)] 1083384K->34968K(4019712K), 0.0382536 secs] [Times: user=0.07 sys=0.00, real=0.04 secs]
2016-06-12T19:06:29.536-0400: 7467.664: [GC (Allocation Failure) [PSYoungGen: 1083536K->34903K(1223168K)] 1083544K->34911K(4019712K), 0.0377560 secs] [Times: user=0.07 sys=0.00, real=0.04 secs]
2016-06-12T19:37:32.785-0400: 9330.912: [GC (Allocation Failure) [PSYoungGen: 1083479K->34512K(1223168K)] 1083487K->34520K(4019712K), 0.0382442 secs] [Times: user=0.07 sys=0.00, real=0.04 secs]
2016-06-12T20:06:35.474-0400: 11073.602: [GC (Allocation Failure) [PSYoungGen: 1083088K->34624K(1362432K)] 1083096K->34632K(4158976K), 0.0445968 secs] [Times: user=0.07 sys=0.01, real=0.04 secs]
2016-06-12T20:45:03.191-0400: 13381.318: [GC (Allocation Failure) [PSYoungGen: 1361216K->6579K(1333248K)] 1361224K->36444K(4129792K), 0.0489775 secs] [Times: user=0.08 sys=0.01, real=0.05 secs]
2016-06-12T21:20:41.637-0400: 15519.764: [GC (Allocation Failure) [PSYoungGen: 1333171K->6080K(1349120K)] 1363036K->36112K(4145664K), 0.0122650 secs] [Times: user=0.01 sys=0.01, real=0.01 secs]
2016-06-12T21:56:05.197-0400: 17643.325: [GC (Allocation Failure) [PSYoungGen: 1313728K->5728K(1313792K)] 1343760K->36072K(4110336K), 0.0107582 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-06-12T22:32:47.264-0400: 19845.391: [GC (Allocation Failure) [PSYoungGen: 1313376K->5728K(1347584K)] 1343720K->36512K(4144128K), 0.0113968 secs] [Times: user=0.03 sys=0.00, real=0.00 secs]
The application has approximately 15 threads. The JVM is jdk-1.8.0_40 on a Sles 11 sp3 machine.
My questions are:
This appears to be an issue where the application is growing considerably off-heap. Is that safe to assume, or am I missing something?
Is there a way to pinpoint where the memory consumption is coming from?
