1

We are having an issue with virtual servers (VMs) running out of native memory. These VMs are running: Linux 7.2(Maipo) Wildfly 9.0.1 Java 1.8.0._151 running with (different JVMs have different heap sizes. They range from 0.5G to 2G) The JVM args are:

-XX:+UseG1GC 
-XX:SurvivorRatio=1 
-XX:NewRatio=2 
-XX:MaxTenuringThreshold=15 
-XX:-UseAdaptiveSizePolicy 
-XX:G1HeapRegionSize=16m 
-XX:MaxMetaspaceSize=256m 
-XX:CompressedClassSpaceSize=64m
-javaagent:/<path to new relic.jar>

After about a month, sometimes longer, the VMs start to use all of their swap space and then eventually the OOM-Killer notices that java is using too much memory and kills one of our JVMs.

The amount of memory being used by the java process is larger than heap + metaSpace + compressed as revealed by using -XX:NativeMemoryTracking=detail

Are there tools that could tell me what is in this native memory(like a heap dump but not for the heap)?

Are there any tools that can map java heap usage to native memory usage (outside the heap) that are not jemalloc? I have used jemalloc to try to achieve this but the graph that is being drawn contains only hex values and not human readable class names so I cant really get anything out of it. Maybe I'm doing something wrong or perhaps I need another tool.

Any suggestions would be greatly appreciated.

Eugene
  • 117,005
  • 15
  • 201
  • 306
Dave Shifrin
  • 11
  • 2
  • Native bytebuffers perhaps? – Holger Dec 20 '17 at 10:37
  • This looks to be more related to JVM issue and there is possibility that we might to have change the java version.Please refer below links from Oracle for Memory leak for non-heap component. https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8164293 https://www.oracle.com/technetwork/java/javase/8-known-issues-2157115.html#A999961 – Makgigrahul Jul 03 '19 at 20:05
  • See this excellent answer for details about jvm memory usage: https://stackoverflow.com/questions/53451103/java-using-much-more-memory-than-heap-size-or-size-correctly-docker-memory-limi/53624438#53624438. This can be useful too: https://jkutner.github.io/2017/04/28/oh-the-places-your-java-memory-goes.html – Juraj Martinka Jul 09 '19 at 06:16
  • How do you know that your issue is native memory usage? Do you have multiple JVMs running inside the same VM? Can it be that you just need more physical RAM? – Juraj Martinka Jul 09 '19 at 06:20

1 Answers1

0

You can use jcmd.

  1. Start application with -XX:NativeMemoryTracking=summary or - XX:NativeMemoryTracking=detail

  2. Use jcmd to monitor the NMT (native memory tracker)

jcmd "pid" VM.native_memory baseline //take the baseline

jcmd "pid" VM.native_memory detail.diff // use based on your need to analyze more on change in native memory from its baseline

Fairoz
  • 1,616
  • 13
  • 16