12

What is the difference between Metaspace and Native memory?

Metaspace can be tracked using jconsole, jvisualvm, jstat cmds.
It seems Native Memory can be tracked using jcmd. Link

Is metaspace also Native Memory?

Where do NIO buffers get stored? In the metaspace or native memory?

Hearen
  • 7,420
  • 4
  • 53
  • 63
prem kumar
  • 5,641
  • 3
  • 24
  • 36

1 Answers1

27

Is metaspace also Native Memory?

Yes, Metaspace is part of the Native Memory(process memory) and is limited by host operating system

source http://karunsubramanian.com/websphere/one-important-change-in-memory-management-in-java-8/ You can monitor it using jmap -permstat PID. If your application ends up allocating lot of memory in metaspace then it will effect the entire system not just the JVM.That's why its recommended that you use -XX:MetaspaceSize to explicitly set the max metaspace size for your app.

Java NIO APIs use ByteBuffers as the source and destination of I/O calls, and come in two flavours:

  • Heap Byte Buffer (wrap a byte[] array, allocated in the garbage collected Java heap)
  • Direct Byte Buffer (wrap memory allocated outside the Java heap )

Since only "native" memory can be passed to operating system calls, so it won't be moved by the garbage collector, it means that when you use a heap ByteBuffer for I/O, it is copied into a temporary direct ByteBuffer. The JDK caches one temporary buffer per thread, without any memory limits (i.e. an unbounded cache). As a result, if you call I/O methods with large heap ByteBuffers from multiple threads, your process can use a huge amount of additional native memory.

You can refer to this & this article for more details.

Updated

RSS=OffHeap (mapped files, JVM internal code (.bss segments), thread stacks, direct buffers) + GC internal structure+ Heap + Structures used & allocated by native libraries (e.g IO libraries) + Metaspace + Shared Libraries of the JVM + CodeCache

You can use, on linux pmap -x to analyse the memory map of your JVM process. Furthermore, Jemalloc can help to write a profile to disk after every x GB/ x kB of memory allocation/stack trace by setting appropriate MALLOC_CONF env variables. Once you have a generated file try using
Jeprof for visualisation. E.g. 

 jeprof --show_bytes --pdf `which w` jeprof.heap > sample.pdf

to generate the PDF for call graphs.

Hearen
  • 7,420
  • 4
  • 53
  • 63
sol4me
  • 15,233
  • 5
  • 34
  • 34
  • I am using Tomcat enabled with NIO connectors. RSS value in `ps` unix cmd is not matching with (heap+metaspace). So if native memory is more than metaspace, how to measure it. And if possible could you mention the break up of memory used(RSS) – prem kumar Sep 24 '16 at 11:16
  • 1
    There can be lot of reasons like to many threads, may be problem with glibc version > 2.10, etc. You can use `pmap - x` to get more insight into the memory. – sol4me Sep 24 '16 at 11:44
  • 2
    A large chunk of your answer has been copied from http://www.evanjones.ca/java-bytebuffer-leak.html – TheLostMind Sep 24 '16 at 13:38
  • 1
    @TheLostMind I reformatted my post , for some reason the formatting was not applied properly the first time. It should be fine now – sol4me Sep 24 '16 at 16:06
  • if metaspace is part of native memory and its usage can be tracked, then how to track the overall native memory usage? I am able to track heap, metaspace. But still need to account for memory difference of RSS(resident memory) - Heap -Metaspace . This difference for me varies from 200MB till around 1GB. – prem kumar Sep 25 '16 at 06:05
  • JVM also need some memory for internals .`RSS=OffHeap (mapped files, JVM internal code (.bss segments), thread stacks, direct buffers) + GC internal structure+ Heap + Structures used & allocated by native libraries (e.g IO libraries) + Meatspace + Shared Libraries of the JVM + CodeCache` . Did you tried `pmap -x` & analysed the output? Are you experiencing some memory errors? You can also use `Jemalloc` to write a profile to disk after every x GB/ x kB of memory allocation/stack trace by setting MALLOC_CONF env variables & use jeprof to visualise the generated file – sol4me Sep 25 '16 at 08:53
  • HotSpot (JVM implementation) also has something called Native Memory Tracking which can be used to see exactly how much memory are used for the different internal data structures. https://docs.oracle.com/javase/8/docs/technotes/guides/vm/nmt-8.html – Rickard Oct 05 '16 at 09:02
  • 1
    I knew about metaspace, but until this answer, I'd never heard about "meatspace" Is that where the JVM allocates memory for tube steak? – neildo Dec 06 '17 at 19:00