1

I'm having a weird issue with a java process which is consuming a lot of resources on a linux VM. The output of top for the process is the below :

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
1182 blabla     20   0 25.5g  21g  45m S 382.9 33.9 139840:17 java

So this shows that the process is actually consuming 21G of physical memory ?

When checking the process in more detail I can see it was started with -Xmx4G . Is there any logical explanation for the 21G shown on "top" ? (Ignore the first -Xmx from ps output below) 

blabla@dtraflocorh391:~ $ ps aux | grep 1182
blabla     1182  249 33.8 26716716 22363868 ?   Sl   Feb13 139847:01 java -Xms32M -Xmx128M -showversion -Djavax.xml.stream.XMLEventFactory=com.ctc.wstx.stax.WstxEventFactory -Djavax.xml.stream.XMLInputFactory=com.ctc.wstx.stax.WstxInputFactory -Djavax.xml.stream.XMLOutputFactory=com.ctc.wstx.stax.WstxOutputFactory -Xms256M -Xmx4G -verbose:gc -XX:+PrintGCTimeStamps ... ... ...

... ...

nullPointer
  • 4,419
  • 1
  • 15
  • 27
  • I see `-Xmx128M`, not `-Xmx4G`. I suspect that there is some confusion going on here. – Mad Physicist Mar 24 '17 at 15:36
  • OK, now I see the `-Xmx4G1`. What happens when you have multiple copies of that flag? – Mad Physicist Mar 24 '17 at 15:37
  • More importantly, how do you know you don't have even more copies of this flag? maybe auxwww or somesuch – pvg Mar 24 '17 at 15:38
  • @MadPhysicist it appears that it takes the last one (e.g. `java -Xmx2g -Xmx1g ...`) produces: (`-XX:CICompilerCount=4 -XX:InitialHeapSize=268435456 -XX:MaxHeapSize=**1073741824**`) with `jcmd`. – Kedar Mhaswade Mar 24 '17 at 15:49
  • 2
    -Xmx controls the maximum heap size that the virtual machine will allocate. Top shows the memory used by the process, of which the heap is only a part. It may be that some native module has gone haywire. To see what is happening inside the virtual machine memory-wise you'd have to connect to it with a profiler. – Gimby Mar 24 '17 at 15:49
  • @FelicePollano It's 382%. It is normal on a multicore machine. It means that at the time the top output was seen, the `java` process was using `~3.8` cores. – Kedar Mhaswade Mar 24 '17 at 15:57
  • @KedarMhaswade what about writing 98% that is correct by a numeric point of view ;) – Felice Pollano Mar 24 '17 at 16:11
  • 1
    Look at the off-heap storage of your app like StephenC suggests. Your heap storage can't exceed 4g since you specified -Xmx4g (get rid of the duplication, please). This does hint at memory-mapped files, many number of stacks (what is -Xss?) etc. – Kedar Mhaswade Mar 24 '17 at 16:49
  • @KedarMhaswade the invocation doesn't even have the full arglist so for all we know there's yet another, overriding -Xmx in there. Pretty much could be anything (although it's probably no 18 gigs of stacks). – pvg Mar 24 '17 at 17:32
  • Possible duplicate of [Java using more memory than the allocated memory](http://stackoverflow.com/questions/6527131/java-using-more-memory-than-the-allocated-memory) – James Jithin Mar 24 '17 at 17:54

2 Answers2

2

So this shows that the process is actually consuming 21G of physical memory ?

Yes. The "top" command is reporting the processes physical memory usage as provided by the operating system. There is no reason to distrust it.

Is there any logical explanation for the 21G shown on "top" ?

It could be lots of off-heap memory allocation, or a large memory-mapped file, or ... a huge number of thread stacks.

Or potentially other things.

Note that the VIRT figure represents the total virtual memory that has been committed to the process, one way or another. The 'RES' figure represents total physical memory usage to hold the subset of the 'VIRT' figure that is currently "paged in".

So, the VIRT figure is a better measure of the memory resources the Java process has asked for. The fact that VIRT and RES are so different is evidence that points to your process causing "thrashing".

See also:

Community
  • 1
  • 1
Stephen C
  • 698,415
  • 94
  • 811
  • 1,216
2

-Xmx controls the maximum heap size - essentially where stuff created with the new keyword is stored.

Java has other pools of memory - for the execution stack, for caching JIT compiled code, for storing loaded classes and dynamically created classes, for memory-mapped files, the heap space used by native code (JNI, or just Hotspot itself): see How is the java memory pool divided?

JConsole is one tool you can use to see what your Java program is using memory for. Once JConsole has told you which area is big/full, you can start to reason about why it's got like that.

For example, I once had a program that loaded a new Jackson object mapper for every parse -- and this version of Jackson had a bug whereby it left dynamic classes in non-heap memory which never got GCd. Your problem might be this, or it might be something completely different. JConsole will tell.

Community
  • 1
  • 1
slim
  • 40,215
  • 13
  • 94
  • 127
  • There are several unknowns here. It's not clear from OP's question that it is indeed a 4-core machine. The load factor is also not provided. 382% for a 8-core machine means half the number of cores are available at the time of this `top` reading. So, your answer does not answer why OP is seeing `21g` in RSS whereas `-Xmx` is `4g` (as apparently, the OP insists). – Kedar Mhaswade Mar 24 '17 at 16:38
  • The machine is an 8 core VM. Also in the GC logs of the process it was doing Full GC at 4G, which makes sense as per the -Xmx4G flag : 3018629.868: [Full GC 47213K->47213K(4193792K), 0.0891930 secs] – nullPointer Mar 24 '17 at 16:45
  • Removed the bit about CPU - we can't conclude very much from 382% on 8 cores. – slim Mar 24 '17 at 17:05
  • @KedarMhaswade the whole point of the answer is that we can't explain 21GB with the info available. I have explained that Java has lots of memory pools that are *not* limited by -Xmx. JConsole will tell all. – slim Mar 24 '17 at 17:08