21

I have read through this Process Memory Vs Heap -- JVM and i have the same problem.

The jvm process memory usage keeps increasing and never shrinks.I checked by doing a top on the linux server. The application is scheduling jobs to a cluster ( Using Quartz + Sun Java DRMAA API )

The java heap space is staying within the limits during the application life cycle but the jvm process is showing a steady climb in memory usage and never coming down.

Is this a memory leak ? If so , why is heap space being within the limits. Can someone explain this.

UPDATE: I have -Xmx1600m -Xms1600m when i track through jconsole i can see the heap space well within this limit aroung 450m but the top command shows the process is using more than 900m.

trincot
  • 317,000
  • 35
  • 244
  • 286
Kathir
  • 6,136
  • 8
  • 36
  • 67
  • 1
    Please provide more detail... how much memory is being used, what's the largest it's ever gotten and after how long, what -X and related flags you are using, etc. – jkraybill Jul 22 '11 at 04:59
  • You can play around with `-XX:MaxHeapFreeRatio` and see if that solves your problem. – trutheality Jul 22 '11 at 05:34
  • I am looking into a similar problem where `top` shows virt size as more than 1GB and res size less than 50 MB. The virt/res ratio is thus really high. Heap dumps with jmap or the `-XVerboseGC`switch show that the heap size is very small (less than 50 MB). How can I find out what is taking the non-heap memory? Thers is some socket IO involved usnig NIO, but I need evidence and need to track any memory leaks/ find roots. – CruiZen Nov 01 '11 at 14:20

4 Answers4

17

The total virtual memory used is the sum of the maximum heap + thread stacks + direct memory + perm gen + share libraries. This never shrinks.

The actual main memory used depends on how much of the virtual memory has been occupied. Shared libraries are shared so having multiple JVMs won't result in this memory doubling etc.

The JVM never releases memory to the OS, however if main memory is not used for a long time it can be swapped out if this is need.

Peter Lawrey
  • 525,659
  • 79
  • 751
  • 1,130
  • So if the application is using the allocated 1600M, why would the kernal kill the process with oom killer. I am using centos and when my webapp runs for few hours, the kernel kills the process saying oom. The jconsole for the same java process shows a far less value as the usage. I am totally confused. – Kathir Jul 22 '11 at 21:56
  • The OS doesn't trigger an OOME and an OOME doesn't kill a process though it may cause one thread to die and the application to not work. You need to look at the OOME as may be another area is filling up like the PermGen, direct memory or too many threads. It could also be that you are trying to allocate a large array which will just fail (you won't see anything in jConsole as it only shows used memory, not attempts) – Peter Lawrey Jul 22 '11 at 22:02
  • 2
    I see the following error on the system log message, this basically killed the java process `Jul 18 04:55:31 seqwd5 kernel: Out of memory: Killed process 16333, UID 501, (java). Jul 18 04:55:31 seqwd5 kernel: java invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0` – Kathir Jul 22 '11 at 22:39
  • This error occurs when you run out of memory and swap. I suggest you increase your swap space, or ideally your main memory. You could try reducing the amount of memory you use as it appear you don't have 1600M free. – Peter Lawrey Jul 23 '11 at 06:33
  • The server has 4gb or ram and there are no other processes running. I am starting to think my application is leaking memory somehow but its really difficult as the profilers are showing heap usage within limits. Thanks for your inputs. – Kathir Jul 23 '11 at 06:36
  • When you program exists, try to see how much free memory the system has before and after it dies. i.e. write a script which polls the memory usage regularly. This will tell you how much memory was released when the program exists. 4 GB should be more than enough. How big is your swap space? – Peter Lawrey Jul 23 '11 at 06:39
  • Java HotSpot(TM) 64-Bit Server VM (build 23.1-b03, mixed mode) releases memory back to the OS. Using the G1 garbage collector. http://www.stefankrause.net/wp/?p=14. I use -Xms32m -Xmx256m -XX:+UseG1GC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 on my server application and it releases memory back to the OS after running System.gc(). – Fireblaze Jul 31 '12 at 08:13
  • @Firebase I don't understand what you mean. – Peter Lawrey Jul 31 '12 at 08:15
  • @PeterLawrey please have a look at my question thanks. http://stackoverflow.com/questions/31108772/why-do-heap-memory-usage-and-number-of-loaded-classes-keep-increasing – Daniel Newtown Jun 29 '15 at 06:38
  • 2
    Just a comment, JVM is able to release the memory back to OS.. Please refer the page from Oracle. http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6498735 – huican Aug 25 '15 at 21:56
  • @huican it is able to but you have no control over this and it rarely happens. – Peter Lawrey Aug 26 '15 at 09:44
  • Where can I see the "direct memory" memory size and "share libraries" memory size be used? and any way to control "share libraries" memory size? – fcbflying May 05 '17 at 04:26
2

The actual memory consumption is more than what what you set with Xmx etc, that's normal. "java will allocate memory for other things, including a stack for each thread. It is not unusual for the total memory consumption of the VM to exceed the value of -Xmx."

ApriOri
  • 2,618
  • 29
  • 48
1

The paramaters -Xmx1600m -Xms1600m tells the JVM to allocate 1600MB of memory at minimum and 1600MB of memory at maximum. So the JVM should allocate 1600MB on start up and the never release it.

If you want the JVM to release memory back to the OS then the -Xms should be as low, and you probably have to use Java 1.7 with the new G1 garbage collector. stefankrause.net/wp/?p=14.

Using Mac OS X 10.8 and Java 1.7 with -Xms32m -Xmx256m -XX:+UseG1GC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 the memory is release back to the OS after a System.gc() is run.

Fireblaze
  • 242
  • 3
  • 5
  • 1
    This is actually incorrect. It results in the JVM allocating the full Xms 1.6GB and then releasing almost all of it after the first GC cycle if it's not in use. See http://java-monitor.com/forum/showthread.php?t=427 – Jim Mitchener Oct 29 '12 at 20:21
  • 1
    java-monitor.com/forum/showthread.php?t=427 <-- That article measure heap memory used, the question is about OS memory that the JVM is using. – Fireblaze Nov 07 '12 at 14:22
0

In the heap the Java Virtual Machine (JVM) stores all objects created by the Java application, e.g. by using the "new" operator. The Java garbage collector (gc) can logically separate the heap into different areas, so that the gc can faster identify objects which can get removed

The memory for new objects is allocated on the heap at run time. Instance variables live inside the object in which they are declared.

Stack is where the method invocations and the local variables are stored. If a method is called then its stack frame is put onto the top of the call stack. The stack frame holds the state of the method including which line of code is executing and the values of all local variables. The method at the top of the stack is always the current running method for that stack. Threads have their own call stack.

As said earlier in Java objects are created in the heap. The programming language does not offer the possibility to let the programmer decide if an objects should be generated in the stack. But in certain cases it would be desirable to allocate an object on the stack, as the memory allocation on the stack is cheaper then the memory allocation in the heap, deallocation on the stack is free and the stack is efficiently managed by the runtime.

The JVM uses therefore internally escape analysis to check if an object is used only with a thread or method. If the JVM identify this it may decide to create the object on the stack, increasing performance of the Java program. (http://www.ibm.com/developerworks/java/library/j-nativememory-linux/)

catch23
  • 17,519
  • 42
  • 144
  • 217