Java native memory leak on Debian

Question

I have serious issues with a Java application deployed in Tomcat:

OS: Debian 6.06 (kernel 3.2.13-grsec-xxxx-grs-ipv6)
Tomcat: 6.0.35
JDK: 1.6.0_37-b06
JVM params: -Xms3584m -Xmx3584m -XX:MaxPermSize=256m -XX:ThreadStackSize=1024
Thread counts: 200

After couple of hours of usage the RSS (Resident memory size) is 13GB and VSZ (Virtual memory size) is 15 GB.

The application has 2 servlets: a Spring DispatcherServlet for some simple HTTP requests and CXFServlet for handling WebServices. The application has no any custom native code.

The used head is around 2 GB, so this is not an issue.

A SOAP request that returns a response of 4 MB generates a 500 MB increase of RSS. Running garbage collector has no effect on RSS.

Do you know any possible reason of this memory increase or any tool that can help me to investigate the issue. Thanks.

pmap output

mapped: 14226976K writeable/private: 13772580K shared: 286844K

10 largest "anon" blocks:

0000000720000000 3670016 rw--- 0000000000000000 000:00000   [ anon ]
0000000000601000 2529344 rw--- 0000000000000000 000:00000   [ anon ]
0000000710000000  173504 rw--- 0000000000000000 000:00000   [ anon ]
00007f7484000000  131072 rw--- 0000000000000000 000:00000   [ anon ]
00007f7414000000  131068 rw--- 0000000000000000 000:00000   [ anon ]
00007f7424000000  131068 rw--- 0000000000000000 000:00000   [ anon ]
00007f7434000000  131068 rw--- 0000000000000000 000:00000   [ anon ]
00007f7494000000  131068 rw--- 0000000000000000 000:00000   [ anon ]
00007f737c000000  131024 rw--- 0000000000000000 000:00000   [ anon ]
00007f738c000000  131024 rw--- 0000000000000000 000:00000   [ anon ]

Please refer http://stackoverflow.com/questions/686397/how-can-a-java-process-with-xmx1024m-occupy-3gb-resident-memory — Subba, Feb 08 '13 at 16:54
What does *pmap* say? Is all of the virtual memory being consumed by "anon" blocks or by mapped files? If the former, please edit your question with the 10 largest "anon" blocks. — parsifal, Feb 08 '13 at 17:12

score 0 · Answer 1 · answered Feb 08 '13 at 16:16

0

You can get a good look at what is going on inside your JVM with JVisualVM, an analytical tool included with java. The manual can be found here.

JVisualVM

There are other, better tools(JProfiler 7 comes to mind), but they usually have a license fee. You could try the free trial of JProfiler.

answered Feb 08 '13 at 16:16

The Ox

35
1
5

I second that. VisualVM is a beautiful tool for profiling. I'd add you better use the "sampler" profiler instead of the "Profiler" profiler. It's way faster and does no byte code weaving tricks. If you set up JMX for remote connections you can check it all from your desk. Start Tomcat with: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port= -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=. You'll be able to remote connect with VisualVM on :. – Jan Goyvaerts Feb 08 '13 at 16:21
1

I don't have any issue with the heap, it doesn't go over 2 GB. I integrated JavaMelody in my app so i know everything I have in heap. I've used JProfiler and JVisualVM for heap analyzing, I'm not sure how much they can help me investigating the native memory. – user2054933 Feb 08 '13 at 16:24

score 0 · Answer 2 · edited May 23 '17 at 11:51

0

What exactly is the problem? Is the OS swapping? Perhaps you only need to read the accepted answer to this question: Virtual Memory Usage from Java under Linux, too much memory used

If you are using NIO, that could also be a source of memory consumption outside the heap, as described here: Why does the Sun JVM continue to consume ever more RSS memory even when the heap, etc sizes are stable?

edited May 23 '17 at 11:51

Community

1
1

answered Feb 08 '13 at 16:36

lbalazscs

17,474
7
42
50

The problem is that once the RES consume all the memory the kernel is killing the process. Thanks for the link. It looks interesting. – user2054933 Feb 08 '13 at 17:01

Java native memory leak on Debian

2 Answers2