5

It's a Java web application on Websphere6.1, Solaris 10, JDK 1.5.0_13. We set the maximum heap size to 1024m. jmap shows the heap status is healthy. The heap memory usage is only 57%. No OutOfMemory at all.

But we saw very high RSS (3GB) for this java process from ps. pmap shows a block of 1.9G private memory.

3785:   /dmwdkpmmkg/was/610/java/bin/java -server -Dwas.status.socket=65370 -X
 Address  Kbytes     RSS    Anon  Locked Pgsz Mode   Mapped File
...
0020A000    2008    2008    2008       -   8K rwx--    [ heap ]
00400000 1957888 1957888 1957888       -   4M rwx--    [ heap ]
8D076000      40      40      40       -   8K rw--R    [ stack tid=10786 ]
...

Is it a C heap memory leak in native code? What approach is recommended to find out the root cause?

Michael Myers
  • 188,989
  • 46
  • 291
  • 292
gengmao
  • 113
  • 1
  • 6

4 Answers4

4

This Troubleshooting Memory Leaks document from Sun may help you with finding the problem why your high RSS, specially in section 3.4.

As you are running Websphere, maybe you can use the -memorycheck on your VM. For details see here.

It's not necessarily a leak in native code. If you look here, on Solaris there might be an issue with files being kept open.

It's just a bunch of links and hints, but maybe helpful to track down your problem.

MicSim
  • 26,265
  • 16
  • 90
  • 133
  • They're helpful. While -memorycheck seems only available for IBM JDK? There's no IBM JDK on Solaris, just SUN JDK, right? – gengmao Mar 27 '09 at 16:46
  • Don't know, I have no experience with Solaris, but likely not ... Maybe the IBM JDK is coming with Websphere as it's also from IBM. That's what I thought of. – MicSim Mar 27 '09 at 17:26
3

This might happen even when there isn't a native memory leak (like unclosed zip resources).

I ran in to the same problem. This is a known problem with glibc >= 2.10

The cure is to set this env variable export MALLOC_ARENA_MAX=4

IBM article about setting MALLOC_ARENA_MAX https://www.ibm.com/developerworks/community/blogs/kevgrig/entry/linux_glibc_2_10_rhel_6_malloc_may_show_excessive_virtual_memory_usage?lang=en

Google for MALLOC_ARENA_MAX or search for it on SO to find a lot of references.

You might want to tune also other malloc options to optimize for low fragmentation of allocated memory:

# tune glibc memory allocation, optimize for low fragmentation
# limit the number of arenas
# requires glibc >= 2.16 since there was a bug in 
# MALLOC_ARENA_TEST parameter handling that cause MALLOC_ARENA_MAX not to work
export MALLOC_ARENA_MAX=2
# disable dynamic mmap threshold, see M_MMAP_THRESHOLD in "man mallopt"
export MALLOC_MMAP_THRESHOLD_=131072
export MALLOC_TRIM_THRESHOLD_=131072
export MALLOC_TOP_PAD_=131072
export MALLOC_MMAP_MAX_=65536

You can call the native malloc_info function to get information about memory allocations. Here is an example of using JNA to call the native malloc_info method.

Lari Hotari
  • 5,190
  • 1
  • 36
  • 43
  • Thanks Lari! Your answer is very convicing. However I don't have the app and solaris env to verify anymore - the origin question was raise 6 years ago :) But thanks for letting me know anyway. The question has been mysterious for me for a while. – gengmao Apr 28 '15 at 00:31
  • Check this answer: https://stackoverflow.com/a/35610063 , it has more details about Java native memory leaks. – Lari Hotari Sep 13 '19 at 05:11
2

The heap size os the Java heap size, there is still the VM and other libraries which are part of the process.

Try running Hello World with a 1024m heap size and a "for(;;)" in it and see how much it takes up. That should give you a baseline for overall memory usage.

TofuBeer
  • 60,850
  • 18
  • 118
  • 163
1

Are you using JNI libraries? I'm not sure how native code allocates RAM but that's where I'd start looking.

Chris Nava
  • 6,614
  • 3
  • 25
  • 31