11

I am running a number of jobs on a computing cluster, and they are killed when they go over a requested resource usage - one of these uses is virtual memory size.

In my java startup command I use -Xmx8000m to indicate an initial stack size of 8GB, I have not yet seen my program's real memory usage go above 4GB, but wanted to be on the safe side.

However, when I use the top command I am seeing a virtual memory size for my java process of 12GB - which is right at the limit of the requested virtual memory space. I can't increase my requested VM size as the jobs are already submitted and the more I ask for the longer they take to be scheduled.

Does Java consistently request more VM heap space than is specified? Is this a constant amount, or a constant % or random? Can the heap space grow above a) the requested VM size (8GB) or b) the allocated VM size (12GB).

Edit: Using jre-1.7.0-openjdk on Linux

Dropout
  • 13,653
  • 10
  • 56
  • 109
Zack Newsham
  • 2,810
  • 1
  • 23
  • 43
  • What JRE are you using? – Silviu Burcea May 13 '15 at 09:45
  • are you using too many static variables? – Partha Bisoi May 13 '15 at 09:47
  • 4
    Heap never goes above the limit. But the VM requests consistently more virtual memory than its heap, because heap isn't the only memory it needs. There's also metaspace (in Java 8), memory for the VMs internal workings, memory-mapped libraries and so on. – biziclop May 13 '15 at 09:48
  • 3
    Stack size? Ain't that heap size? Also I'm concerned about "initial" - it is "maximum", isn't it? – Thomas Weller May 13 '15 at 09:49
  • This one is interesting. @biziclop. What is metaspace ? – Partha Bisoi May 13 '15 at 09:49
  • 1
    On Windows, virtual memory is also consumed by libraries (DLLs), stack, memory mapped files etc. This is probably similar on Linux (as you say you're using `top`). On Windows there is VMMap which shows you the type of virtual memory that is spent on a process. – Thomas Weller May 13 '15 at 09:50
  • 1
    @ParthaBisoi There is a very short introduction [here](https://blogs.oracle.com/poonam/entry/about_g1_garbage_collector_permanent), but roughly what used to be the PermGen and stored as part of the Java heap is now Metaspace and stored on a separate (native) heap. – biziclop May 13 '15 at 09:52
  • @Thomas Yes, let's not forget about stacks either. They can take up quite a huge chunk of allocated virtual address space, though their actual memory footprint is usually much smaller. – biziclop May 13 '15 at 10:00
  • @SilviuBurcea OpenJRE 1.7 – Zack Newsham May 13 '15 at 10:06
  • @ParthaBisoi what would be "too many"? I dont use many as far as I know, and my RES memory never goes above 2.7GB – Zack Newsham May 13 '15 at 10:07
  • @biziclop I'm using Java 7 (I think) so no metaspace I think, does the amount of memory reserved for non heap usage increase proportionately with the heap requested - 4GB of non heap space seems a little high, but being exactly 50% of the requested seems quite "neat" – Zack Newsham May 13 '15 at 10:10
  • This one is interesting. I suspect there might be some jars which internally would be consuming the memory space. @ Zack Newsham Do you suspect any jars ? – Partha Bisoi May 13 '15 at 10:18

3 Answers3

11

This article gives a good analysis of the problem: Why does my Java process consume more memory than Xmx And its author offers this approximate formula:

Max memory = [-Xmx] + [-XX:MaxPermSize] + number_of_threads * [-Xss]

But besides the memory consumed by your application, the JVM itself also needs some elbow room. - Garbage collection. - JIT optimization. - Off-heap allocations. - JNI code. - Metaspace.

But be carefull as it may depend on both the platform and the JVM vendor/version.

epoch
  • 16,396
  • 4
  • 43
  • 71
Volatile
  • 411
  • 4
  • 7
5

This could be due to the change in malloc behavior in glibc 2.10+, where malloc now creates per-thread memory pools (arenas). The arena size on 64-bit is 64MB. After using 8 arenas on 64-bit, malloc sets the number of arenas to be number_of_cpus * 8. So if you are using a machine with many processor cores, the virtual size is set to a large amount very quickly, even though the actual memory used (resident size) is much smaller.

Since you are seeing top show 12GB virtual size, you are probably using a 64-bit machine with 24 cores or HW threads, giving 24 * 8 * 64MB = 12GB. The amount of virtual memory allocated varies with number of cores, and the amount will change depending on the number of cores on the machine your job gets sent to run on, so this check is not meaningful.

If you are using hadoop or yarn and get the warning, set yarn.nodemanager.vmem-check-enabled in yarn-site.xml to false.

References:

See #6 on this page:

http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/

which links to more in-depth discussion on this page:

https://www.ibm.com/developerworks/community/blogs/kevgrig/entry/linux_glibc_2_10_rhel_6_malloc_may_show_excessive_virtual_memory_usage

Note this is already partially answered on this stackoverflow page:

Container is running beyond memory limits

jamesy
  • 51
  • 1
  • 3
0

You you are really keen to investigate the issue and you are on Linux then check PID of your jvm process and view the file /proc/<PID>/smaps. There you will find the whole OS process memory map - as seen by the kernel. You will see how mach heap(OS heap) process uses, which memory regions are mapped from files(libraries), ... etc.

PS: you can also files various tools, to analyze the smaps file, on the Internet.

ibre5041
  • 4,903
  • 1
  • 20
  • 35