14

I have 7 different java daemons that I run (all 7) on 3 different servers. The java command line has -Xmx2048m and -Xss1024k. On these 3 servers, all 21 processes show just under 2.5 GB for VIRT size in top and atop. RES size varies from 300 to 1.9 GB according to which daemon it is.

That is all as it should be.

Enter the new server. Faster CPU, more RAM (16 GB instead of 8 GB), slightly newer java (1.6.0_10-b33 on the old servers, 1.6.0_31-b04 on the new server). Both systems (and JVMs) are 64bit.

Moved 2 of the daemons to the new server. On the new server, given the same task, the daemons are both consuming vastly more CPU (about a core's worth) and getting Less done. (Moved from 5110 processors on the old systems to 5620s on the new one).

Pretty much a full extra core of CPU usage (GC thread??) and reporting 5 GB VIRT and 2 GB RES for one daemon and 10.5 GB VIRT and 2 GB RES for the other daemon.

Any ideas what would cause java to ignore (or appear to ignore if that is the case) the memory limits?

Saurabh
  • 71,488
  • 40
  • 181
  • 244
Wayne Walker
  • 2,316
  • 3
  • 23
  • 25
  • VIRT is virtual memory, which has no direct relationship to heap space. I recommend reading answers to this question: http://stackoverflow.com/questions/4893192/process-memory-vs-heap-jvm – Matt Ball May 13 '12 at 21:20
  • Did you try with the same version of Java on the new machine? Are they both the same bit-ness? – Dave Newton May 13 '12 at 21:22
  • `Xmx` does not specify the memory usage of the JVM; it specifies the maximum size of the heap allocation pool, which is just one of the memory pools used by the JVM. – skaffman May 13 '12 at 21:24
  • GC1, larger perm space. (wildy guessing, but fits the versions.) You can try by forcing the vm's to use another gc. – esej May 13 '12 at 21:24
  • @MattBall Interesting reading. I'm not watching memory for memroy's sake, the app is performing poorly under load, and is consuming 4 times the VIRT print performing the exact same task (same data set). Since any memory that shows up in VIRT has been malloc'd and therfore may be actually used at any moment, and its pointers are _probably_ ( but not necessarily) walked by the GC thread, it seems that it may be indicative, if not guilty, of whatever is causing horrible performance. – Wayne Walker May 13 '12 at 22:00
  • Are you maybe running the others using the `-XX:+UseCompressedOops` option and the new one not? (That option is recommended anyways on a 64bit system when you don't need more than 32GB of heap space) –  May 13 '12 at 22:42
  • I'm going to go left and suggest that you check that all DNS entries for the new server match the hostname, and have rev-ip lookups set up as well. – Jonathan May 13 '12 at 23:28

1 Answers1

14

Turns out this is a glibc problem.

The short answer for me was:

export MALLOC_ARENA_MAX=1

This decreased process footprint (VIRT in top) by as much as 5x. Back to the levels seen in CentOS 5.

Recent versions of glibc have a new feature "per-thread memory pools":

http://www.centos.org/docs/5/html/5.4/Technical_Notes/glibc.html

The last item in the 1.71.1 log section discusses it (and refers to a non-public bug....)

Wayne Walker
  • 2,316
  • 3
  • 23
  • 25
  • 5
    You might want to set MALLOC_ARENA_MAX to 4 instead of 1, to get some of the multi-threading memory benefits that the change was intended to address. See this: issues.apache.org/jira/browse/HADOOP-7154 – michael Jul 23 '14 at 22:02