1

I have below configuration for my java actvemq process where Xmx is 15360M but when I see the process usage of memory using glance I see RSS 41.8gb and VSS 51.6gb . pmap |grep total gives me total 54059348K. I am not sure what is happening in activemq that is causing this much amount of memory being utilized. While heap usage is pretty low till 60%.

In the output of pmap I see lot of anon blocks. Please find attached PMAP output 

/XXXXX 111922      1 22 14:30 pts/0    00:04:26 
/u/users/user/jdk1.7.0_71/bin/java -Xms1024M -Xmx15360M -Xss512K 
-XX:+UseG1GC -XX:NewRatio=4 -XX:InitiatingHeapOccupancyPercent=45 
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/x/logs/AMQGC.log 
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=20M 
-Djava.util.logging.config.file=logging.properties 
-Djava.security.auth.login.config=/u/users/user/POSInterfaces/apache-activemq-5.13.2//conf/login.config 
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote 
-Djava.awt.headless=true 
-Djava.io.tmpdir=/u/users/user/POSInterfaces/apache-activemq-5.13.2//tmp 
-Dactivemq.classpath=/u/users/user/POSInterfaces/apache-activemq-5.13.2//conf:/u/users/user/POSInterfaces/apache-activemq-5.13.2//../lib/: 
-Dactivemq.home=/u/users/user/POSInterfaces/apache-activemq-5.13.2/ 
-Dactivemq.base=/u/users/user/POSInterfaces/apache-activemq-5.13.2/ 
-Dactivemq.conf=/u/users/user/POSInterfaces/apache-activemq-5.13.2//conf 
-Dactivemq.data=/u/users/user/POSInterfaces/apache-activemq-5.13.2//data 
-jar /u/users/user/POSInterfaces/apache-activemq-5.13.2//bin/activemq.jar 
start/

Please find pmap file here

Abhinav Suryawanshi
  • 161
  • 3
  • 13
  • The anon blocks could be thread stacks. A thread dump (jstack) may provide more information. I'd be interested to know why you need to set Xmx=15Gb -- this is colossally high for a message broker. – Kevin Boone Sep 05 '17 at 08:12
  • Dear kevin Glad you replied . We have previosly worked together when you were at RedHat. We have provided Xss 512k to reduce stack size for threads.I have given 15Gb to make sure that AMQ has all the heap it requires as we are processing very high magnitude of data. – Abhinav Suryawanshi Sep 05 '17 at 16:30
  • Tried reducing thread count by Xss 256K added -Dorg.apache.activemq.UseDedicatedTaskRunner=FALSE and set optimizedDispatch=true. This reduced thread count from 257 to 85 on start up but glance command still shows RSS 489mb and VSS 25gb even on startup without any messages. Over the time values are going up till RSS 40gb and VSS 52gb with 400 threads and No Xss set. – Abhinav Suryawanshi Sep 05 '17 at 20:31
  • I am still at Red Hat -- we are closing our proprietary public forums and moving to StackOverflow. However, this looks like a problem you should raise a support ticket for, if you still are able to. ActiveMQ shouldn't use huge amounts of heap, even if the data throughput is huge, unless you're actually working with non-persistent (in memory) messages. Using large (megabyte) messages with default values of pre-fetch (usually 1000) will eat the heap, as will things that cause large number of threads to be spawned. This situation really needs detailed analysis of heap and thread dumps. – Kevin Boone Sep 06 '17 at 08:15
  • Not able to raise ticket anymore. Problem here is Heap is normal but still huge memory is being consumed. – Abhinav Suryawanshi Sep 06 '17 at 08:25
  • Sure. The problem is probably a large number of thread stacks. Look at the jstack output for clues. Sorry, I'm sure you'll understand that Red Hat won't want me doing paid support work via a non-paid channel, We're only support to use SO for developer guidance. – Kevin Boone Sep 06 '17 at 08:29

3 Answers3

0

I can confirm that ActiveMQ is unable to use any more JVM heap than you configure with -Xmx. ActiveMQ is 100% Java, so no other heap memory than that of the JVM will be used by the broker.

To find out how much Java heap is really used, query the JMX statistics of the JVM, it should always remain within the configured limit.

I am not familiar with pmap, so I cannot really comment on that part but am wondering what numbers the top command gives you?

Torsten Mielke
  • 239
  • 1
  • 3
  • I am not talking about Java heap here. ActiveMQ is consuming more memory than configured Xmx and it is very normal for multi threaded Enterprise Java applications to consume more memory than configured Java Heap. ActiveMQ is not Pure Java solution it uses lot of JNI code. – Abhinav Suryawanshi Sep 05 '17 at 16:28
  • Torsten My bad I apologize for naive comment, I confused JNI client with JNI code but still Java will use more memory than allocated Heap . Its not for heap but for something else which I am not able to figure out. Heap alwasy remains within limits but still AMQ java process is consuming 4 times more space of RAM than allocated heap. Top and ps are also 4 times showing more memory usage than actual Heap alloated. – Abhinav Suryawanshi Sep 13 '17 at 16:51
0

I take it from reference to RSS that you are running on Unix - not sure if that's Linux.

I'm wondering if you should be looking at PSS (Proportional Set Size) rather than RSS (Resident Set Size) - see this Unix & Linux Question on memory - I have found smem to be a great tool for assessing actual memory usage.

For example - for an ActiveMQ process on my server (set with -Xmx512m)

[user@vm ~]$ sudo pmap 13727 | grep total
 total          3161064K

[user@vm ~]$ sudo smem | grep -e '13727' -e 'PSS'
  PID User     Command                         Swap      USS      PSS      RSS
13727 activemq /usr/bin/java -Xms64M -Xmx1        0   212604   214962   229468

i.e. pmap giving me a much higher memory figure (3GB) than smem PSS (and indeed RSS) (~200M)

Dazed
  • 1,069
  • 9
  • 34
0

To profile memory usage of your application (including not only Java Heap), you can use this answer.

Most of your anon blocks have close size. In most cases a lot of anon talk about threads. You could have a lot of threads, if you have a lot of destinations in ActiveMQ. But sizes of your anon blocks are not close to Thread Stack Size. So it could be not threads, I suppose.

Note that 15Gb is very high for ActiveMQ and sometimes for Java too, it could affect GC very strong. You can read how to configure ActiveMQ for system memory usage here and here.

egorlitvinenko
  • 2,736
  • 2
  • 16
  • 36