0

I have installed a cluster of 3 hazelcast nodes. Each hazelcast node is installed in a dedicated VM (running with centos 7). Each VM has 16GB RAM and a 15GB HDD. for each hazelcast node I set the heap memory size in the start.sh script as MIN HEAP SIZE=8G and MAX HEAP SIZE=10G.

However, when I monitor the memory usage on the VM, it is not matching the heap memory usage of hazelcast at all. Let me give you an example: the VM is showing the memory usage 80%. however when I check hazelcast heap memory usage it is at 45%. it is like the VM memory usage is following the higher memory usage of hazelcast. then when the GC starts in hazelcast the heap memory usage on hazelcast decreases but the VM memory usage never decreases. in the VM I also disable THP and set the swappiness to 10 as mentioned in the performance tuning documentation Performance Tuning (hazelcast.com)

So is there a missing configuration that I should do for hazelcast? What can/should I do?

Nimantha
  • 6,405
  • 6
  • 28
  • 69
karim
  • 11
  • 2

1 Answers1

1

it is like the VM memory usage is following the higher memory usage of hazelcast. then when the GC starts in hazelcast the heap memory usage on hazelcast decreases but the VM memory usage never decreases.

This is typical behaviour for any JVM application, it is not specific to Hazelcast. By default JVM rarely returns the memory to OS, but it is possible to configure it that way.

See e.g. this answer for more details on behaviours of particular JVMs.

You say you are using a dedicated VM for Hazelcast. In such scenario, it doesn't make sense to return the memory to OS and it is best to use the same MIN and MAX heap size so there is no heap resizing when Hazelcast is running.

František Hartman
  • 14,436
  • 2
  • 40
  • 60
  • Hello Frantisek, thank you a lot for you answer. what about the swappiness that hazelcast suggests to disable it on the VM which is hosting hazelcast? Actually I did a test with disabling the swappiness but it did not work at all. the memory continued increasing until reaching 100% so I set it back to 10. So if I well understood you I should set the same value for min and max heap size and do not return the memory to OS since I have a dedicated VM for hazelcast. Should I keep swappiness set with 10 or should I increase it? – karim Dec 08 '21 at 16:00