3

Setup:

  • Hazelcast: 3.12.3
  • Spring Boot: 2.1.6
  • Java: 1.8
  • Args: -XX:+UseG1GC
  • Xms7g -Xmx7g These are running in Docker: openjdk:8
  • they run in dedicated vmware nodes CentOS 7.5.1804

I have 7 nodes in total that form a hazelcast cluster. 5 of them are CP members. The groups size is set to 3. Other notable config changes: setSessionTimeToLiveSeconds(200), setMissingCPMemberAutoRemovalSeconds(240).

Usage: I have 8 clients that try to acquire the fenced lock for a Long key, the interface is cached for 5 minutes. The keys are almost always different, but in some cases we receive many actions for the same key and we want to avoid async update for these keys. In the past I used IMap for this but I noticed that it not always respected the lock. Fenced lock seems to perform very well until it reaches a certain memory usage.

Issue: What I observed is that one cp member is used heavily and the G1 Old Gen heap is ever so slightly increases over time. In one day it reached 5GB. And even after I stop the requests for lock the memory is never cleared. enter image description here

I tried to create two groups to utilize more nodes in the cluster, it did help, but the issue persists, the used heap on a few nodes increases over time and at point they become very slow.

Question: What am I doing wrong? Is this even a valid use case for this tool?

Miklós Kosárkár
  • 117
  • 1
  • 13
  • Do you create a new lock for each key or use the same lock? A reproducer test code would help. – wildnez Nov 17 '19 at 23:11
  • For a unique key I cache the lock for 5 minutes, so if I receive 30 updates for the same ID the lock is reused. But in 90% of the cases a new lock is created. The load is about 10k lock in a minute. All I'm wondering about is what can linger in the memory for days. I would expect the lock objects to expire after some time. I see no way to clean them manually so I guess my use case is just not what this is meant for. – Miklós Kosárkár Nov 18 '19 at 16:38
  • I was thinking about playing with the groups. I think groups can be destroyed, so I could change between a few groups to delete unused locks. – Miklós Kosárkár Nov 19 '19 at 20:27

1 Answers1

2

Lock objects do not expire on their own and Hazelcast does not automatically perform garbage collection on unused locks. If you no longer need a lock, you can destroy it using lock.destroy(), otherwise you run the risk of running into OOME.

wildnez
  • 1,078
  • 1
  • 6
  • 10
  • I don't think It works the way I want it to. If I destroy the lock it I cannot create it again. I did a time based destroy, if the lock was not accessed for a certain amount of time I destroyed it but after that if I try to get a lock with the same key it will say that it is already destroyed. – Miklós Kosárkár Nov 22 '19 at 15:31
  • Not sure what you intend to achieve. If there are objects created, they will consume some memory. If there are lots of objects created then they will consume lots of memory, and if you do not clean up memory then all of your memory will end up getting utilized. And when you clean up memory by removing objects you dont need then of course the objects will not exist. What is it that I am missing here? – wildnez Dec 02 '19 at 01:43
  • 1
    Let's say I want to lock an object now and two hours later. If I create the lock, it will take up space, I want to delete it if I don't use it for some time. But when I tried to create it again, I got an already destroyed exception. I think I'll use the IMap lock which as far as I know can clean up unused locks. – Miklós Kosárkár Dec 13 '19 at 12:48
  • 1
    Did you find correct way of handling this? I am in same situation where I want to destroy() FencedLock if not used and then re-create on demand at later point in time. – amit.bhayani Jun 19 '20 at 05:32