2

I have a use case where I want to have a globally distributed lock. We started out using SELECT .. FOR UPDATE, but that quickly started to have problems as we scaled up the number of servers. Also it didn't account for processes that checked out the lock and then died and failed to return the lock.

We need to be able to set an expiration on the lock (i.e. if the process who checked out the lock does not return it in 2 hours, the lock is automatically returned to the pool). I realize that this introduces the issue where we are ignoring locks, but we are fairly certain that the process has died if not complete in 2 hours. Also the job is idempotent, so if it is done more than once it's not a big deal.

I've looked through a number of distributed locking systems and come across this questions that have been extremely helpful. All of the solutions extend off of Java's java.util.concurrency.locks.Lock, which actually may be the issue I'm coming across because that interface doesn't have the expiration feature I need. We have a similar strategy to mongo-java-distributed-lock where we use MongoDB's findAndModify. We're considering:

as our distributed locking mechanism (all happen to implement java.util.concurrency.locks.Lock).

The biggest problem is that because java.util.concurrency.locks.Lock doesn't have an option for expiring a lock, these don't fit all the goals. This answer probably gets closest with hazelcast, but it is reliant on an entire server failing, not just a thread taking too long. Another option is possibly using a Samaphore with hazelcast as described here. I could have a reaper thread that is then able to cancel the locks of others if they are taking too long. With Mongo and Redis I could take advantage of their ability to expire objects, but that doesn't seem to be part of either of the libraries since they just implement java.util.concurrency.locks.Lock in the end.

So this was just a long winded way of asking, is there a distributed locking mechanism out there that I can have automatically expire after N seconds? Should I be looking at a different mechanism than java.util.concurrency.locks.Lock in this situation altogether?

Community
  • 1
  • 1
Scott
  • 16,711
  • 14
  • 75
  • 120
  • If you're happy rolling your own then you could consider extending Lock and adding a static watcher thread that keeps a list of live locks and periodically checks expiration times. – selig Apr 01 '14 at 14:34
  • I'd love rolling my own, but also don't want to re-invent the wheel if there is a solution already out there to do this. – Scott Apr 01 '14 at 16:59
  • Possible duplicate: http://stackoverflow.com/questions/1059580/distributed-lock-service – flup Apr 14 '14 at 22:43
  • @flup, I'm explicitly asking for a solution that offers an expiring lock. All the other SO questions are only about distributed locks, not distributed expiring locks, the key feature I care about. – Scott Apr 16 '14 at 20:18

4 Answers4

3

You should consider using zookeeper. And there is an easy to use library for these kind of "distributed" stuff which is built on top of zookeeper : curator framework. I think what you are looking for is shared reentrant lock. You can also check other locks in recipes.

frail
  • 4,123
  • 2
  • 30
  • 38
  • I did not see where zookeeper provided expiring locks. I'm aware of it as a solution for distributed locks, but it didn't seem to offer the expiring feature I was looking for. If that feature does exist, can you provide a link to it as I was not able to find it. – Scott Apr 16 '14 at 20:19
  • well in zookeeper the lock would expire if the lock owner session drop which happens just like your hazelcast example. I can understand your "crash" concerns; but giving a timeout for a lock itself seems weird at best :) – frail Apr 16 '14 at 21:17
  • I get the 'crash' portion. The use case is I have many Actors locking assets. One of those actors dies, but the server still hums along. The work that actor was working on needs to go back into the pool after a period of time to avoid that work never being done. The work is idempotent, so I don't care it gets done twice, just that it gets done. I don't think this is 'weird', it has come up more than a few times as a requirement for me. – Scott Apr 17 '14 at 14:08
3

You may use Redisson based on Redis server. It implements familiar Java data structures including java.util.Lock with distributed and scalable abilities. Including ability to setup lock release timeout. Usage example:

Config config = new Config();
// for single server
config.useSingleServer()
      .setAddress("127.0.0.1:6379");
// or 
// for master/slave servers
config.useSentinelConnection()
      .setMasterName("mymaster")
      .addSentinelAddress("127.0.0.1:26389", "127.0.0.1:26379");

Redisson redisson = Redisson.create(config);

Lock lock = redisson.getLock("anyLock");
try {
   // unlock automatically after 10 seconds of hold
   lock.lock(10, TimeUnit.SECONDS);

} finally {
   lock.unlock();
}

...

redisson.shutdown();
Nikita Koksharov
  • 10,283
  • 1
  • 62
  • 71
  • Linking to other answer as well for this feature in case someone else comes across. http://stackoverflow.com/a/21072066/311525 – Scott Aug 26 '14 at 22:48
  • What happens to the lock if the holding process dies? – Mike Stoddart Oct 05 '18 at 16:23
  • 1
    @MikeStoddart Redisson maintains lock watchdog, it prolongs lock expiration while lock holder Redisson instance is alive. – Nikita Koksharov Oct 05 '18 at 16:28
  • Thanks @NikitaKoksharov, which I assume means the lock isn't released when the holding process dies, but rather when the watchdog cleans it up? – Mike Stoddart Oct 06 '18 at 17:44
  • 1
    @MikeStoddart lock would be released automatically if watchdog won't extend it. If process dies then watchdog dies either and thus lock will be released since it wasn't extended – Nikita Koksharov Oct 08 '18 at 14:24
2

What about this one? http://www.gemstone.com/docs/5.5.0/product/docs/japi/com/gemstone/gemfire/distributed/DistributedLockService.html

Its lock method seems to have what you need:

public abstract boolean lock(Object name, long waitTimeMillis, long leaseTimeMillis)

Attempts to acquire a lock named name. Returns true as soon as the lock is acquired. If the lock is currently held by another thread in this or any other process in the distributed system, or another thread in the system has locked the entire service, this method keeps trying to acquire the lock for up to waitTimeMillis before giving up and returning false. If the lock is acquired, it is held until unlock(Object name) is invoked, or until leaseTimeMillis milliseconds have passed since the lock was granted - whichever comes first.

xxa
  • 1,258
  • 9
  • 20
1

Actually, as far as I can tell mongo-java-distributed-lock has the ability to expire a lock through the use of DistributedLockOptions.setInactiveLockTimeout()

I haven't tried it yet, but think I will...

EDIT: I have now also tried it, and it works well...

String lockName = "com.yourcompany.yourapplication.somelock";
int lockTimeoutMilliSeconds = 500;

String dbURI = CWConfig.get().getMongoDBConfig().getDbURI();
DistributedLockSvcFactory lockSvcFactory = new DistributedLockSvcFactory(new DistributedLockSvcOptions(dbURI));
DistributedLockSvc lockSvc = lockSvcFactory.getLockSvc();

DistributedLock lock = lockSvc.create(lockName);
lock.getOptions().setInactiveLockTimeout(lockTimeoutMilliSeconds);
try {
    lock.lock();
    // Do work
} finally {
    lock.unlock();
}
linusn
  • 11
  • 4