1

I am looking for a readers-writer lock that can be used in python in a distributed system.

So far I've found:

  1. redlock, which is based on redis. Does not provide a readers-writer lock.
  2. The distributed package of dask offers a lock, but again, no readers-writer lock.
  3. kazoo, which works with Zookeeper, offers a readers-writer lock. However Zookeeper is an extremely heavy dependency, as it written in Java and therefore requires the JDK.

Is there a more lightweight alternative to kazoo / Zookeeper? Ideally a pure python solution that is nevertheless battle tested?

Arco Bast
  • 3,595
  • 2
  • 26
  • 53
  • 2
    Can this module help you https://github.com/veshboo/redisrwlock ? – Anton Pomieshchenko Apr 02 '20 at 19:09
  • Maybe ... but as the last change was 3 years ago I'd say it is unmaintained. I'd need to backport to python 2.7. – Arco Bast Apr 02 '20 at 19:46
  • I feel that the requirements should be more precise. What communication mechanisms allowed? What queueing discipline is required? What are the recovery expectation if a node goes down while holding a lock or the distributed system becomes partitioned? – Dima Tisnek Apr 06 '20 at 07:36
  • @DimaTisnek Thanks for your comment. Communication: I'd imagine some service to which I can connect using IP and port, similar to how it is done in redlock. No enncryption required. Queueing discipline: ideally writers priority. Recovery expectation: When a node goes down any locks it holds are released. I am fine not to be prepared for the case the distributed system becomes partitioned. Do you feel this clarifies your questions a bit? – Arco Bast Apr 06 '20 at 10:36

1 Answers1

2

Redis is single-threaded and read/write operations are atomic; which means multiple simultaneous read/write operations will not succeed and readers and writer(s) will be obtaining access to the resource sequentially in atomic fashion. I believe this would solve your problem if the resource is kept in as one of the redis data structures. You do not need a separate reader-writer lock mechanism when using redis and python as combination where the resource is kept within redis.

Hope it helps

choppe
  • 493
  • 2
  • 11
  • Can you illustrate how that would be done? I'd like to organize access to my resource such that readers can access it simultaneously and writers only sequentially. If a writer has access, no reader has. If a writer requests access, no new access to readers is granted ('writers priority'). – Arco Bast Apr 05 '20 at 20:37
  • If your critical resource is located in redis; then you do not have to worry specifically about access protecting it with locks since, redis executes all commands sequentially whether it is read or write; Since redis is in-memory database the throughput of such operations are very high. Please note again that this applies for critical resource kept in as a redis data structure. Additionally, redis supports a concept such as transactions; even transactions are executed sequentially; please refer here https://redis.io/topics/transactions. – choppe Apr 06 '20 at 12:46
  • https://stackoverflow.com/questions/45364256/why-redis-is-single-threadedevent-driven – choppe Apr 06 '20 at 13:24
  • Additionally, when you use redis for such activity you might not be able address second or third readers writers problem ( ref https://en.wikipedia.org/wiki/Readers%E2%80%93writers_problem); which is used to ensure no process/thread shall starve. by design redis will starve any pending commands ( read or write ) until current command execution completes. but again without understanding your problem at hand in detail it is difficult to propose a solution on the fly. But redis will not be able to solve your requirement where you need writers priority. – choppe Apr 06 '20 at 13:37
  • Ah, I see your point, if the resource *was* in redis, there wouldn't be a need for a lock. But the resource is a bunch of files (millions, amounting to several TB, containing simulated data in research), and all existing simulation and analysis routines developed in the last decade work great with that data structure. I therefore do not want to change it. My question is, if there is a readers-writer lock mechanism already available in python. 'No' would also be a valid answer ... – Arco Bast Apr 06 '20 at 14:26
  • Of course python alone would not support it. But what you need is a distributed locking mechanism in python. This might not be achievable without a back-end for lock synchronization. I believe you have following choices when it comes to back-end [ mechached , redis etc] . for example https://github.com/vaidik/sherlock this provides distributed locking mechanism; you might have to customize it to fit your write-priority locking. Hope it was helpful. Thank you. – choppe Apr 06 '20 at 15:28
  • 1
    my point was to share with you that redis as a backend would not fit your requirement of write-priority locking as it is limited by design for your purpose. so you can leave redis in your quest for the choice of backend. try memchaced or anyother means. – choppe Apr 06 '20 at 15:51