How to lock a critical section in Django?

Question

I can't find a good clean way to lock a critical section in Django. I could use a lock or semaphore but the python implementation is for threads only, so if the production server forks then those will not be respected. Does anyone know of a way (I am thinking posix semaphores right now) to guarantee a lock across processes, or barring that a way to stop a Django server from forking.

score 17 · Answer 1 · answered Nov 30 '10 at 20:04

17

If you use RDBMS, you can use its "LOCK" mechanism. For example, while one "SELECT FOR UPDATE" transaction locks a row, the other "SELECT FOR UPDATE" transactions with the row must wait.

# You can use any Python DB API.
[SQL] BEGIN;

[SQL] SELECT col_name FROM table_name where id = 1 FOR UPDATE;

[Process some python code]

[SQL] COMMIT;

answered Nov 30 '10 at 20:04

takaomag

1,545
1
16
26

4

This is really cool and it would work in a distributed system. Nice work. – stinkypyper Dec 01 '10 at 22:09
this is the very definition of genius. – Mike McMahon Sep 23 '15 at 23:05
3

Django has native support for this. See http://stackoverflow.com/a/33480015/78234 – Tal Weiss Nov 02 '15 at 14:35
Note that many DBs also support locks that aren't dependent on a row, so this works in the general case as well. In postgreSQL they call them advisory locks. – DylanYoung Jun 01 '20 at 19:09

score 16 · Answer 2 · answered Nov 02 '15 at 14:33

Use the Django builtin select_for_update function.
https://docs.djangoproject.com/en/1.8/ref/models/querysets/#select-for-update
From the docs:
Returns a queryset that will lock rows until the end of the transaction, generating a SELECT ... FOR UPDATE SQL statement on supported databases.

For example:

entries = Entry.objects.select_for_update().filter(author=request.user)

All matched entries will be locked until the end of the transaction block, meaning that other transactions will be prevented from changing or acquiring locks on them.

Remember that the select_for_update code block must in a transaction. — Venusor, Sep 03 '18 at 09:14

score 9 · Accepted Answer · answered Jul 20 '09 at 05:10

9

You need a distributed lock manager at the point where your app suddenly needs to run on more than one service. I wrote elock for this purpose. There are bigger ones and others have chosen to ignore every suggestion and done the same with memcached.

Please don't use memcached for anything more than light advisory locking. It is designed to forget stuff.

I like to pretend like filesystems don't exist when I'm making web apps. Makes scale better.

answered Jul 20 '09 at 05:10

Dustin

89,080
21
111
133

6

Your bad-ass Erlang written, clean code looking, test case having, distributed file locker is clearly better then my simple non-distributed lock, so shall it be known, so shall it written. One thing though, do you have any example usage(non set-up) of it? – stinkypyper Jul 21 '09 at 16:13
Thanks for the review. :) The only docs I've written so far are linked from the project page. Should lead you here: http://dustin.github.com/elock/admin.html – Dustin Jul 21 '09 at 18:52
Now I am guessing this thing is basically HTTP (I see 200, 409), right? Can I change the post the server binds too? – stinkypyper Jul 23 '09 at 02:38
http semantics wouldn't work here, but it's always possible to change a port number. :) – Dustin Jul 23 '09 at 16:25

score 4 · Answer 4 · answered Jul 14 '09 at 06:35

4

You could use simple file locking as a mutual exclusion mechanism, see my recipe here. It won't suit all scenarios, but then you haven't said much about why you want to use this type of locking.

answered Jul 14 '09 at 06:35

Vinay Sajip

95,872
14
179
191

1

I am doing provisioning of virtual servers. I don't want to queue it because I want the xml-rpc interface exposed through DJango to return whether or not the system has room for a new virtual server. Someone using the interface could loop through and light up ten servers quickly, so the provisioning algorithm needs to be locked as a critical section so I don't get errors in that situation. – stinkypyper Jul 14 '09 at 16:12
Then, as long as requesters don't mind getting locked out and retrying, it should be possible to use the file locking approach. – Vinay Sajip Jul 14 '09 at 16:39

score 3 · Answer 5 · answered Jul 19 '09 at 23:29

I ended up going with a solution I made myself involving file locking. If anyone here ends up using it remember that advisory locks and NFS don't mix well, so keep it local. Also, this is a blocking lock, if you want to mess around with loops and constantly checking back then there is instructions in the code.

import os
import fcntl

class DjangoLock:

    def __init__(self, filename):
        self.filename = filename
        # This will create it if it does not exist already
        self.handle = open(filename, 'w')

    # flock() is a blocking call unless it is bitwise ORed with LOCK_NB to avoid blocking 
    # on lock acquisition.  This blocking is what I use to provide atomicity across forked
    # Django processes since native python locks and semaphores only work at the thread level
    def acquire(self):
        fcntl.flock(self.handle, fcntl.LOCK_EX)

    def release(self):
        fcntl.flock(self.handle, fcntl.LOCK_UN)

    def __del__(self):
        self.handle.close()

Usage:

lock = DJangoLock('/tmp/djangolock.tmp')
lock.acquire()
try:
    pass
finally:
    lock.release()

This: lock.acquire(); print("acquired 1"); lock.acquire() print("acquired 2") prints: acquired 1 acquired 2... It doesn't block the second try. — Etam, Jun 13 '12 at 16:14

score 2 · Answer 6 · answered Feb 24 '11 at 20:33

I did not write this article, but I found it supremely helpful faced with this same problem:

http://chris-lamb.co.uk/2010/06/07/distributing-locking-python-and-redis/

Basically you use a Redis server to create a central server that exposes the locking functionality.

How to lock a critical section in Django?

6 Answers6

Linked