Atomic increment of a counter in django

Question

I'm trying to atomically increment a simple counter in Django. My code looks like this:

from models import Counter
from django.db import transaction

@transaction.commit_on_success
def increment_counter(name):
    counter = Counter.objects.get_or_create(name = name)[0]
    counter.count += 1
    counter.save()

If I understand Django correctly, this should wrap the function in a transaction and make the increment atomic. But it doesn't work and there is a race condition in the counter update. How can this code be made thread-safe?

To me it looks like such a waste not to use `+=` to avoid race conditions. Python users should already know there is a difference between `a += b` and `a = a + b`, so why not use that? Maybe it'll conflict with some cache data? Not sure. — aliqandil, Apr 19 '17 at 16:04

score 121 · Accepted Answer · edited Apr 26 '21 at 16:20

121

Use an F expression:

from django.db.models import F

either in update():

Counter.objects.get_or_create(name=name)
Counter.objects.filter(name=name).update(count=F("count") + 1)

or on the object instance:

counter, _ = Counter.objects.get_or_create(name=name)
counter.count = F("count") + 1
counter.save(update_fields=["count"])

Remember to specify update_fields, or you might encounter race conditions on other fields of the model.

A note on the race condition avoided by using F expressions has been added to the official documentation.

edited Apr 26 '21 at 16:20

Ellis Percival

4,632
2
24
34

answered Oct 21 '09 at 06:43

Oduvan

2,607
3
24
24

3

should this be wrapped in a commit_on_success method? – alexef Nov 30 '11 at 14:38
8

One issue with this is if you need the updated value afterwards, you need to fetch it from the database. In certain cases, like ID generation, this can cause race conditions. For instance, two threads might increment an ID atomically (say from 1 to 3), but then both query for the current value and get 3, try to insert, explosion... Just something to think about. – Bialecki Apr 02 '12 at 20:23
1

In the second version, why not use the defaults kwarg to get_or_create, and then put the F object inside a `if created` block? Should be faster in the case of creation, right? I went ahead and put an answer demoing what I mean. – mlissner Sep 21 '13 at 20:42
This is definitely the right answer. Check the [django doc about F()](https://docs.djangoproject.com/en/dev/ref/models/queries/#django.db.models.F): _Another benefit of using F() is that having the database - rather than Python - update a field’s value avoids a race condition._ – Han He Feb 12 '14 at 09:54
4

`get_or_create` returns a pair, so it should be `counter, created = ...` like in mlissner's answer. – Alex Hall Dec 11 '16 at 11:11
@Bialecki you could just write `new_count = counter.count + 1` before doing the update. – Alex Hall Dec 11 '16 at 11:12
So, basically after updating a field like that,model will hold an instance of django.db.models.expressions.CombinedExpression, instead of the actual result. If you want to access the result immediately: "counter.refresh_from_db()" – Nandesh Oct 25 '18 at 03:19
Every once in a while this fails for me and I end up with a smaller count than expected. Does every database backend support this? More specifically, does sqlite3 support this? – Whadupapp Feb 07 '19 at 15:32
I'm looking for a way to increment a field value, but on the page I don't want to need to refresh it, to get the new value when I click the increment button. – AnonymousUser Nov 24 '21 at 06:28

score 19 · Answer 2 · edited Feb 11 '20 at 08:24

If you don't need to know the value of the counter when you set it, the top answer is definitely your best bet:

counter, _ = Counter.objects.get_or_create(name = name)
counter.count = F('count') + 1
counter.save()

This tells your database to add 1 to the value of count, which it can do perfectly well without blocking other operations. The drawback is that you have no way of knowing what count you just set. If two threads simultaneously hit this function, they would both see the same value, and would both tell the db to add 1. The db would end up adding 2 as expected, but you won't know which one went first.

If you do care about the count right now, you can use the select_for_update option referenced by Emil Stenstrom. Here's what that looks like:

from models import Counter
from django.db import transaction

@transaction.atomic
def increment_counter(name):
    counter = (Counter.objects
               .select_for_update()
               .get_or_create(name=name)[0]
    counter.count += 1
    counter.save()

This reads the current value and locks matching rows until the end of the transaction. Now only one worker can read at a time. See the docs for more on select_for_update.

This answer has the best explanation. It wasn't till reading this one that I was convinced that `count = F('count') + 1` would work — Aaron McMillin, Nov 20 '17 at 18:49

score 18 · Answer 3 · answered Jan 17 '12 at 21:26

18

In Django 1.4 there is support for SELECT ... FOR UPDATE clauses, using database locks to make sure no data is accesses concurrently by mistake.

answered Jan 17 '12 at 21:26

Emil Stenström

13,329
8
53
75

This was the solution I ended up going with combined with wrapping the block in the transaction.commit_on_success. – Bialecki Apr 02 '12 at 20:24
Exactly. The real problem is the race of the same data by different clients. An internal lock can make sure each client gets its own serial for its transaction. – George Y Apr 15 '21 at 08:51

score 8 · Answer 4 · answered Sep 21 '13 at 20:44

Keeping it simple and building on @Oduvan's answer:

counter, created = Counter.objects.get_or_create(name = name, 
                                                 defaults={'count':1})
if not created:
    counter.count = F('count') +1
    counter.save()

The advantage here is that if the object was created in the first statement, you don't have to do any further updates.

score 7 · Answer 5 · answered Jan 30 '15 at 03:53

7

Django 1.7

from django.db.models import F

counter, created = Counter.objects.get_or_create(name = name)
counter.count = F('count') +1
counter.save()

answered Jan 30 '15 at 03:53

derevo

9,048
2
22
19

score -2 · Answer 6 · answered Aug 26 '10 at 07:33

-2

Or if you just want a counter and not a persistent object you can use itertools counter which is implemented in C. The GIL will provide the safety needed.

--Sai

answered Aug 26 '10 at 07:33

Sai Venkat

1,208
9
16

2

The asker was specifically asking for how to atomically increment a field in the database. – slacy Jan 27 '12 at 18:35
In this guy's defence, that isn't stated explicitly. Clearly intended though. – N. McA. Sep 24 '16 at 11:04

Atomic increment of a counter in django

6 Answers6

Linked