Using transactions on Django AppEngine

Question

The AppEngine documentation has a few examples of transactions, using the AppEngine native technique to make transacted queries on native objects.

Per the docs at http://www.allbuttonspressed.com/projects/django-nonrel [1], I'd like to use AppEngine transactions to query Django objects. Is this possible?

def txn():
    index = random.randint(0, NUM_SHARDS - 1)
    shard_name = "shard" + str(index)
    counter = SimpleCounterShard.objects.filter(name=shard_name)
    # Make counter if it doesn't exist 
    if not len(counter):
        counter = SimpleCounterShard(name=shard_name)
    counter.count += 1
    counter.put()
db.run_in_transaction(txn)

This currently fails with 'Only ancestor queries are allowed inside a transaction.' I understand this is asking me to do something involving an ancestor class, but I'm not sure what or why.

Any tips?

[1] "You can't use Django's transactions API. If your particular DB supports a special kind of transaction (e.g., run_in_transaction() on App Engine) you have to use the platform-specific functions."

score 1 · Answer 1 · answered Apr 06 '11 at 01:40

1

As the error message suggests, only certain types of query are permissible inside a transaction on App Engine - specifically, ones that apply a .ancestor() filter. Queries such as the one you're attempting to execute can't be executed transactionally.

One option would be to execute the query outside the transaction, and pass the results in. It looks like you're trying to retrieve a specific shard of a sharded counter by name, though, and that ought to be possible without using a query at all, since it ought to be identified by its key name. Since I'm not familiar with Django's model API, though, I can't suggest how you'd do this in Django.

answered Apr 06 '11 at 01:40

Nick Johnson

100,655
16
128
198

@nailer Queries with ancestor filters are fine, because they only operate on a single entity group. – Nick Johnson Apr 07 '11 at 01:52
@nailer No, an ancestor query only returns entities who have a parent entity that's the ancestor you specified. That's independent of entity types. – Nick Johnson Apr 12 '11 at 22:40
@nailer You should read the datastore docs on Entities, Entity Groups, and Transactions: http://code.google.com/appengine/docs/python/datastore/entities.html – Nick Johnson Apr 13 '11 at 22:07
@nailer I'm not a Django expert (or even a Django user) - you'll have to ask the Django-nonrel folks how they exposed App Engine's fetch-by-key capability. You definitely don't need to be using a query here. What is 'woeful' about the App Engine docs? – Nick Johnson Apr 15 '11 at 01:19
@nailer I'm sorry you're unhappy with my answer. The reference I provided to the docs describes what an entity group is, which is exactly what you were asking. As I just said, too, I'm not familiar with django-nonrel. If you want to know how to fetch-by-key in Django-nonrel, you should look up the django-nonrel docs, ask them on the mailing list, or post another question here on SO asking that specific question. – Nick Johnson Apr 18 '11 at 01:02
@Nick: sorry if I sounded frustrated earlier, but I find there's an unfortunate situation where Django AppEngine users get squashed between Google and Django non-rel. I appreciate you were trying to help, but it felt like we were yak-shaving on raw AE on a djangoappengine specific question. – mikemaccana Apr 20 '11 at 11:02
@nailer I don't think there's any 'squashed' about it - the Django Nonrel folks clearly know how it works, since they wrote it. If they're not here to answer, you'll have to seek them out elsewhere. Unfortunately, we (developer relations) can't be experts on every framework people use with App Engine. – Nick Johnson Apr 21 '11 at 00:58
@Nick: 'squashed' is my own personal feelings as an AE developer using Django, while I appreciate your own feelings may be different, you mentioned you don't use Django on appengine yourself so I'm not quite sure how relevant they are. While officially supporting the most popular web framework on AppEngine would, quite obviously, be good for Google in terms of AppEngine adoption - rather than AE docs assuming you're using a Google-specific framework like WebApp - I don't think this is the place for that discussion. – mikemaccana Apr 21 '11 at 09:31

score 1 · Answer 2 · answered Feb 15 '12 at 07:04

Nailer hit it on the head in his answer (sorry for the pun): DjangoAE does not support entity groups. However, there is unofficial support in this enterprising developer's branch of djangoappengine.

https://github.com/django-nonrel/djangoappengine/pull/10

The patch is not yet complete, but I plan on trying it out in the next few weeks and will update here.

score 0 · Answer 3 · answered Feb 13 '15 at 13:26

A more elegant way is available:

from django.db.models import F
Accumulator.objects.filter(pk=1).update(counter=F('counter') + 5)

https://www.allbuttonspressed.com/blog/django/f-objects-and-queryset-update-support-in-djangoappengine

Another example is availabe here: https://www.allbuttonspressed.com/blog/django/2010/01/Sharding-with-Django-on-App-Engine#django-s-advantage

YouTubeVideo.objects.filter(pk=keyname).update(
    views_count=F('views_count')+1)

mikemaccana · Answer 4 · 2011-04-21T09:07:57.783

A Django port of the AppEngine WebApp Sharded Counter example, including transactions on regular Django objects, lives in:

https://bitbucket.org/twanschik/sharded-counters-nonrel. Check out sharded_counters/models.py , which includes a single read/increment/write operation as discussed.

Specifically the @commit_locked decorator can be used to read/write/increment a Django model atomically.

Note however you are limited in the queries you can make inside a transaction: Django nonrel, as of Jan 2011, does not support entity groups, which is the cause of the error above.

Using transactions on Django AppEngine

4 Answers4

Linked