Creating an asynchronous method with Google App Engine's NDB

Question

I want to make sure I got down how to create tasklets and asyncrounous methods. What I have is a method that returns a list. I want it to be called from somewhere, and immediatly allow other calls to be made. So I have this:

future_1 = get_updates_for_user(userKey, aDate)
future_2 = get_updates_for_user(anotherUserKey, aDate)
somelist.extend(future_1)
somelist.extend(future_2)

....    

@ndb.tasklet
def get_updates_for_user(userKey, lastSyncDate):
    noteQuery = ndb.GqlQuery('SELECT * FROM Comments WHERE ANCESTOR IS :1 AND modifiedDate > :2', userKey, lastSyncDate)
    note_list = list()
    qit = noteQuery.iter()
    while (yield qit.has_next_async()):
        note = qit.next()
        noteDic = note.to_dict()
        note_list.append(noteDic)
    raise ndb.Return(note_list)

Is this code doing what I'd expect it to do? Namely, will the two calls run asynchronously? Am I using futures correctly?

Edit: Well after testing, the code does produce the desired results. I'm a newbie to Python - what are some ways to test to see if the methods are running async?

score 4 · Accepted Answer · answered Aug 27 '12 at 01:21

It's pretty hard to verify for yourself that the methods are running concurrently -- you'd have to put copious logging in. Also in the dev appserver it'll be even harder as it doesn't really run RPCs in parallel.

Your code looks okay, it uses yield in the right place.

My only recommendation is to name your function get_updates_for_user_async() -- that matches the convention NDB itself uses and is a hint to the reader of your code that the function returns a Future and should be yielded to get the actual result.

An alternative way to do this is to use the map_async() method on the Query object; it would let you write a callback that just contains the to_dict() call:

@ndb.tasklet
def get_updates_for_user_async(userKey, lastSyncDate):
  noteQuery = ndb.gql('...')
  note_list = yield noteQuery.map_async(lambda note: note.to_dict())
  raise ndb.Return(note_list)

Advanced tip: you can simplify this even more by dropping the @ndb.tasklet decorator and just returning the Future returned by map_async():

def get_updates_for_user_Async(userKey, lastSyncDate):
  noteQuery = ndb.gql('...')
  return noteQuery.map_async(lambda note: note.to_dict())

This is a general slight optimization for async functions that contain only one yield and immediately return the value yielded. (If you don't immediately get this you're in good company, and it runs the risk to be broken by a future maintainer who doesn't either. :-)

So in your last code segment, what exactly does the future returned my map_async() contain (if I were to call get_result() on it)? A list of all the notes in dict format? Where is the list being formed? How does it to know to make a list? — Snowman, Aug 27 '12 at 02:57
map_async() always makes a list; it returns a Future that can be used by map(), which essentially returns [callback(ent) for ent in ]. — Guido van Rossum, Aug 27 '12 at 20:57

Creating an asynchronous method with Google App Engine's NDB

1 Answers1

Linked