0

First of all i know i can use threading to accomplish such task, like so:

import Queue
import threading


# called by each thread
def do_stuff(q, arg):
    result = heavy_operation(arg)
    q.put(result)

operations = range(1, 10)

q = Queue.Queue()

for op in operations:
    t = threading.Thread(target=do_stuff, args = (q,op))
    t.daemon = True
    t.start()

s = q.get()
print s

However, in google app engine there's something called ndb tasklets and according to their documentation you can execute code in parallel using them.

Tasklets are a way to write concurrently running functions without threads; tasklets are executed by an event loop and can suspend themselves blocking for I/O or some other operation using a yield statement. The notion of a blocking operation is abstracted into the Future class, but a tasklet may also yield an RPC in order to wait for that RPC to complete.

Is it possible to accomplish something like the example with threading above?

I already know how to handle retrieving entities using get_async() (got it from their examples at doc page) but its very unclear to me when it comes to parallel code execution.

Thanks.

vertazzar
  • 1,053
  • 7
  • 10
  • Note that threads are only true parallel if they release the GIL, so it has to be called on a compiled extension with nogil, I/O, etc. – Davidmh Jun 19 '14 at 10:09
  • i saw this presentation http://www.dabeaz.com/python/GIL.pdf and it obviously states what you get with it, however maybe in my case i could get better performance... but it still doesn't answer my question. – vertazzar Jun 19 '14 at 16:55
  • If you are trying to.use threads in front facing requests, they can't run longer than the request life time. In addition if you are not utilising any google service (rpc) you will not get any any real concurrency as you have a single core. My tests in the past show front end threads with no irpc's just make things slower. – Tim Hoffman Jun 20 '14 at 02:17

1 Answers1

0

The answer depended on what your heavy_operation really is. If the heavy_operation use RPC (Remote Procedure Call, such as datastore access, UrlFetch, ... etc), then the answer is yes.

In how to understand appengine ndb.tasklet? I asked a similar question, you may find more details there.

May I put any kind of code inside a function and decorate it as ndb.tasklet? Then used it as async function later. Or it must be appengine RPC?

The Answer

Technically yes, but it will not run asynchronously. When you decorate a non-yielding function with @tasklet, its Future's value is computed and set when you call that function. That is, it runs through the entire function when you call it. If you want to achieve asynchronous operation, you must yield on something that does asynchronous work. Generally in GAE it will work its way down to an RPC call.

Community
  • 1
  • 1
lucemia
  • 6,349
  • 5
  • 42
  • 75
  • I read that topic a while ago, and did not understand wether its going to do them asynchronously or not. As for my code, no it will not perform any RPC calls, so I guess it cant work asynchronously – vertazzar Jun 19 '14 at 19:21