Python loop in a coroutine

Question

I've read all the documentation on the subject, but it seems I can't grasp the whole concept of Python coroutines well enough to implement what I want to do.

I have a background task (which generates some random files, but that doesn't much matter), and it does this in an infinite loop (this is a watcher).

I would like to implement this background task in the most efficient way possible, and I thought that microthreads (aka coroutines) were a good way to achieve that, but I can't get it to work at all (either it the background task runs or either the rest of the program, but not both at the same time!).

Could someone give me a simple example of a background task implemented using coroutines? Or am I being mistaken in thinking that coroutines could be used for that purpose?

I am using Python 2.7 native coroutines.

I am well versed into concurrency, particularly with DBMSes and Ada, so I know a lot about the underlying principles, but I'm not used to the generator-as-coroutines concept which is very new to me.

/EDIT: here is a sample of my code, which I must emphasize again is not working:

@coroutine
def someroutine():
    with open('test.txt', 'a') as f:
        f.write('A')
    while True:
        pass
    yield 0

@coroutine
def spawnCoroutine():
    result = yield someroutine()

    yield result

routine = spawnCoroutine()
print 'I am working in parallel!'

# Save 'A' in the file test.txt, but does not output 'I am working in parallel!'

Note: @coroutine is a decorator from coroutine.py provided by David Beazley

/FINAL EDIT AND SOLUTION RECAP

Ok my question was closed because it was seemingly ambiguous, which as a matter of fact is the very purpose of my question: to clarify the usage of Coroutines over Threading and Multiprocessing.

Luckily, a nice answer was submitted before the dreadly sanction occurred!

To emphasize the answer to the above question: no, Python's coroutines (nor bluelet/greenlet) can't be used to run an independent, potentially infinite CPU-bound task, because there is no parallelism with coroutines.

This is what confused me the most. Indeed, parallelism is a subset of concurrency, and thus it is rather confusing that the current implementation of coroutines in Python allow for concurrent tasks, but not for parallel tasks! This behaviour is to be clearly differentiated with the Tasks concept of concurrent programming languages such as Ada.

Also, Python's Threads are similar to coroutines in the fact that they generally switch context when waiting for I/O, and thus are also not a good candidate for independent CPU-bound tasks (see David Beazley's Understanding the GIL).

The solution I'm currently using is to spawn subprocesses with the multiprocessing module. Spawning background processes is heavy, but it's better than running nothing at all. This also has the advantage to allow for distributing computation.

As an alternative, on Google App Engine, there are the deferred module and the background_thread module which can offer interesting alternatives to multiprocessing (for example by using some of the libraries that implement the Google App Engine API like typhoonae, although I'm not sure they have yet implemented these modules).

Answer 1

If you look at the (trivial) coroutine.py library you're using, it includes an example that shows how grep works "in the background". There are two differences between your code and the example:

1. grep repeatedly yields while doing its work—in fact, it yields once per line. You have to do this, or nobody but your coroutine gets a chance to run until it's finished.

2. the main code repeatedly calls send on the grep coroutine, again once per line. You have to do this, or your coroutines never get called.

This is about as trivial a case as possible—a single coroutine, and a trivial dispatcher that just unconditionally drives that one coroutine.

Here's how you could translate your example into something that works:

@coroutine
def someroutine():
    with open('test.txt', 'a') as f:
        yield
        f.write('A')
    while True:
        yield
    yield 0

routine = someroutine()
print 'I am working in parallel!'
routine.send()
print 'But only cooperatively...'
routine.send()

And so on.

But normally you don't want to do this. In the case of the grep example, the coroutine and the main driver are explicitly cooperating as a consumer and producer, so that direct coupling makes perfect sense. You just have some completely independent tasks that you want to schedule independently.

To do that, don't try to build threading yourself. If you want cooperative threading, use an off-the-shelf dispatcher/scheduler, and the only change you have to make to all of your tasks is to put in yield calls often enough to share time effectively.

If you don't even care about the threading being cooperative, just use threading or multiprocessing, and you don't even need the yields:

def someroutine():
    with open('test.txt', 'a') as f:
        f.write('A')
    while True:
        pass
    return 0

routine = threading.Thread(someroutine)
print 'I am working in parallel!'

PS, as I said in one of the comments, if you haven't worked through http://www.dabeaz.com/coroutines/index.html or an equivalent, you really should do that, and come back with any questions you find along the way, instead of writing code that you don't understand and asking why it doesn't work. I'm willing to bet that if you make it to part 4 (probably even earlier), you'll see why your initial question was silly.

Answer 2

while True: pass

ENDLESS LOOP.

So it will not execute yeld after that. In fact its real end of function, everything after that is pure useless decoration.

And since someroutine get STUCK before it can yeld (pun intended ;) ), yeld someroutine() will also not yeld.

So you get your script busily doing nothing. (infinite empty loop).