How can I do asynchronous programming but hide it in Python?

Question

Am just getting my head round Twisted, threading, stackless, etc. etc. and would appreciate some high level advice.

Suppose I have remote clients 1 and 2, connected via a websocket running in a page on their browsers. Here is the ideal goal:

for cl in (1,2):
    guess[cl] = show(cl, choice("Pick a number:", range(1,11)))
checkpoint()
if guess[1] == guess[2]:
    show((1,2), display("You picked the same number!"))

Ignoring the mechanics of show, choice and display, the point is that I want the show call to be asynchronous. Each client gets shown the choice. The code waits at checkpoint() for all the threads (or whatever) to rejoin.

I would be interested in hearing answers even if they involve hairy things like rewriting the source code. I'd also be interested in less hairy answers which involve compromising a bit on the syntax.

From people writing code using my library, who need simplicity. — , May 12 '15 at 08:30

score 1 · Answer 1 · edited May 23 '17 at 11:58

The most simple solution code wise is to use a framework like Autobahn which support remote procdure calls (RPC). That means you can call some JavaScript in the browser and wait for the result.

If you want to call two clients, you will have to use threads.

You can also do it manually. The approach works along these lines:

You need to pass a callback to show().
show() needs to register the callback with some kind of string ID in a global dict
show() must send this ID to the client
When the client sends the answer, it must include the ID.
The Python handler can then remove the callback from the global dict and invoke it with the answer
The callback needs to collect the results.
When it has enough results (two in your case), it must send status updates to the client.

You can simplify the code using yield but the theory behind is a bit complex to understand: What does the "yield" keyword do in Python? and coroutines

This is helpful, including the hint for Autobahn. What about the question of how to hide it? I would like to make this style of programming simple for client code. — , May 12 '15 at 08:24
See the links at the bottom and Autobahn example code for how to make this "simple". Personally, I prefer the obvious approach over `yield` and coroutines. — Aaron Digulla, May 12 '15 at 09:24

ʇsәɹoɈ · Accepted Answer · 2015-05-13T22:26:22.743

In Python, the most widely-used approach to async/event-based network programming that hides that model from the programmer is probably gevent.

Beware: this kind of trickery works by making tasks yield control implicitly, which encourages the same sorts of surprising bugs that tend to appear when OS threads are involved. Local reasoning about such problems is significantly harder than with explicit yielding, and the convenience of avoiding callbacks might not be worth the trouble introduced by the inherent pitfalls. Perhaps just as important to a library author like yourself: this approach is not pure Python, and would force dependencies and interpreter restrictions on the users of your library.

A lot of discussion about this topic sprouted up (especially between the gevent and twisted camps) while Guido was working on the asyncio library, which was called tulip at the time. He summarized the main issues here.

How can I do asynchronous programming but hide it in Python?

2 Answers2