Any ready solution for basic asynchronous (non-blocking) HTTP clients with Stackless Python 3.1?

Question

UPDATE: after much laboring with Py3, including writing my own asynchronous webserver (following a presentation given by Dave Beazley), i finally dumped Python (and a huge stack of my code )-: in favor of CoffeeScript running on NodeJS. Check it out: GitHub (where you'll find like 95% of all interesting code these days), npm (package manager that couldn't be any user friendly; good riddance, easy_install, you never lived up to your name), an insanely huge repository of modules (with tons of new stuff being published virtually 24/7), a huge and vibrant community, out-of-the-box asynchronous HTTP and filehandling..., all that (thanks to V8) at one third the speed of light — what's not to like? read more propaganda: "The future of Scripting" (slide hosting courtesy SpreeWebdesign).

I am looking for a way to serve HTTP (and do HTTP requests) in an asynchronous, non-blocking fashion. This seems to be hard to do when you’ve decided on Stackless Python 3.1 (also see here for docs) as i did.

There are some basic examples, like the pretty informative and detailed article How To Use Linux epoll with Python, and there is a a Google code project named stacklessexamples which contains some valuable information (but no Python 3.x compatible code).

So, after many days of doing research on the web and trying to put together the pieces i’ve found so far: does anyone know of a fairly usable asynchronous HTTP library? It doesn’t have to be WSGI-compliant (I am not interested in that).

The server part should be able to serve multiple non-blocking HTTP requests (and possibly do the basics of HTTP header processing); the HTTP client part should be able to retrieve, in a non-blocking way, web content via HTTP requests (also doing basic header processing, but no fancy stuff like authorization or so).

My research so far has shown me that non-blocking HTTP

1. is the only way that makes sense in a stackless, cooperatively scheduled environment;

2. is feasible in Stackless Python 3 by virtue of the standard library’s select epoll (introduced in Py2.6; some solutions prefer libevent, but that means another hurdle as the pyevent project seems to have stopped developing at Py2.5);

3. is sadly still not a household item, with most people relying on blocking HTTP.

The way it looks like now, i would have to learn the basics of socket programming and roll my own HTTP server/client library. I still shy away from that task as i have very little background in that area and am bound to ‘repeat history’ that way.

I would be very happy about any relevant pointers. I prefer very much solutions that make use of select.epoll; i seem to remember it is much more scalable that the older asyncore (but maybe someone has more precise data on this). As a minimum requirement, solutions should run on Ubuntu 9.10.

Answer 1

I know this is like resurrecting the dead (and flow has probably long since solved his problem), but for completeness stackless is available for 3.1.3:

• http://www.stackless.com/download

For information on implementing a HTTP server using stacklesssocket:

• http://code.google.com/p/stacklessexamples/wiki/StacklessNetworking

Answer 2

Non blocking HTTP case is very well handled with twisted, what is does is creating a series of callbacks, and registering those callbacks with deferred. Twisted documentation is worth checking out. Stackless uses microthreads but twisted is coding the entire web framework using fragment by fragment non bloking code chained with callbacks, errbacks and deferreds running is a main reactor loop over a single thread. Think this should the Async HTTP thing better.