Using gevent with pyramid

Question

I'm building a website using pyramid, and I want to fetch some data from other websites. Because there may be 50+ calls of urlopen, I wanted to use gevent to speed things up.

Here's what I've got so far using gevent:

import urllib2    
from gevent import monkey; monkey.patch_all()
from gevent import pool

gpool = gevent.pool.Pool()

def load_page(url):
    response = urllib2.urlopen(url)
    html = response.read()
    response.close()
    return html

def load_pages(urls):
    return gpool.map(load_page, urls)

Running pserve development.ini --reload gives:

NotImplementedError: gevent is only usable from a single thread.

I've read that I need to monkey patch before anything else, but I'm not sure where the right place is for that. Also, is this a pserve-specific issue? Will I need to re-solve this problem when I move to mod_wsgi? Or is there a way to handle this use-case (just urlopen) without gevent? I've seen suggestions for requests but I couldn't find an example of fetching multiple pages in the docs.

Update 1:

I also tried eventlet from this SO question (almost directly copied from this eventlet example):

import eventlet
from eventlet.green import urllib2

def fetch(url):
    return urllib2.urlopen(url).read()

def fetch_multiple(urls):
    pool = eventlet.GreenPool()
    return pool.imap(fetch, urls)

However when I call fetch_multiple, I'm getting TypeError: request() got an unexpected keyword argument 'return_response'

Update 2:

The TypeError from the previous update was likely from earlier attempts to monkeypatch with gevent and not properly restarting pserve. Once I restarted everything, it works properly. Lesson learned.

For your `eventlet` code… where does `get_html` come from? — abarnert, Jan 11 '13 at 22:39
OK, that's what I figured. Anyway, IIRC, what `eventlet` does in this case is take over the current thread to run greenlets on until the pool is finished. I think you're better off dispatching to a single greenlet thread anyway, even if this does work. But if this works, and you want to use it, I guess this might be an advantage of `eventlet` over `gevent` for you. — abarnert, Jan 11 '13 at 23:01
Right, I was hoping it would do that but I'm getting `request() got an unexpected keyword argument 'return_response'`. I tried adding a pool.waitall() in case returning the map was the problem, but no luck. — , Jan 11 '13 at 23:07
Actually, I just tried restarting the whole stack and it was good. Must have been the monkey patching somewhere didn't get cleared when pserve restarted everything, which was keeping around junk. Thanks for the help! — , Jan 11 '13 at 23:17

abarnert · Accepted Answer · 2013-01-11T22:47:54.103

There are multiple ways to do what you want:

Create a dedicated gevent thread, and explicitly dispatch all of your URL-opening jobs to that thread, which will then do the gevented urlopen requests.
Use threads instead of greenlets. Running 50 threads isn't going to tax any modern OS.
Use a thread pool and a queue. There's usually not much advantage to doing 50 downloads at the same time instead of, say, 8 at a time (as your browser probably does).
Use a different async framework instead of gevent, one that doesn't work by magically greenletifying your code.
Use a library that has its own non-magic async support, like pycurl.
Instead of mixing and matching incompatible frameworks, build the server around gevent too, or find some other framework that works for both your web-serving and your web-client needs.

You could simulate the last one without changing frameworks by loading gevent first, and have it monkeypatch your threads, forcing your existing threaded server framework to become a gevent server. But this may not work, or mostly work but occasionally fail, or work but be much slower… Really, using a framework designed to be gevent-friendly (or at least greenlet-friendly) is a much better idea, if that's the way you want to go.

You mentioned that others had recommended requests. The reason you can't find the documentation is that the built-in async code in requests was removed. See, an older version for how it was used. It's now available as a separate library, grequests. However, it works by implicitly wrapping requests with gevent, so it will have exactly the same issues as doing so yourself.

(There are other reasons to use requests instead of urllib2, and if you want to gevent it it's easier to use grequests than to do it yourself.)

I'd taken a look at grequests but as you mentioned, same problem. I also looked at eventlet, but I'm getting an unusual error (edited above). I might try creating a dedicated gevent thread... — , Jan 11 '13 at 20:59
@NumberOverZero: Yeah, I think that's probably the easiest path from where you are today (which is why I put it first). — abarnert, Jan 11 '13 at 22:35

score 2 · Answer 2 · answered Jan 11 '13 at 22:32

I've had similar problems with gevent when trying to deploy a web application. The thing you could do that would take the least hassle is to use a WSGI deployment that runs on gevent; examples include gUnicorn, uWSGI, or one of gevent's built-in WSGI servers. Pyramid should have a way of using an alternate deployment. If large portions of your code rely on gevent, it's easier to just use a server that runs on gevent as well.

So, basically the last bullet on the above answer.

Using gevent with pyramid

Update 1:

Update 2:

2 Answers2