How to make asynchronous HTTP GET requests in Python and pass response object to a function

Question

Update: Problem was incomplete documentation, event dispatcher passing kwargs to the hook function.

I have a list of about 30k URLs that I want to check for various strings. I have a working version of this script using Requests & BeautifulSoup, but it doesn't use threading or asynchronous requests so it's incredibly slow.

Ultimately what I would like to do is cache the html for each URL so I can run multiple checks without making redundant HTTP requests to each site. If I have a function that will store the html, what's the best way to asynchronously send the HTTP GET requests and then pass the response objects?

I've been trying to use Grequests (as described here) and the "hooks" parameter, but I'm getting errors and the documentation doesn't go very in-depth. So I'm hoping someone with more experience can shed some light.

Here's a simplified example of what I'm trying to accomplish:

import grequests

urls = ['http://www.google.com/finance','http://finance.yahoo.com/','http://www.bloomberg.com/']

def print_url(r):
    print r.url

def async(url_list):
    sites = []
    for u in url_list:
        rs = grequests.get(u, hooks=dict(response=print_url))
        sites.append(rs)
    return grequests.map(sites)

print async(urls)

And it produces the following TypeError:

TypeError: print_url() got an unexpected keyword argument 'verify'
<Greenlet at 0x32803d8L: <bound method AsyncRequest.send of <grequests.AsyncRequest object at 0x00000000028D2160>>
(stream=False)> failed with TypeError

Not sure why it's sending 'verify' as a keyword argument by default; it would be great to get something working though, so if anyone has any suggestions (using grequests or otherwise) please share :)

Thanks in advance.

Answer 1

I tried your code and could get it work by adding an additional parameter kwargs to your print_url function.

def print_url(r, **kwargs):
    print r.url

I figured what was wrong in this other stackoverlow question: Problems with hooks using Requests Python package.

It seems when you use the response hook in grequests you need to add **kwargs in your callback definition.