5

This is the first time I've tried to use a library with less-than-ideal levels of documentation and example code, so bear with me. I have a tiny bit of experience with the Requests library, but I need to send separate requests to a specific address every second:

  • Without waiting for the first request to complete, handling the individual responses as they come in
  • The responses' content need to be parsed separately
  • While limiting the total number of connections

I can't figure out how to satisfy these conditions simultaneously. grequests.map() will give me the responses' content that I want, but only in a batch after they've all completed. grequests.send() seems to only return a response object that doesn't contain the html text of the web page. (I may be wrong about grequests.send(), but I haven't yet found an example that pulls content from that object)

Here's the code that I have so far:

import grequests
from time import sleep

def print_res(res, **kwargs):
    print res
    print kwargs

headers =   {'User-Agent':'Python'}

req = grequests.get('http://stackoverflow.com', headers=headers, hooks=dict(response=print_res), verify=False)

for i in range(3):
    job = grequests.send(req, grequests.Pool(10))
    sleep(1)

The response I get:

1
<Response [200]>
{'verify': False, 'cert': None, 'proxies': {'http': 'http://127.0.0.1:8888', 'ht
tps': 'https://127.0.0.1:8888'}, 'stream': False, 'timeout': None}
2
<Response [200]>
{'verify': False, 'cert': None, 'proxies': {'http': 'http://127.0.0.1:8888', 'ht
tps': 'https://127.0.0.1:8888'}, 'stream': False, 'timeout': None}
3
<Response [200]>
{'verify': False, 'cert': None, 'proxies': {'http': 'http://127.0.0.1:8888', 'ht
tps': 'https://127.0.0.1:8888'}, 'stream': False, 'timeout': None}

I've tried accessing the html response with req.content, and job.content, but neither work.

Neodied
  • 119
  • 1
  • 8

1 Answers1

5

Of course, while writing up this question I realized that I hadn't tried to access res.content, which turns out to be exactly what I needed.

Lesson learned: The object that is returned to the response hook in the grequests.get() statement has a content attribute which contains the text of the response sent from the server.

Neodied
  • 119
  • 1
  • 8