Deferred callback not being called using Python requests-threads

Question

I am trying to perform async HTTP requests by using the requests library in Python. I found that the last version of the library does not directly support async requets. To achive it they provide the requests-threads library that makes use of Twisted to handle asynchronicity. I tried modifying the examples provided to use callbacks instead of await/yield, but the callbacks are not being called.

My sample code is:

session = AsyncSession(n=10)

def processResponse(response):
  print(response)

def main():
  a = session.get('https://reqres.in/api/users')
  a.addCallbacks(processResponse, processResponse)
  time.sleep(5)

The requests-threads library: https://github.com/requests/requests-threads

notorious.no · Accepted Answer · 2017-10-02T22:07:18.057

I suspect the callbacks are not called because you aren't running Twisted's eventloop (known as the reactor). Remove your sleep function and replace it with reactor.run().

from twisted.internet import reactor
# ...
def main():
    a = session.get('https://reqres.in/api/users')
    a.addCallbacks(processResponse, processResponse)
    #time.sleep(5)    # never use blocking functions like this w/ Twisted
    reactor.run()

The catch is Twisted's reactor cannot be restarted, so once you stop the event loop (ie. reactor.stop()), an exception will be raised when reactor.run() is executed again. In other words, your script/app will only "run once". To circumvent this issue, I suggest you use crochet. Here's a quick example using a similar example from requests-thread:

import crochet
crochet.setup()
print('setup')

from twisted.internet.defer import inlineCallbacks
from requests_threads import AsyncSession

session = AsyncSession(n=100)

@crochet.run_in_reactor
@inlineCallbacks
def main(reactor):
    responses = []
    for i in range(10):
        responses.append(session.get('http://httpbin.org/get'))

    for response in responses:
        r = yield response
        print(r)

if __name__ == '__main__':
    event = main(None)
    event.wait()

And just as an FYI requests-thread is not for production systems and is subject to significant change (as of Oct 2017). The end goal of this project is to design an awaitable design pattern for requests in the future. If you need production ready concurrent requests, consider grequests or treq.

score 0 · Answer 2 · answered Oct 02 '17 at 21:12

I think the only mistake here is that you forgot to run the reactor/event loop.

The following code works for me:

from twisted.internet import reactor
from requests_threads import AsyncSession

session = AsyncSession(n=10)


def processResponse(response):
    print(response)


a = session.get('https://reqres.in/api/users')
a.addCallbacks(processResponse, processResponse)
reactor.run()

Deferred callback not being called using Python requests-threads

2 Answers2