How to send out two request at the same time with python

Question

So I was following a guide at http://tavendo.com/blog/post/going-asynchronous-from-flask-to-twisted-klein/ to create an asynchronous web service.

in my code, I had a function that will send out the request like

def query(text):
    resp = yield treq.get("http://api.Iwanttoquery")
    content = yield treq.content(resp)
    returnValue(content)

@inlineCallbacks
def caller():
    output1 = yield query("one")
    output2 = yield query("two")

Since each query to the api usually take about 3 seconds, with my current code the result comes back after 6 seconds. I wonder is there a way to send out two queries at the same time so after 3 seconds I can get the content of both output1 and output2? Thanks.

you could use threads or find a way to merge the queries into one query — depperm, Jul 14 '16 at 14:31
Your "issue" is that you're using ``inlineCallbacks`` which (for lack of better words) acts like synchronous code. So basically ``output1`` has to yield a value before it can move on to ``output2``. — notorious.no, Jul 14 '16 at 22:22

notorious.no · Accepted Answer · 2016-07-15T13:18:36.027

What you need to do is use a DeferredList instead of inlineCallbacks. Basically you provide a list of deferreds and after each one completes, a final callback with the results of all the deferreds is executed.

import treq
from twisted.internet import defer, reactor

def query(text):
    get = treq.get('http://google.com')
    get.addCallback(treq.content)
    return get

output1 = query('one')
output2 = query('two')

final = defer.DeferredList([output1, output2])  # wait for both queries to finish
final.addCallback(print)    # print the results from all the queries in the list

reactor.run()

Each query() function will execute requests concurrently then return a Deferred. This happens almost immediately, so basically output1 and output2 are executing at the same time. Then you append the deferreds (ie. output1 and output2) inside a list and pass it in DeferredList, which itself returns a Deferred. Finally, you add a callback to the DeferredList to do something with the results (in this case I just print them). This is all done without the use of threads, which is the best part in my opinion! Hope this makes sense and please comment if it doesn't.

PS

If you need further help with Klein, I'm working on revamping the documentation here https://github.com/notoriousno/klein-basics (hopefully I'll make a blog post one of these days). Please take a look at some of the docs (the files with .rst). My shameless plug is now concluded :D

How to send out two request at the same time with python

1 Answers1

PS