Python: running multiple web requests in parallel

Question

I'm new to Python and I have a basic question but I'm struggling to find an answer online, because a lot of the examples online seem to refer to deprecated APIs, so sorry if this has been asked before.

I'm looking for a way to execute multiple (similar) web requests in parallel, and retrieve the result in a list.

The synchronous version I have right now is something like:

urls = ['http://example1.org', 'http://example2.org', '...']

def getResult(urls):
  result = []
  for url in urls:
    result.append(get(url).json())
  return result

I'm looking for the asynchronous equivalent (where all the requests are made in parallel, but I then wait for all of them to be finished before returning the global result).

From what I saw I have to use async/await and aiohttp but the examples seemed way too complicated for the simple task I'm looking for.

Thanks

Does this answer your question? [A very simple multithreading parallel URL fetching (without queue)](https://stackoverflow.com/questions/16181121/a-very-simple-multithreading-parallel-url-fetching-without-queue) — Grismar, Sep 16 '20 at 04:00
Well that's the thing, I don't know. I've seen so many questions/answers like this one that I don't know what's the correct way to do it. This question dates from 2013 and async didn't exist back then. — luke, Sep 16 '20 at 04:08
You indicated you felt using async was complicating matters too much anyway and this solution does not use it - what exactly are you asking? For someone to write the `asyncio` solution for you? — Grismar, Sep 16 '20 at 04:10
Well I didn't say I don't want to use it, just that I didn't manage to find or create a solution that seemed simple, but maybe I'm using it wrong. So if you could provide a solution with async that would certainly be helpful. Also, in the question you gave, there's just a side effect (print) whereas I want to return something. It seems to me it's different, but maybe it's not. — luke, Sep 16 '20 at 04:13
Surely you can take that code and replace the `print` statement with a statement adding the value to a thread-safe store (like a queue?) - again, StackOverflow is here to help answer technical questions, not to write your code for you — Grismar, Sep 16 '20 at 04:15
Also, your question has been answered more generally a dozen times here https://stackoverflow.com/questions/9110593/asynchronous-requests-with-python-requests — Grismar, Sep 16 '20 at 04:17
that's one of the stackoverflow answers that got me struggling. Async is now a reserved keyword and can't be used in an import, so the accepted answer is no longer valid. I'm really looking to an example of how it would work with the async/await method in python3, if you or someone else could show it to me I'd be grateful. Thanks — luke, Sep 16 '20 at 04:20

Mohamed Ahmed Nabil · Answer 1 · 2020-09-16T04:14:49.483

I am going to try to explain the simplest possible way to achieve what you want. Im sure there are more cleaner/better ways to do this but here it goes.

You could preform what you want using the python "threading" library. You can use it to create separate threads for each request and then run all the threads concurrently and get an answer.

Since you are new to python, to simplify things further I am using a global list called RESULTS to store in the results of the get(url) rather than returning them from the function.

import threading

RESULTS=[] #List to store the results

#Request Single Url Result and store in global RESULTS
def getSingleResult(url):
    global RESULTS
    RESULTS.append( ( url, get(url).json()) )

#Your Original Function
def getResult(urls)
    ths=[]
    for url in urls:
        th=threading.Thread(target=getSingleResult, args=(url,)) #Create a Thread
        th.start() #Start it
        ths.append(th) #Add it to a thread list

    for th in ths:
        th.join() #Wait for all threads to finish

The usage of the global results is to make it easier rather than collecting results from the threads directly. If you wish to do that you can check out this answer How to get the return value from a thread in python?

Of course one thing to note that multi-threading in python doesnt provide true parallelism but rather concurrency especially if you are using the standard python implementation due to what is known as the Global Interpreter Lock

However for your use case it would still provide for you the speed up you need.

thanks. Could you please show how it would work without the global variable ? Do you think this solution is better / more elegant / shorter / easier to read than using async/await and aiohttp? — luke, Sep 16 '20 at 04:23

Python: running multiple web requests in parallel

1 Answers1