Faster request sending in python

Question

I want it to send ~25 requests per second. Please help.

import requests
x = 1

for i in range(100):
    requests.get("...")
    print(x)
    x += 1

well you could start multiple subprocess treads or use more programm istances together — Leonardo Scotti, Dec 19 '21 at 19:32
im beginner, that multiple subprocess treads doesnt say anything to me — aurora., Dec 19 '21 at 19:37
Try aiohttp https://stackoverflow.com/questions/51726007/fetching-multiple-urls-with-aiohttp-in-python — jetpack_guy, Dec 19 '21 at 19:39
Are you trying to HTTP GET for the same URL or are you perhaps working through a list of URLs. Also, IMHO, multithreading is ideal for this. Give more detail about the URLs and I'll provide an answer using multithreading. Bear in mind that although you may be able to initiate 25 GETs/second, they won't necessarily complete in that time because it depends initially on DNS performance and subsequently on the servers' capabilities — DarkKnight, Dec 19 '21 at 20:02
Perhaps try this blog post where sync & async as well as multi-threading methods are compared, the author managed to send 100 requests under a second. https://python.plainenglish.io/send-http-requests-as-fast-as-possible-in-python-304134d46604 — UdonN00dle, Dec 19 '21 at 21:06

score 2 · Accepted Answer · answered Dec 19 '21 at 20:22

You can use treads to run multiple requests at the same time. Look at this tutorial for more information about threading in python.

In your case, you can use something like this:

from threading import Thread

# target function
def foo():
    for i in range(4): 
        requests.get("...")

# create 25 new threads
threadList = []
for t in range(25):
    newThread = Thread(target=foo)
    threadList.append(newThread)

# start all threads
for t in threadList:
    t.start()

for t in threadList:
    t.join()

Faster request sending in python

1 Answers1