Send multiple get requests for the same url using python

Question

I have to send a couple of thousand get requests for the same url. It's quite long when I do that by using for loop, so I looking a better/faster/ ̶s̶t̶r̶o̶n̶g̶e̶r̶- solution. Do you have any ideas?

url = 'https://site.ru:port/api/v2/getRequest'

for index, row in tqdm.tqdm(data.iterrows(), total=data.shape[0]):

        params = {

            'param1_key' : row['param1_value'],
            'param2_key' : row['param2_value'}

        response = requests.get(api_endpoint, params, headers={'apikey': api_key}, timeout=30)

Faboor · Accepted Answer · 2020-05-13T14:01:02.790

0

When you do it in a for loop, you have to wait for a response each time. So to make it ~~better/~~faster~~/stronger~~... Do it async with requests-futures or grequests!

(Also see Asynchronous Requests with Python requests - it's pretty dated though and points you to grequests anyway)

edited May 13 '20 at 14:01

answered May 13 '20 at 13:53

Faboor

1,365
2
10
23

@Fabor, thanks so much, grequest works, but I'm confused cause can't find how to pass api key there (grequest.get). Do you know? – G_V May 13 '20 at 16:11
1

@G_V grequests sends all kwargs to requests, so you should be able to do it the same way. `grequests.get("http://localhost:8075", params={"p1": 5}, headers={'apikey': "my_key"}).send()` has worked for me. It results in this HTTP request: `GET /?p1=5 HTTP/1.1 Host: localhost:8075 User-Agent: python-requests/2.23.0 Accept-Encoding: gzip, deflate Accept: */* Connection: keep-alive apikey: my_key` – Faboor May 13 '20 at 18:21

score 0 · Answer 2 · answered May 13 '20 at 13:57

Hard to say without more info. iterrows if knows not to be fast, but I would bet a coin that the most time consuming part is waiting for the response from the server.

In that case multiprocessing.dummy.Pool can be a handy tool to start multiple concurrent requests without waiting for each to terminate before starting the next one. But beware, sending too many requests to a single server can be seen as an attack...

Send multiple get requests for the same url using python

2 Answers2