0

I want to fetch 20000+ records so I tried to code the following python code and its work perfectly. but it's kinda slow. it takes 2-3sec/records so how to speed it up.

import requests
import sys
import time
from tqdm import tqdm

url = "https://www.example.org/web/api/Profile/users"
headers = {
    'Content-Type': "application/x-www-form-urlencoded",
    'Access-Control-Allow-Origin': "*",
    'Accept-Encoding': "gzip, deflate",
    'Accept-Language': "en-US",
    }

n = (int(sys.argv[1]))
sum = 0
for i in tqdm(range(0,n)):
    payload = "user_id={}".format(i+1)
    response = requests.request("POST", url, data=payload, headers=headers)
    print(response.text)
    sum = sum + i
Amit kumar
  • 151
  • 1
  • 4
  • 20
  • 2
    The simplest solution in my opinion is to use aiohttp. – cglacet Mar 20 '19 at 09:08
  • I agree with cglacet. Instead of multithreading, it sounds like you want asynchronous web calls. Currently, your loop has to wait for each response to complete and the sum to run. If you call asynchronously, since order doesn't matter, you can receive more responses in a shorter amount of time. This is likely taking the most time. – Jack Moody Mar 20 '19 at 09:10
  • can you give me an example code? for my above code – Amit kumar Mar 20 '19 at 09:12
  • @JackMoody can you provide me an example code of above... – Amit kumar Mar 20 '19 at 09:21
  • Try `requests-futures` https://github.com/ross/requests-futures – Adrian Krupa Mar 20 '19 at 09:23
  • @Amitkumar this question has been answered in [this SO post](https://stackoverflow.com/questions/2632520/what-is-the-fastest-way-to-send-100-000-http-requests-in-python). – Jack Moody Mar 20 '19 at 09:36
  • @JackMoody and if record order matters? then – Amit kumar Mar 20 '19 at 09:54
  • @Amitkumar if order matters, you need to find a way to run the loop after getting the responses you have such that it is in order. – Jack Moody Mar 20 '19 at 10:19
  • Possible duplicate of [What is the fastest way to send 100,000 HTTP requests in Python?](https://stackoverflow.com/questions/2632520/what-is-the-fastest-way-to-send-100-000-http-requests-in-python) – Jack Moody Mar 20 '19 at 10:24

0 Answers0