0

I have a python code and I want to speed it up using threads but when I try to I get the same lines getting duplicated, is there is any way I could speed it up without getting duplicate lines

code

import requests
import json

f = open("urls.json")
data = json.load(f)
def urls():
    for i in data['urls']:
        r = requests.get("https://" + i)
        print(r.headers)
user19966157
  • 3
  • 1
  • Split the data urls into e.g. ten chunks and let each thread handle one of those chunks. – luk2302 Sep 10 '22 at 17:41
  • [Multiprocessing HTTP get requests in Python](https://stackoverflow.com/questions/27547170/multiprocessing-http-get-requests-in-python) –  Sep 10 '22 at 17:45
  • Thanks for that , can you put a piece of code as an example – user19966157 Sep 10 '22 at 18:17

2 Answers2

0

Make async or threaded calls.

So, you would do something like this:

import aiohttp
import asyncio
import time

start_time = time.time()


async def main():

    async with aiohttp.ClientSession() as session:

        for number in range(1, 151):
            pokemon_url = f'https://pokeapi.co/api/v2/pokemon/{number}'
            async with session.get(pokemon_url) as resp:
                pokemon = await resp.json()
                print(pokemon['name'])

asyncio.run(main())

Could also do multiprocessing as per the comment, but async is better for i/o type tasks.

D.L
  • 4,339
  • 5
  • 22
  • 45
0

You can use ThreadPoolExecutor class from concurrent.futures. It is efficient way according to Thread class.

You can change the max_workers value according to your task

Here is the piece of code:

import requests
from concurrent.futures import ThreadPoolExecutor
import json

with open("urls.json") as f:
    data = json.load(f)

def urls():
    urls = ["https://" + url for url in data['urls']]
    print(urls)

    with ThreadPoolExecutor(max_workers=5) as pool:
        iterator = pool.map(requests.get,urls)

    for response in iterator:
        print(response.headers)
        print("\n")
Veysel Olgun
  • 552
  • 1
  • 3
  • 15
  • You're welcome. By the way, don't forget to close the file if you don't need the file object `f`. Closing the file before program finishes is good practise – Veysel Olgun Sep 11 '22 at 17:18