0

Is there any way to get the total content-length of multiple urls faster?

So this is my code. I'm only calculating 143 urls but it takes 1:32mins

def calculate_remote_file_size(urls: list, desc='Getting remote file size'):
size = 0
with tqdm(total=len(urls), desc=desc, ncols=(100-1), initial=0, miniters=1, bar_format="{l_bar}{bar}{r_bar}") as pbar:
    for url in urls:
        r = requests.head(url, stream=True, headers={'User-Agent': UA}, verify=False)
        size += int(r.headers['content-length'])
        pbar.update()
    return size

this is the result:

Getting remote file size: 100%|██████████████████████████████████| 143/143 [01:32<00:00,  1.54it/s]

I appreciate the advice and help. Thank you

mark12345
  • 233
  • 1
  • 4
  • 10

0 Answers0