Is there any way to get the total content-length of multiple urls faster?
So this is my code. I'm only calculating 143 urls but it takes 1:32mins
def calculate_remote_file_size(urls: list, desc='Getting remote file size'):
size = 0
with tqdm(total=len(urls), desc=desc, ncols=(100-1), initial=0, miniters=1, bar_format="{l_bar}{bar}{r_bar}") as pbar:
for url in urls:
r = requests.head(url, stream=True, headers={'User-Agent': UA}, verify=False)
size += int(r.headers['content-length'])
pbar.update()
return size
this is the result:
Getting remote file size: 100%|██████████████████████████████████| 143/143 [01:32<00:00, 1.54it/s]
I appreciate the advice and help. Thank you