1

Why doesn't tqdm work here when combined with requests? When I run the script below, all I get is a blinking cursor, no progress bar. Thank you.

import requests, re
from tqdm import tqdm

link = "https://web.archive.org/cdx/search/cdx?url=twitter.com/realdonaldtrump/status&matchType=prefix&filter=statuscode:200"
flamingo = []

y = requests.get(link).text
urls = re.findall(r'http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+', y)

for url in tqdm(urls):
    flamingo.append(f"{url}")
petezurich
  • 9,280
  • 9
  • 43
  • 57
facialrecognition
  • 239
  • 2
  • 13

1 Answers1

1

I tried your code and it works fine if you just add a limit argument to your URL.

link = "https://web.archive.org/cdx/search/cdx?url=twitter.com/realdonaldtrump/status&matchType=prefix&filter=statuscode:200&limit=10"

The list of tweets you get without limit is very, very long. I assume that your code doesn't reach your loop before timing out. (This at least was the case when I tested your script)

petezurich
  • 9,280
  • 9
  • 43
  • 57
  • Pete, thanks! Appreciate your help :) You're correct: the list of tweets is really long. But ultimately, it's a file. Just like how Firefox shows me a progress bar for downloading it, I think `requests` combined with `tqdm` should do the same. I think it's a bug. – facialrecognition Feb 05 '22 at 06:49
  • But this has nothing to do with tqdm and requests combination. It just takes long to get this urls list. The donwload is finished in the line wäre you put that file into y – Simon Hawe Feb 05 '22 at 07:10
  • Hi Simon. Do you think `tqdm` can be used to track the time it takes for Python to get the URLs list? Or is that not possible? Thanks a lot. – facialrecognition Feb 05 '22 at 07:23
  • https://stackoverflow.com/questions/37573483/progress-bar-while-download-file-over-http-with-requests – Simon Hawe Feb 05 '22 at 08:00