2

I'm using Python, and I'm trying to render web pages with Selenium and download all of their images. The method I've read about is to save the src attribute URL values and then submit another HTTP request, like so:

import urllib.request

url = driver.find_element_by_id("your-image-id").get_attribute("src")
urllib.request.urlretrieve(url, "local-filename.jpg")

But this is really inefficient at scale, since I'd be submitting a ton of extra requests for images I already loaded (I tried benchmarking with this approach, and it was many times slower than Selenium's downloading and rendering, which was surprising)

I couldn't find a way of saving the images directly from Selenium, even though this seems like a pretty obvious feature they'd have. Is there a way of doing this? (I can switch away from Python if I absolutely need to.)

saml
  • 21
  • 2
  • 1
    There is the browser's cache, but that gets complicated. (filenames used are not very helpful and they are temporary files...) – pcalkins Nov 16 '21 at 18:44
  • another option would be to take screenshots of the elements... not ideal because it would limit the resolution to the screen's dpi. – pcalkins Nov 19 '21 at 21:22

1 Answers1

1

The trick a browser uses is to download images asynchronously. Like download them all at once, to save waiting for the webpage to respond.

If you could offload the actual downloading like here Python - Example of urllib2 asynchronous / threaded request using HTTPS it would surely go a LOT quicker.

And as said, reading out the browser cache is tricky. Implementatie might change with the next browser version

Taco Verhagen
  • 222
  • 1
  • 6