0

I have built a scraper in python 3.6 using selenium and scrapinghub crawlera. I am trying to fetch this car and download its photos. https://www.cars.com/vehicledetail/detail/800885995/overview/ but the page just keep loading for long periods of time. What I am trying to figure out is how can I stop the browser from continuously loading after 4 mins.

I have tried both explicit and implicit wait and none has worked. 

driver = webdriver.Chrome('/usr/bin/chromedriver', 
                            desired_capabilities=capabilities, 
                            options=chrome_options)
driver.implicitly_wait(180)
driver.get(url)
dcarlo56ave
  • 253
  • 5
  • 18

1 Answers1

2

You need to set the max waiting time for loading with driver.set_page_load_timeout(). In case the page passes its loading time, the browser will throw a TimeoutException. All you need to do is to take care of it

from selenium import webdriver
from selenium.common.exceptions import TimeoutException

driver = webdriver.Chrome('/usr/bin/chromedriver', 
                            desired_capabilities=capabilities, 
                            options=chrome_options)

driver.set_page_load_timeout(time_to_wait)
try:
    driver.get(url)
except TimeoutException:
    # Do what you need here
KobiM1
  • 81
  • 2