1


I'm using selenium to get the html for this site:
http://timesofindia.indiatimes.com/world/us

I'm using selenium because this site only gives you all the html if you scroll down. However when I run this code: 

 # Open the Driver
 driver = webdriver.Chrome(chromedriver)

#create a list to store the htmls
master_lst = []

#looping through the times of india urls
for url in urls[:3]:

#make the driver the the url
driver.get(url)
#this is to scroll down twelve time
for i in range(12):
    # wait 5 seconds
    time.sleep(5)
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

html_source = driver.page_source
data = Beautifulsoup(html_source, 'lxml')
master_lst.append(data)

I get this error:
TimeoutException: Message: timeout: Timed out receiving message from renderer: -0.004

I've tried to change the sleep times and the times I scroll down to no avail.
I've seen similar question in here but none that address this kind of problem.
Let me know what you all think!
Thanks!

Fernando Hidalgo
  • 59
  • 7

1 Answers1

1

You may need to adjust the script timeout:

driver.set_script_timeout(10)
alecxe
  • 462,703
  • 120
  • 1,088
  • 1,195
  • Thanks you!!! Just attaching another useful link https://stackoverflow.com/questions/48941260/what-does-selenium-set-script-timeoutn-do-and-how-is-it-different-from-driver – rudolfe Mar 14 '20 at 21:24