How to get a new page after selenium times out on indefinitely loading page

Question

I've come across a problematic page that causes Selenium Chrome (selenium version 3.10.0 in python 3, chromedriver Version 2.35.528157) on MacOSX to time out, I think because there is something indefinitely loading on the page. The problem is that after that timeout, all future requests to the driver to .get() a new url also fail with a timeout, even if they worked before. In fact, observing the browser it is never sent to the new url. This, of course, renders the browser useless for further sessions.

How can I "reset" the driver so that I can carry on using it? Or failing that, how can I debug why the .get() command doesn't seem to work after visiting the problematic page. The code and my output are below (problematic page is http://coastalpathogens.wordpress.com/2012/11/25/onezoom/: I'd be interested if other people see the same thing, and with other pages too

from selenium import webdriver
from selenium.common.exceptions import TimeoutException

browser = webdriver.Chrome()
browser.set_page_load_timeout(10)
browser.implicitly_wait(1)

for link in ("http://www.google.com", "http://coastalpathogens.wordpress.com/2012/11/25/onezoom/","http://www.google.com"):
    try:
        print("getting {}".format(link))
        browser.get(link)
        print("done!")
    except TimeoutException:
        print("Timed out")
        continue

result:

getting http://www.google.com
done!
getting http://coastalpathogens.wordpress.com/2012/11/25/onezoom/
Timed out
getting http://www.google.com
Timed out

This seems to be a particular problem if multiple chrome applications are running (if, for example, I run this script twice without calling browser.close() between them. But it still manifests itself without, most of the time, — user2667066, Apr 25 '18 at 21:20
To "reset" the browser, have you tried calling `browser.quit()` followed by `browser = webdriver.Chrome()`? — SiKing, Apr 26 '18 at 00:01
@SIKing: thanks, I can indeed do that, but since in my real code this is being done in a recursive way, the outer code then loses track of the selenium session, which makes it a bit complicated, so I wanted to triage before going down that route. Another option would be to open a new window before .get()ing the pages, then closing the window if it hangs. — user2667066, Apr 26 '18 at 07:39

undetected Selenium · Answer 1 · 2018-04-26T20:46:33.847

As per your question and your own code block I have executed your own code tweaking a few ChromeDriver settings through the chrome.options class as below and it works perfecto :

Code Block :

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.common.exceptions import TimeoutException

options = Options()

options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
browser = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
browser.set_page_load_timeout(10)

for link in ("http://www.google.com", "http://coastalpathogens.wordpress.com/2012/11/25/onezoom/","http://www.google.com"):
    try:
        print("getting {}".format(link))
        browser.get(link)
        print("done!")
    except TimeoutException:
        print("Timed out")
    continue

Console Output :

getting http://www.google.com
done!
getting http://coastalpathogens.wordpress.com/2012/11/25/onezoom/
done!
getting http://www.google.com
done!

Issue at your end and the solution

There are a couple of things which you need to consider as follows :

Until and unless your usecase have a constraint on Page Load Timeout you must not use set_page_load_timeout() as on slow networks, while invoking urls e.g. http://coastalpathogens.wordpress.com/2012/11/25/onezoom/ the Browser Client may require more then 10 seconds (i.e. the configured time through set_page_load_timeout(10)) to send document.readyState equal to "complete" to Selenium.

If your usecase have a dependency on Page Load Timeout, catch the exception and invoke quit() to shutdown gracefully as follows :

from selenium import webdriver

driver = webdriver.Chrome(executable_path=r'C:\path\to\chromedriver.exe')
driver.set_page_load_timeout(2)
try :
    driver.get("https://www.booking.com/hotel/in/the-taj-mahal-palace-tower.html?label=gen173nr-1FCAEoggJCAlhYSDNiBW5vcmVmaGyIAQGYATG4AQbIAQzYAQHoAQH4AQKSAgF5qAID;sid=338ad58d8e83c71e6aa78c67a2996616;dest_id=-2092174;dest_type=city;dist=0;group_adults=2;hip_dst=1;hpos=1;room1=A%2CA;sb_price_type=total;srfid=ccd41231d2f37b82d695970f081412152a59586aX1;srpvid=c71751e539ea01ce;type=total;ucfs=1&#hotelTmpl")
    print("URL successfully Accessed")
    driver.quit()
except :
    print("Page load Timeout Occured. Quiting !!!")
    driver.quit()

Console Output :
```
Page load Timeout Occured. Quiting !!!
```
You can find a detailed discussion on set_page_load_timeout() in How to set the timeout of 'driver.get' for python selenium 3.8.0?
Consider replacing the usage of implicitly_wait() by ExplicitWait. Modern websites uses JavaScript, Ajax Calls and React Native where WebDriverWait will come into play and you can't mix up implicitly_wait() with WebDriverWait().

Thanks a lot for this, especially the options. I realise that 10 seconds may be too little for some pages - and I'm happy about this causing a TimeoutException. What I'm not so happy about is the fact that on my setup, all future pages also result in a TimeoutException. But since you aren't getting this, I suspect it is some weirdness in my chrome version. I don't want to quit() but opening 2 windows and closing the stalled one seems to work in my case, so I'm doing that. — user2667066, Apr 26 '18 at 16:35
Thanks, I'm not sure this is quite "the Answer", because I still get the pathological behaviour (that is, not the initial timeout on a slow loading page, but the following ones on fast loading pages) even when setting the options you recommended: I suspect a chrome/selenium bug. But I'll vote up the answer. — user2667066, Apr 26 '18 at 16:56
This didn't work for me either... I've tried using different chrome driver versions as well.. — Sheshank S., Jul 29 '19 at 22:54

How to get a new page after selenium times out on indefinitely loading page

1 Answers1

Issue at your end and the solution