1

I am using selenium with Firefox driver (geckodriver) to get some page source from a list of urls.

I notice that if the Internet is slow, FireFox does not wait until the page is fully loaded (the execution does not wait at line 5). As a result, the page_source in line 9 is actually from the previous url.

How can I make Firefox to wait for the page to fully load?

Selenium: 3.14.1
Geckodriver: 0.23.0 linux64

1        browser = webdriver.Firefox()
2    
3        for url in url_list:
4          
5            browser.get(url)
6    
7            sleep(1)
8    
9            page_source = browser.page_source
10    
11           if  html == page_source:
12    
13                print "error: page not fully loaded"
14    
15                exit(1)
16           html = page_source

Update: I have tested with Chrome driver. Chrome driver does wait until the page is fully loaded. So maybe the issue is with FireFox driver.

Kelvin Tan
  • 982
  • 1
  • 12
  • 33

3 Answers3

1

Usually ".get" in Selenium webdriver is done using an HTTP GET operation, and the method will block until the load is complete. So, I don't think slow internet connection causing the issue but there is a chance for the issue to happen if you have interrupted internet connection.

If the page is same, you can leverage Explicit Wait to wait for any element on the page and if the element does not visible or not loaded then you can again hit ".get" method or you can do a ".refresh()" method as you need.

Apparently, if pages are different then wait for the body tag[ xpath: "//body"] to be loaded using the Explicit wait.

For more details about Explicit wait refer here

Community
  • 1
  • 1
Dhamo
  • 1,171
  • 3
  • 19
  • 41
  • If the connection is good, the method will block until the load is complete. If the connection is disrupted. FireFox driver does not get blocked. I have tested with Chrome. Chrome driver does get blocked until the load is complete. So maybe it is the issue with FireFox driver. – Kelvin Tan Oct 26 '18 at 08:58
1

You have to check that ready state is complete for the loaded page in browser or not, till that you can put wait for driver using, t will throw error so you should put this inside of try catch block. 

driver.get(url)
WebDriverWait(driver, 30).until(readystate_complete)

there are different Ready states,like
loading, complete and interactive
usually in javascript people does document.readystate
and the complete means that the document was fully read and all resources (like images) are loaded too

Ashish Kamble
  • 2,555
  • 3
  • 21
  • 29
0

You can wait using WebDriverWait until particular element is located or in case of network is slowed down and also add implicit wait then get page source

Please find below java code:

WebDriverWait wait = new WebDriverWait(driver, 20);

wait.until(ExpectedConditions.stalenessOf(element));
Ashish Kamble
  • 2,555
  • 3
  • 21
  • 29
Mitul Lakhani
  • 74
  • 5