I am using Python3 and Selenium firefox to submit a form and then get the URL that they then land on. I am doing it like this
inputElement.send_keys(postnumber)
inputElement.submit()
time.sleep(5)
# Get Current URL
current_url = driver.current_url
print ( " URL : %s" % current_url )
This is working most of the time but sometimes the page takes longer than 5 seconds to load and I get the old URL as the new one hasn't loaded yet.
How should I be doing this?
url_changes helper from expected_conditions is exactly for this purpose:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
# some work on current page, code omitted
# save current page url
current_url = driver.current_url
# initiate page transition, e.g.:
input_element.send_keys(post_number)
input_element.submit()
# wait for URL to change with 15 seconds timeout
WebDriverWait(driver, 15).until(EC.url_changes(current_url))
# print new URL
new_url = driver.current_url
print(new_url)
In my code I have created a context manager that does the following:
html element goes stale (which means the page has started to reload)document.readyState to be "complete" (which means the page has finished initial loading)If the page has content that is populated with additional ajax calls, I may add another wait after that for an element that I know doesn't appear immediately after the above four steps.
For a thorough description, see this blog post: How to get Selenium to wait for page load after a click
Try following approach:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait as wait
from selenium.webdriver.support import expected_conditions as EC
title = driver.title
inputElement.send_keys(postnumber)
inputElement.submit()
wait(driver, 15).until_not(EC.title_is(title))
current_url = driver.current_url
print ( " URL : %s" % current_url )
This will allow you to wait up to 15 seconds until page title is changed (in case there are different titles on new and old pages) after form submission to get new URL. If you want to handle element on new page, then you might need to use below code:
inputElement.send_keys(postnumber)
inputElement.submit()
text_of_element_on_new_page = wait(driver, 15).until(EC.presence_of_element_located((By.ID, "some_element_id"))).text
print ( " Text of element is : %s" % text_of_element_on_new_page )
driver.find_element_by__link_text('Next').click()
After click to a link, button to go to a new page, you can either:
wait until some element which not in the old page but in the new one appeard;
WebDriverWait(driver, 600).until(expected_conditions.presence_of_element_located((By.XPATH, '//div[@id="main_message"]//table')))
# or just wait for a second for browser(driver) to change
driver.implicitly_wait(1)
when new page is loading(or loaded), now you can check on its readyState by execute javascript script, which will output the 'complete' message(value) when page is loaded.
def wait_loading():
wait_time = 0
while driver.execute_script('return document.readyState;') != 'complete' and wait_time < 10:
# Scroll down to bottom to load contents, unnecessary for everyone
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
wait_time += 0.1
time.sleep(0.1)
print('Load Complete.')
This idea worded for me in my case and I think it can suit most cases, and it's easy.
from selenium.common.exceptions import StaleElementReferenceException
def wait_for(condition_function):
start_time = time.time()
while time.time() < start_time + 10:
if condition_function:
return True
else:
time.sleep(0.1)
raise Exception(
'Time out, waiting for {}'.format(condition_function.__name__)
)
def click_xpath(xpath):
link = driver.find_element_by_xpath(xpath)
link.click()
def link_staled():
try:
link.find_element_by_id('seccode_cSA')
return False
except StaleElementReferenceException:
return True
wait_for(link_staled())
click_xpath('//button[@name="loginsubmit"]')
And this method is from 'https://blog.codeship.com/get-selenium-to-wait-for-page-load/' (may be shared from somewhere else)
If you want to create a general algorithm that can determine if an arbitrary webpage has loaded, you will be disappointed because it is not possible. Checking for a url change is not enough. The problem is that there is no way for the driver to know about all the elements that will eventually be present or visible on an arbitrary webpage. Some elements can take a long time to load (to become present or visible).
You can see this for yourself:
When you compare the two html files, you'll see that there are elements present in the second file that are not present in the first.
Therefore you will have to address page loading on a case-by-case basis. To determine if a webpage has loaded, first manually determine which element is the last to become present or visible on the page, then hardcode a check for that element.
I ran into this problem when having my driver gather hyperlinks from a webpage after submitting a login form. My program would get part way through the hyperlinks before crashing because the page source would change when a new element suddenly became present or visible. To solve this issue, I had to hardcode a check for the presence of that element before doing anything else.
