Can Selenium detect when webpage finishes loading in Python3?

Question

In python I wrote:

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=chrome_options)
driver.get(url)
try:
    WebDriverWait(driver, 10).until(
        lambda driver: driver.execute_script('return document.readyState') == 'complete')
except se.TimeoutException:
    return False
# Start Parsing

Even though I have waited for readyState for some websites when I parse it I see that there is no checkbox. But, If I add time.sleep(5) Before parsing for the same website I get that there is a checkbox.

My question is, how can I have a general solution that works with the majority of websites? I can't just write time.sleep(5) as some websites might need much more and some might finish within 0.001 seconds (which will have bad impact on performance...)

I just want to stimulate a real browser and not to handle anything before the refresh button appears again (which means everything was loaded).

don't check for readystate... that's hacky and you don't really know when that will execute. A get() will, by default, wait for page to load already. It sounds like what you have there is a "lazy load" style site, that initially puts up placeholders and then populates things via JS after load. It's a messy thing to deal with and you need a solution that would work for that particular site. WebDriverWaits with proper/targeted expected conditions should solve it, though some lazy loading sites are trickier than others. — pcalkins, Mar 23 '22 at 22:07

undetected Selenium · Accepted Answer · 2022-03-23T20:18:23.350

Ideally web applications when accessed through get(), returns the control to the WebDriver only when document.readyState equals to complete. So unless the AUT(Application under Test) behaves otherwise, the following line of code is typically an overhead:

WebDriverWait(driver, 10).until(lambda driver: driver.execute_script('return document.readyState') == 'complete')

However, as per your test requirements you can configure the pageLoadStrategy either as:

none
eager
normal

You can find a detailed discussion in What is the correct syntax checking the .readyState of a website in Selenium Python

At this point, it is to be noted that using time.sleep(secs) without any specific condition to achieve defeats the purpose of Automation and should be avoided at any cost.

Solution

The generic approach that would work with all the websites is to induce WebDriverWait as per the prevailing test scenario. As an example:

To wait for the presence of an element you need to invoke the expected_conditions of presence_of_element_located()
To wait for the visibility of an element you need to invoke the expected_conditions of visibility_of_element_located()
To wait for the element to be visible, enabled and interactable such that you can click it you need to invoke the expected_conditions of element_to_be_clickable()

Can Selenium detect when webpage finishes loading in Python3?

1 Answers1

Solution