How do I get Selenium to wait until the page is finished loading before moving on?

Question

I am scraping some data from a webpage and need to wait until all the data is loaded on the page. I want Selenium to wait until all the data is loaded before it moves on. I have seen other posts that say to use something along the lines of

webDriverWait.until(ExpectedConditions.visibilityOfElementLocated( elementLocator)

This method inherently won't work for me because I need to wait until 100's of elements are loaded and it wouldn't be feasible to list out the location of every element. Is there a way to get Selenium to wait until everything on a page is loaded?

I don't think there is a single good/easy solution to this. In my case, our application was when all requests ended, so I used that to determine when I was ready to start. Also, if it is possible, use the element locator on the last generated element. — Word Rearranger, Dec 18 '19 at 16:29
There is no global solution. I got some website that seems to be full loaded, but continue loading after 1-2 seconds. My solutions is sleeps, too many sleeps. Also use functions like you post in question. And algorithm to allow code crash some time allowing self re-launch — Wonka, Dec 18 '19 at 16:30
As long as the delay, wait, and try-except strategies work, go for them. I only turned to the create-and-execute-js-function when these other strategies failed. More specifically they failed when I was running multiple browsers in parallel with the cpu maxing out. — Jortega, Dec 18 '19 at 16:47

score 2 · Answer 1 · answered Dec 18 '19 at 16:37

The best trick is to ask selenium to wait for the element which is the longest to load to be visible. For example, after your loading screen, you have this element on your page:

<div class='visible_long_after_loading_only'>

You can then implement this :

 WebDriverWait(driver, time).until(EC.visibility_of_element_located((By.CLASS_NAME, 'visible_long_after_loading_only')))

Otherwise, if there is a loader on the screen, such as:

<div class='loading_spinner'>

You can ask selenium to wait until this element is not visible anymore:

 WebDriverWait(driver, time).until(EC.invisibility_of_element_located((By.CLASS_NAME, 'loading_spinner')))

score -1 · Answer 2 · answered Dec 18 '19 at 16:24

One strategy is to create and execute a java script function.

driver.execute_script("""

                function myFunction() {
                      console.log("Window loaded")
                }

                if(window.attachEvent) {
            window.attachEvent('onload', myFunction());
        } else {
            if(window.onload) {
                var curronload = window.onload;
                var newonload = function(evt) {
                    curronload(evt);
                    myFunction(evt);
                };
                window.onload = newonload;
            } else {
                window.onload = myFunction();
            }
        }
                """)

This could be chained with a command to scroll to the bottom of the page.

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

score -1 · Answer 3 · answered Dec 18 '19 at 16:38

When you use the .get() method, the driver will wait for the whole page to finish loading by default. There is nothing extra you can do, as far as loading the page is concerned. After the page is loaded, then the next line of code will be executed.

If a problem occurs, you can search for the item that takes the most time to load and wait for it, with this piece of code:

delay = 20
myElem = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.ID, 'elementID')))

How do I get Selenium to wait until the page is finished loading before moving on?

3 Answers3