Selenium scroll down slowly

Question

I'm trying to do dynamic web scraping on a javascript-rendered webpage using Python.

1) However, the elements only load when I scroll down the page slowly.

I have tried:

driver.execute_script("window.scrollTo(0, Y)")

(this doesn't work because it only scrolls to a certain point on the page, missing out other results)

and

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

(this doesn't work because the elements don't load when scrolled down to the end of the page - it requires the user to scroll through the entire page slowly)

2) How do I make Selenium wait for all my elements to be loaded before returning them to me?

I understand that this solution exists:

myElem = WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.ID, 'IdOfMyElement')))

But how would this work if results are continuously appearing as the user scrolls down the page? Won't this code make Selenium stop once it detects the first occurrence of said element?

how about you scroll a tiny bit many times, waiting between the scrolls ? — , May 07 '19 at 14:29
If you are looking for any specific records and know the number of records you want? If answer is yes, then we can handle this using `location_once_scrolled_into_view` — supputuri, May 07 '19 at 15:14

score 2 · Answer 1 · answered Oct 07 '21 at 17:39

Scroll down to the end of the page slowly using execute_async_script:

driver.execute_async_script(
            """
        count = 400;
        let callback = arguments[arguments.length - 1];
        t = setTimeout(function scrolldown(){
            console.log(count, t);
            window.scrollTo(0, count);
            if(count < (document.body.scrollHeight || document.documentElement.scrollHeight)){
              count+= 400;
              t = setTimeout(scrolldown, 1000);
            }else{
              callback((document.body.scrollHeight || document.documentElement.scrollHeight));
            }
        }, 1000);"""
        )

score 0 · Answer 2 · answered May 07 '19 at 14:36

You can write a function that sends arrow key down till you can locate the element. Preferably in a loop done via some kind of FluentWait (that is a java class) but I have seen this done in Python too: python fluent wait The goal would be to keep sending the arrow key down a set amount of time, while ignoring NoSuchElementException

score 0 · Answer 3 · answered Oct 14 '20 at 23:48

This may help you

    SCROLL_PAUSE_TIME = 0.5 
    # Get scroll height
    last_height = driver.execute_script("return document.body.scrollHeight")

    while True:
        # Scroll down to bottom
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

        # Wait to load page
        time.sleep(SCROLL_PAUSE_TIME)

        # Calculate new scroll height and compare with last scroll height
        new_height = driver.execute_script("return document.body.scrollHeight")
        if new_height == last_height:
            break
        last_height = new_height

You can combine this code with this function :

myElem = WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.ID, 'IdOfMyElement')))

Selenium scroll down slowly

3 Answers3

Linked