Scroll a web page using selenium webdriver in python with infinite loading

Question

I'm not able to scroll a website using code from How can I scroll a web page using selenium webdriver in python?

This code only scroll one page. How do I fix it?

SCROLL_PAUSE_TIME = 0.5
# Get scroll height
last_height = driver.execute_script("return document.body.scrollHeight")

while True:
    # Scroll down to bottom
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

    # Wait to load page
    time.sleep(SCROLL_PAUSE_TIME)

    # Calculate new scroll height and compare with last scroll height
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

Answer 1

I've tested your code and it's working on my end. I've used this infinite scroll example page. I consider, you may use scrollTop attributes to double-check your current vertical position. Have you checked if the page you are working with allows you to scroll by calling javascript directly?

document.documentElement.scrollTop || document.body.scrollTop

    import time
    from selenium import webdriver
    
    driver = webdriver.Chrome('chromedriver.exe')  # Optional argument, if not specified will search path.
    try:
        driver.get('https://infinite-scroll.com/demo/full-page/');
        time.sleep(5)  # Let the user actually see something!
        SCROLL_PAUSE_TIME = 1
    # Get scroll height
        last_height = driver.execute_script("return document.body.scrollHeight")
        print(last_height)
        while True:
            # Scroll down to bottom
            driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
            # Wait to load page
            time.sleep(SCROLL_PAUSE_TIME)
            # Calculate new scroll height and compare with last scroll height
            new_height = driver.execute_script("return document.body.scrollHeight")
            if new_height == last_height:
                break
            last_height = new_height
    finally:
        driver.quit()