Infinite scrolling of webpage using Selenium Python

Question

I want to scroll till all the element is loaded. My code works well for starting till 7 8 pages using this code.

driver.get('https://www.bigbasket.com/ps/?q=rice')
last_height = driver.execute_script("return document.body.scrollHeight")

while True:
    # Scroll down to bottom
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    # Wait to load page
    WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//div[@class="show-more"]/button')))
    driver.find_element_by_xpath('//div[@class="show-more"]/button').click()
    # Calculate new scroll height and compare with last scroll height
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

The last height became equal to new height even though not all the element is loaded. I need to scroll till all element is loaded.

score 0 · Accepted Answer · answered Jun 14 '23 at 21:26

To scroll down till the bottom of the page and keep on clicking the Show More button you can use the following locator strategy:

Code block:

driver.get("https://www.bigbasket.com/ps/?q=rice")
while True:
  try:
      driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
      print("Browser scrolled")
      driver.execute_script("arguments[0].click();", WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//button[text()='Show More']"))))
      print("Show More button clicked")
      time.sleep(3)
  except WebDriverException:
      print("No more scrolling")
      break

Console Output:

Browser scrolled
Show More button clicked
Browser scrolled
Show More button clicked
Browser scrolled
Show More button clicked
...
...

Thanks for the solution. It worked well. – Salman Ansari Jun 15 '23 at 07:56 — Salman Ansari, Jun 15 '23 at 07:56

Infinite scrolling of webpage using Selenium Python

1 Answers1