0

I am trying to scrape all the data in the website as shown in the image. here For that, I'll need to keep on scrolling down the infinite scroll. But I am unable to access the inner scroll. I use the following code for infinite scrolling

last_height = driver.execute_script("return document.body.scrollHeight")
while True:
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(3)
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

It doesn't work. It scrolls down the main outer scroll instead of the inner one, as shown below. Kindly help me scroll down the inner scroll

enter image description here

babban babu
  • 83
  • 1
  • 10

1 Answers1

3

1 You have to find the scrollable element.

Try some of this suggestions

Find first scrollable parent

2 And then

scrollable_element = driver.find_element... # locate using css or xpath the element, found during point 1 research

driver.execute_script("arguments[0].scroll(0, arguments[0].scrollHeight);", scrollable_element)
Max Daroshchanka
  • 2,698
  • 2
  • 10
  • 14