How to scroll down in Python Selenium step by step

Question

Hi guys I am new to Selenium and Python. I was just scraping the site pagalguy website. I know how to scroll down to the bottom of the page but what I need is to scroll down step by step so that the Selenium will click all the readmore buttons,but I don't know how to scroll down step by step like that so I hard coded it like following one

browser.execute_script("window.scrollTo(0,300);")
browser.find_element_by_link_text("Read More...").click()

browser.execute_script("window.scrollTo(300,600);")
browser.find_element_by_link_text("Read More...").click()

browser.execute_script("window.scrollTo(600,900);")
browser.find_element_by_link_text("Read More...").click()

browser.execute_script("window.scrollTo(900,1200);")
browser.find_element_by_link_text("Read More...").click()

browser.execute_script("window.scrollTo(1200,1500);")
browser.find_element_by_link_text("Read More...").click()

browser.execute_script("window.scrollTo(1500,1800);")
browser.find_element_by_link_text("Read More...").click()

browser.execute_script("window.scrollTo(1800,2100);")
browser.find_element_by_link_text("Read More...").click()

browser.execute_script("window.scrollTo(2100,2500);")
browser.find_element_by_link_text("Read More...").click()
it goes on .......

I tried automating it using a while loop but it resulted in error, the above one works but I want it short and looped so that I can use it for all the other pages with different page length.

initial_value = 0
next_value = 300
while next_value<300000: 
  browser.execute_script("window.scrollTo(initial_value,next_value);")
  browser.find_element_by_link_text("Read More...").click()
  initial_value=next_value
  next_value+=300

JavascriptException: Message: ReferenceError: initial_value is not defined

But I have defined the value, I think I explained what I am actually trying to do, I want to automatically scroll down and click all the readmore buttons and then I will get the full text content

You can look into this answer [link](https://stackoverflow.com/questions/41744368/scrolling-to-element-using-webdriver) — Ankur Singh, Dec 28 '17 at 10:57
maybe just try to inject some Js like in [this link](https://stackoverflow.com/a/45194334/8179099) — Moshe Slavin, Dec 28 '17 at 11:20
@AnkurSingh I tried all of options available in that answer it is scrolling to the page location but it is not clicking and there is no error it executed successfully but thank you for pointing it out though — venkat, Dec 28 '17 at 11:50
@MosheSlavin as i said i am new i dont no angularjs and i refereed to the link that you gave but it didn't help Thank you — venkat, Dec 28 '17 at 11:52

score 9 · Accepted Answer · answered Dec 28 '17 at 12:18

9

Agree with answer of @Rahul Chawla.

But adding one change. You can try this one

driver = webdriver.Chrome()

read_mores = driver.find_elements_by_xpath('//a[text()="Read More..."]')

for read_more in read_mores:
    driver.execute_script("arguments[0].scrollIntoView();", read_more)
    driver.execute_script("$(arguments[0]).click();", read_more)

answered Dec 28 '17 at 12:18

Chanda Korat

2,453
2
19
23

Thanks a lot it worked but i have a doubt why is there a $ in the script driver.execute_script("$(arguments[0]).click();", read_more) it throwed an error then i removed it and now it works – venkat Dec 28 '17 at 12:33
It's all about javascript coding conventions. You can refer [this](https://stackoverflow.com/questions/846585/can-someone-explain-the-dollar-sign-in-javascript) – Chanda Korat Dec 28 '17 at 12:36

score 2 · Answer 2 · answered Dec 28 '17 at 11:52

2

We can do this by finding all read more buttons using find_elements_by_xpath() and looping over them while scrolling them into view one by one.

driver = webdriver.Chrome()

read_mores = driver.find_elements_by_xpath('//a[text()="Read More..."]')

for read_more in read_mores:
    driver.execute_script("arguments[0].scrollIntoView();", read_more)
    read_more.click()
    # your code here

answered Dec 28 '17 at 11:52

Rahul Chawla

1,048
10
15

Thank you for the answer, it is scrolling down to all the readmore clickables but it is not clicking i dont no the reason and there is no error i even added time.sleep(2) between the scrollintoview and click() still it is not cliscking can you give any other suggestions – venkat Dec 28 '17 at 12:06

Mindaugas Vaitkus · Answer 3 · 2019-03-12T04:32:30.193

2

Use loop with javascript window.scrollBy(0, Y) method with coordinates selecting movement step and iterations number.

   for i in range(20): # adjust integer value for need
       # you can change right side number for scroll convenience or destination 
       driver.execute_script("window.scrollBy(0, 250)")
       # you can change time integer to float or remove
       time.sleep(1)

edited Mar 12 '19 at 04:32

answered Mar 12 '19 at 04:17

Mindaugas Vaitkus

151
1
2
11

score -1 · Answer 4 · edited Jul 19 '18 at 10:37

Try like this:

while next_value<300000:
    driver.execute_script("window.scrollTo({},{});".format(initial_value, next_value))
    browser.find_element_by_link_text("Read More...").click()
    initial_value=next_value
    next_value+=300

Basically I just changed this line: browser.execute_script("window.scrollTo(initial_value,next_value);")

for this one:

driver.execute_script("window.scrollTo({},{});".format(initial_value, next_value))

How to scroll down in Python Selenium step by step

4 Answers4

Linked