Scrolling to the next element in Selenium (Python)

Question

I'm trying to make a facebook commenting bot in Selenium in Python. I need to scroll down to the button for uploading photos in comments to make the comment and I have this line of code:

ActionChains(driver).move_to_element(driver.find_element_by_id('my-id')).perform()

When I run it again it scrolls to the same button and I need to scroll to the next one. There's also another issue: this button doesn't have an id. Is there any way to do it?

Zvjezdan Veselinovic · Answer 1 · 2020-10-14T14:24:46.333

So, in order to scroll to multiple elements, you would need to use the driver.execute_script() method. If you have your driver and you have your page opened, then you need to isolate the element that you want to navigate to.

element = driver.find_element(By.XPATH, "Whatever xpath you want")
driver.execute_script("return arguments[0].scrollIntoView();", element)

Once you have this, you will need to get a count of all elements that you want to scroll through. For example, in the Top 100 Greatest Movies List website, I created a method to get the current count of displayed movies

def get_movie_count(self):
        return self.driver.find_elements(By.XPATH, "//div[@class='lister-list']//div[contains(@class, 'mode-detail')]").__len__()

After that, I created another method for scrolling to my element

def scroll_to_element(self, xpath : str):
        element = self.driver.find_element(By.XPATH, xpath)
        self.driver.execute_script("return arguments[0].scrollIntoView();", element)

Once that was done, I created a for-loop and scrolled to each element

for row in range(movieCount):
    driver.scroll_to_element("//div[@class='lister-list']//div[contains(@class, 'mode-detail')][{0}]".format(row + 1))

UPDATE: Added photo to help

Photo Explanation
In this photo, we have our WebDriver class. This is the class that creates our Chrome WebDriver and includes any options that we want. In my case, I did not want to see the "This is being controlled by automation" alert all the time.

In the WebDriver class, I have a method called wait_displayed that has selenium check the browser's DOM for a specific element using xpath. If the element exists in the browser's DOM, all is good. Otherwise, python will raise an Exception and let me know that selenium could not find my element. In this class, I included the scroll_to_element method which allows my chrome driver to find an element on the page and executes JavaScript code to scroll to the element that was found.

In the SampleSite class, we inherit the WebDriver class. This means that my SampleSite class can use the methods that display in my WebDriver class. ( like wait_displayed and scroll_to_element ) In the SampleSite class, I provide an optional parameter in it's constructor method. ( __init__() ) This optional parameter tells the programmer that, if they want to create an instance or object of the SampleSite class, they need to, either, pass in an existing chrome driver; or, pass in None and the SampleSite class will open Google Chrome and navigate to our Top 100 Greatest Movies Page. This is the class where I created the def get_movie_count(self): method.

In our main_program.py file, I imported our SampleSite class. After import, I created an instance of our class and called the get_movie_count method. Once that method returned my movie count ( 100 ), I created a for-loop that scrolls through each of my elements.

also i'm getting an error saying that "Instance of 'WebDriver' has no 'scroll_to_element' member" — wojwesoly, Oct 14 '20 at 06:36
The `def get_movie_count(self):` method should be located in your Page Object. The `def scroll_to_element(self, xpath : str):` method should be located in your WebDriver Object. The `for-loop` should be located inside of your `main_program.py` file — Zvjezdan Veselinovic, Oct 14 '20 at 11:56
I can add a photo so you can see a demonstration. One second — Zvjezdan Veselinovic, Oct 14 '20 at 13:37

score 2 · Answer 2 · answered Oct 13 '20 at 17:48

2

As mentioned in the answer to this question, just use this:

element = driver.find_element_by_xpath(textbox_id)

driver.execute_script("arguments[0].scrollIntoView();", element) #Scrolls down until the element is visible

answered Oct 13 '20 at 17:48

Sushil

5,440
1
8
26

it scrolls to the first post again when ran the second time – wojwesoly Oct 13 '20 at 18:08
btw is xpath unique to every element or is it like an html class? because if it's unique then this code would continuously scroll to the same element – wojwesoly Oct 13 '20 at 18:33
Xpath can be unique or not. When it is not unique, it'll return a collection. – Ryanman Oct 13 '20 at 18:35
@Sushil maybe an if checking whether the button is on the page (you don't have to scroll to get to it), something like is_displayed – wojwesoly Oct 13 '20 at 18:36
Ok...So since you have multiple elements to scroll to, you can use ```driver.find_elements_by_xpath```instead of ```driver.find_element_by_xpath```. This would return a list of elements with the same xpath. Then, you can loop through these list of elements and call ```driver.execute_script("arguments[0].scrollIntoView();", element)``` each time. – Sushil Oct 14 '20 at 01:22

score 0 · Answer 3 · answered Oct 13 '20 at 17:43

0

please add part of code becasue if you know the aprox height can use:

js_scroll = "window.scrollBy(0, 600);"
driver.execute_script(js_scroll)

So can try finding the element and simulate press key down:

 someElement = driver.find_element_by_id('my-id')
 someElement.sendKeys(Keys.PAGE_DOWN)

I can't help you another way with out your code.

answered Oct 13 '20 at 17:43

wrocano

1
2

idk what other code to post, it's only logging into facebook and finding the button – wojwesoly Oct 13 '20 at 18:14
and I don't know the aprox. height beacause sometimes facebook show more comments under a post and sometimes less and the dimension of pictures in posts vary – wojwesoly Oct 13 '20 at 18:20
may be you can try using: `someElement.sendKeys(Keys.PAGE_DOWN)` and set a sleep to waiting to load new data on scroll view – wrocano Oct 14 '20 at 13:06
I can but if the button will still be out of the page then the program would either throw an error or just click the previous one again, if I do it like that it also needs to check whether the button is on the page or not – wojwesoly Oct 14 '20 at 13:29

score 0 · Answer 4 · answered Oct 13 '20 at 18:49

Fundamentally you're trying to scroll to a different element. IDs for a page are supposed to be completely unique for a page. If you select by ID and scroll to that element.... scrolling to that same element is going to scroll to that element.

You need to either

use different selenium selectors to get the collection of buttons and iterate through (Find by XPath, find by class, etc.)
Select the new or different button by its ID before scrolling to it.

I was originally going to use selecting by ID but that element doesn't have one — wojwesoly, Oct 13 '20 at 19:22

Arundeep Chohan · Answer 5 · 2020-10-13T19:54:46.523

0

You need to grab all the elements and loop them.

elements=driver.find_elements_by_css_selector("div[aria-label='Attach a photo or video']>i")
for elem in elements:
  driver.execute_script("arguments[0].scrollIntoView();", elem)

edited Oct 13 '20 at 19:54

answered Oct 13 '20 at 19:46

Arundeep Chohan

9,779
5
15
32

this button doesn't have an id – wojwesoly Oct 13 '20 at 19:49
What does the html element look like? – Arundeep Chohan Oct 13 '20 at 19:51
– wojwesoly Oct 13 '20 at 19:52
it's the button for uploading photos into comments on facebook – wojwesoly Oct 13 '20 at 19:53
Try this it should loop through all the attach photo. – Arundeep Chohan Oct 13 '20 at 19:58

Scrolling to the next element in Selenium (Python)

5 Answers5