Scroll in Selenium Webdriver (Python)

Question

Prerequisites.
You need an account at Instagram to use this script.
Setup a test environment:

Log in, open the needed list(works correctly):

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from time import sleep

driver = webdriver.Chrome(
# driver = webdriver.Firefox(    
# driver = webdriver.PhantomJS(
    service_args=['--ignore-ssl-errors=true', '--ssl-protocol=any'])

driver.get("https://instagram.com/accounts/login")
username = driver.find_element_by_name("username")
password = driver.find_element_by_name("password")

username1 = 'instagram' # change it!
password1 = 'instagrampassword1' # change it!

username.send_keys(username1)
password.send_keys(password1)

submit_button = driver.find_element_by_css_selector(
    '#react-root > div > article > div > div:nth-child(1) > div > form > span > button')
submit_button.click()

sleep(2)

link = 'https://www.instagram.com/youtube/'
driver.get(link)

driver.implicitly_wait(2)
driver.find_elements_by_class_name("_218yx")[2].click()

Wrong scroll. How to fix this block?

How to focus and scroll correctly on this page?

My attempts:

driver.find_element_by_class_name("_cx1ua").send_keys(Keys.NULL) # focus
      #The element has been deleted entirely or
      #The element is no longer attached to the DOM.

driver.find_element_by_class_name("_q44m8").send_keys(Keys.NULL)
# cannot focus element

driver.find_element_by_class_name("_qjr85").send_keys(Keys.NULL) 
# cannot focus element


for i in range(5):
    driver.find_element_by_class_name("_cx1ua").send_keys(Keys.END)

=============================================================
to @Moshisho :

We need to focus on some element to activate it.
The question is what the element we need to choose to focus and how?
This is not a "body":
something like that, but not this:

background = driver.find_element_by_css_selector("body")
# background = driver.find_element_by_css_selector("div._2uju6")

for i in range(5):
    background.send_keys(Keys.SPACE)
    time.sleep(1)

Without it this command do not work.

to @Naveen :

print(driver.find_element_by_css_selector("div._a1rcs").location_once_scrolled_into_view) # {'x': 0, 'y': 0}
print(driver.find_element_by_class_name("_cx1ua").location_once_scrolled_into_view) # {'x': 376, 'y': 229}
print(driver.find_element_by_class_name("_q44m8").location_once_scrolled_into_view) # {'x': 376, 'y': 180}
print(driver.find_element_by_class_name("_qjr85").location_once_scrolled_into_view) # {'x': 376, 'y': 180}

And what's next?

driver.execute_script("window.scrollTo(0, 3000);") # do not working

Your question should be closed as off-topic. Give some details and description to make your issue more clear... not everyone has account on `Instagram` :) — Andersson, Dec 25 '16 at 17:58
Your attempts are for focusing or scrolling a specific element? — Moshisho, Dec 25 '16 at 18:40
Yes, I want to scroll all the list of followers, to see > 10 followers per one visit. — lvcpp, Dec 25 '16 at 18:42

score 5 · Accepted Answer · edited Dec 26 '16 at 15:39

Try the following code:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from time import sleep
from selenium.webdriver.support.ui import Select

driver = webdriver.Chrome(
# driver = webdriver.Firefox(    
# driver = webdriver.PhantomJS(
    service_args=['--ignore-ssl-errors=true', '--ssl-protocol=any'])

driver.maximize_window()

driver.get("https://instagram.com/accounts/login")
username = driver.find_element_by_name("username")
password = driver.find_element_by_name("password")

username1 = 'instagramlogin1'    # change it!
password1 = 'instagrampassword1' # change it!

username.send_keys(username1)
password.send_keys(password1)

submit_button = driver.find_element_by_css_selector(
    '#react-root > div > article > div > div:nth-child(1) > div > form > span > button')
submit_button.click()

sleep(2)

link = 'https://www.instagram.com/youtube/'
driver.get(link)

driver.implicitly_wait(2)
following = driver.find_element_by_xpath("//a[@href='/youtube/following/']/span")
total_following = int(following.text)
print "total no. of users following: ", total_following
# click on 239 following, displays 10 users
following.click()

loaded_following = driver.find_elements_by_xpath("//ul[@class='_539vh _4j13h']/li")
loaded_till_now = len(loaded_following)

while(loaded_till_now<total_following):
    print "following users loaded till now: ", loaded_till_now
    print loaded_following[loaded_till_now-1]
    loaded_following[loaded_till_now-1].location_once_scrolled_into_view
    # driver.execute_script("arguments[0].focus();", loaded_following[loaded_till_now-1])
    driver.find_element_by_tag_name('body').send_keys(Keys.END) # triggers AJAX request to load more users. observed that loading 10 users at a time.
    sleep(1) # tried wihtout sleep but throws StaleElementReferenceException. As it takes time to get the resposne and update the DOM
    loaded_following = driver.find_elements_by_xpath("//ul[@class='_539vh _4j13h']/li")
    loaded_till_now = len(loaded_following)

# All 239 users are loaded. 
driver.quit()

Observed that browser is sending AJAX request to load more users. this action is triggered when you scroll using mouse or enter Space or End keys

thanks, interesting property, but how I can use it to do the needed things? see the explained message to you above. — lvcpp, Dec 26 '16 at 11:23
I am not sure which element you are talking about (the class names are dynamic, so not working for me). Can you just share the screenshot of the screen, highlighting the element and what you want to do with it? either click or send keys? — Naveen Kumar R B, Dec 26 '16 at 11:47
At this case the class names are static, only looks like dynamic. You can check it with different browsers. — lvcpp, Dec 26 '16 at 12:19
without any manipulation self.driver.find_elements_by_class_name("_6jvgy") I'll get only 10 elements, — lvcpp, Dec 26 '16 at 12:19
but I want to scroll down, ideally to the end of list, to get 239 elements(for youtube). — lvcpp, Dec 26 '16 at 12:20
After a lot of trials, achieved the scrolling to load all 239 users. please try and let me know. Observed that AJAX requests are being sent when you scroll, which loads 10 users at a time. — Naveen Kumar R B, Dec 26 '16 at 13:54

Moshisho · Answer 2 · 2016-12-26T07:24:18.933

4

In order to scroll in the window, you need to execute JavaScript, try this:

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

EDIT: in order to focus an element (it needs to be able to get the focus e.g. an anchor, input, button etc...) you also need to use JavaScript executor:

elementToFocus = driver.find_element_by_id("yourID")
driver.execute_script("arguments[0].focus();", elementToFocus)

edited Dec 26 '16 at 07:24

answered Dec 25 '16 at 18:46

Moshisho

2,781
1
23
39

thanks, I saw this code snippet in a few places earlier. We need to focus on some element to activate it. The question is what the element we need to choose to focus and how? This is not a "body": background = browser.find_element_by_css_selector("body") # background = driver.find_element_by_css_selector("div._2uju6") for i in range(5): background.send_keys(Keys.SPACE) time.sleep(1) Without it this command do not work. – lvcpp Dec 25 '16 at 19:01
there no any element with id.
I've tested the second line, but it dont work yet.
So I've selected by class, but maybe need to select 2nd or 3rd element of the class
(I know how to do it, but there a lot of elements, and it looks like the majority of them should be visible and acceptable). – lvcpp Dec 26 '16 at 09:10

score 0 · Answer 3 · answered Mar 24 '21 at 04:03

I'm working with a dynamic React app, I need to scroll to the pages bottom to make react render all the data.

For unknown reasons, solutions based on JS execute_script didn't work. However I got send_keys solution working:

# scroll to bottom to load all
WebDriverWait(driver, 5).until(
    EC.presence_of_element_located((By.XPATH, "//body"))
)

attempt_num = 2
while attempt_num > 0:
    try:
        elem = driver.find_element_by_xpath("//body")
        elem.click()
        elem.send_keys(Keys.END)
    except StaleElementReferenceException as e:
        print(e)
    attempt_num = attempt_num - 1

The click() on body and the retry for StaleElementReferenceException are crucial. I haven't found a more elegant way than to retry.

See top answer of How to avoid "StaleElementReferenceException" in Selenium?

Scroll in Selenium Webdriver (Python)

3 Answers3

Linked