How to continously scroll down a page until an element is located? Python selenium

Question

I'm baffled by the load more button for the skills section in Linkedin. I receive this error in finding the xpath for the button: selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element

The issue is that my element is not visible on the page, so I've been trying to find a way to continuously scroll on the page until the button is visible. I'm trying to do a forloop of multiple profiles.

My relevant code:

import parameters
from selenium.webdriver.common.action_chains import ActionChains
from selenium import webdriver

ChromeOptions = webdriver.ChromeOptions()
driver = webdriver.Chrome('C:\\Users\\Root\\Downloads\\chromedriver.exe')

driver.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')
sleep(0.5)

username = driver.find_element_by_name('session_key')

username.send_keys(parameters.linkedin_username)
sleep(0.5)

password = driver.find_element_by_name('session_password')
password.send_keys(parameters.linkedin_password)
sleep(0.5)

sign_in_button = driver.find_element_by_xpath('//button[@class="btn__primary--large from__button--floating"]')
sign_in_button.click()

driver.get('https://www.linkedin.com/in/kate-yun-yi-wang-054977127/?originalSubdomain=hk')

loadmore_skills=driver.find_element_by_xpath('//button[@class="pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid"]')

Attempts 1.

actions = ActionChains(driver)
actions.move_to_element(loadmore_skills).perform()
#actions.move_to_element_with_offset(loadmore_skills, 0, 0).perform()
loadmore_skills.click()

With actions.move_to_element the page scrolls just below the element, so that the element is no longer visible, and the same error subsequently occurs.

I have also tried move_to_element_with_offset, but this hasn't changed where the page scrolls to.

2.

coordinates = loadmore_skills.location_once_scrolled_into_view 
driver.execute_script('window.scrollTo({}, {});'.format(coordinates['x'], coordinates['y']))

This returns the same error message

3.

loadmore_skills=WebDriverWait(driver,30).until(EC.element_to_be_clickable((By.XPATH, '//button[@class="pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid"]')))

This returns the same error, as well.

4.

driver.execute_script("arguments[0].scrollIntoView();", loadmore_skills)

Not sure how else to do this. Your help is greatly appreciated.

UPDATE: Trying @Dipak solution moves me to the bottom of the page, and the element cannot be clicked:

Error Traceback:

Traceback (most recent call last):
  File "C:/Users/Root/PycharmProjects/Quant/skillstest.py", line 60, in <module>
    EC.element_to_be_clickable((By.XPATH, "//span[text()='Show more']")))
  File "C:\Users\Root\PycharmProjects\Quant\venv\lib\site-packages\selenium\webdriver\support\wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

Just try adding more time to driver wait (as looking towards from the below answers, I think it's the internet problem that you are facing). @YuNa — Abhay Salvi, Apr 22 '20 at 07:40
I've changed it up to 50 seconds, and I still have the same traceback/error — Yu Na, Apr 22 '20 at 17:34

SeleniumUser · Answer 1 · 2020-04-21T23:16:30.797

1

Try below code :

from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains


driver = webdriver.Chrome(executable_path="C:\New folder\chromedriver.exe")

driver.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')
driver.maximize_window()

WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.NAME, "session_key"))).send_keys("email id")
WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.NAME, "session_password"))).send_keys("password")
WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.XPATH, "//button[@class='btn__primary--large from__button--floating']"))).click()

driver.get("https://www.linkedin.com/in/kate-yun-yi-wang-054977127/?originalSubdomain=hk")
driver.maximize_window()

buttonClick = WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.XPATH, "//span[text()='Show more']")))
ActionChains(driver).move_to_element(buttonClick).click().perform()

Output:

edited Apr 21 '20 at 23:16

answered Apr 21 '20 at 18:20

SeleniumUser

4,065
2
7
30

when I ran this code, the page is scrolled to the very bottom of the page. Then the error message appears: ``` raise TimeoutException(message, screen, stacktrace) selenium.common.exceptions.TimeoutException```. This is likely because the skills section is out of view when the page is scrolled to the very bottom. Is there a way to modify the code such that the scroll stops when the element is in view? – Yu Na Apr 21 '20 at 19:28
are you trying to click on show more button? i have verified this and its working fine for me – SeleniumUser Apr 21 '20 at 19:42
I am indeed trying to click on the show more button. Is your page also scrolling to the very bottom? – Yu Na Apr 21 '20 at 19:49
Check updated solution now I have verified this and attached screenshot also – SeleniumUser Apr 21 '20 at 20:21
Hm, I'm still getting the error, and my page is scrolled to the very bottom. Maybe if I understand the code better, I can try and figure out why it's going wrong. Why do you use this line twice? ```driver.execute_script("window.scrollTo(0, document.body.scrollHeight)")```. Isn't this what is moving the page to the bottom? – Yu Na Apr 21 '20 at 22:59
I have used same code and its working for me, what do you mean by very bottom ? – SeleniumUser Apr 21 '20 at 23:00
Updated the post to show the screenshot and traceback – Yu Na Apr 21 '20 at 23:02
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/212212/discussion-between-dipak-bachhav-and-yu-na). – SeleniumUser Apr 21 '20 at 23:05

Abhay Salvi · Accepted Answer · 2020-04-23T05:35:46.580

Here's the fully working code solving your problems as per you stated in the comments.

The answers suggested by others was actually scrolling down to bottom of the page and then also giving errors. Then i noticed that if you scroll drown to bottom then only the bottom section loads out, not all of those sections in between. (Dipak's answer isn't working for me also. Maybe this is a resolution problem for you, as well as for me : ) as stated by him in the chats)

Because what you want is present between the page & not in the bottom. So just the bottom section loads out always. So now we need to do something else.

What we want now, is to scroll down to only those sections which you want. And to make a custom scroll down i used driver.execute_script("scroll(0, 1600)"). I also removed all those unnecessary things from the code and kept it very simple and straight forward.

from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
import time

driver = webdriver.Chrome(executable_path=r"C:\Users\intel\Downloads\Setups\chromedriver")

driver.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')
driver.maximize_window()

WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.NAME, "session_key"))).send_keys("EMAIL")
WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.NAME, "session_password"))).send_keys("PASSWORD")
WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.XPATH, "//button[@class='btn__primary--large from__button--floating']"))).click()

driver.get("https://www.linkedin.com/in/kate-yun-yi-wang-054977127/?originalSubdomain=hk")
driver.maximize_window()

driver.execute_script("scroll(0, 1600)")
time.sleep(5)
buttonClick = driver.find_element_by_xpath("/html/body/div[6]/div[4]/div[3]/div/div/div/div/div[2]/main/div[2]/div[6]/div/section/div[2]/button/span[1]").click()

Also use --headless browser to load the tasks more quicker. And if possible, then use css_selectors other than XPATH's, cauz they are the slowest locators for scraping purpose.

Hi Abhay, thanks so much for looking into this. Wouldn’t this solution only work for this page? When I try to loop all the LinkedIn profiles, wouldn’t the scroll no longer work? — Yu Na, Apr 23 '20 at 17:27
I think it may work for the multiple pages. Did you tried it yet? Also did my answer worked for you or not? — Abhay Salvi, Apr 23 '20 at 17:29
Hi Abhay, This solution works for this specific profile page, but not for all others. For example try: ```driver.get('https://www.linkedin.com/in/annie-simons/?trk=public_profile_samename_mini-profile_title')```. Is there a way to set up a for loop such that it tries different coordinates on the page, and if the element does not exist, it tries the same process at a different coordinate? — Yu Na, Apr 23 '20 at 23:47
Yes it is possible, but it will slow down your entire process and will take more of your time... — Abhay Salvi, Apr 24 '20 at 01:54

Sodium · Answer 3 · 2020-04-21T19:49:19.553

Try this

actions.move_to_element(loadmore_skills).build().perform()

Here is corresponding working Java Code

@Test
public void linkedInTest(){
    driver.findElement(By.name("session_key")).sendKeys(username);
    driver.findElement(By.name("session_password")).sendKeys(password);
    driver.findElement(By.xpath("//button[@class='btn__primary--large from__button--floating']")).click();

    driver.get("https://www.linkedin.com/in/kate-yun-yi-wang-054977127/?originalSubdomain=hk");

    WebElement loadmore_skills = driver.findElement(By.xpath("//button[@class='pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid']"));
    action.moveToElement(loadmore_skills).click().build().perform();
}

this command also raised the same error as my previous ``actions.move_to``` command: ```selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element``` — Yu Na, Apr 21 '20 at 19:29

score 0 · Answer 4 · answered Jul 26 '22 at 12:13

Here is a robust solution that can work with a bit of tweaking.

driver.get("url here")
elements_list = driver.find_elements(By.XPATH, "xpath to elements")
for element in elements_list:
  elem_not_found = True
  y = 10 # setting initial scroll pixel size to 10.. find what suits you 
  while elem_not_found:
      try:
          ActionChains(driver).scroll_by_amount(0, y).perform()
          time.sleep(1)
          hov = ActionChains(driver).move_to_element(element).perform()
          WebDriverWait(driver, 5).until( 
          EC.presence_of_element_located((By.XPATH, "xpath here")))
          #Once element is loaded do whatever you want with it 
          elem_not_found = False
      except:
          y += 35 # scrolling 35 pixels below if element is not found

This code keeps on scrolling the page until the element is not found!!

How to continously scroll down a page until an element is located? Python selenium

4 Answers4

Linked