Dynamically generated element -NoSuchElementException: Message: no such element: Unable to locate element?

Question

I am having trouble selecting a load more button on a Linkedin page. I receive this error in finding the xpath: selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element

I suspect that the issue is that the button is not visible on the page at that time. So I have tried actions.move_to_element. However, the page scrolls just below the element, so that the element is no longer visible, and the same error subsequently occurs.

I have also tried move_to_element_with_offset, but this hasn't changed where the page scrolls to.

How can I scroll to the right location on the page such that I can successfully select the element?

My relevant code:

import parameters
from selenium.webdriver.common.action_chains import ActionChains
from selenium import webdriver

ChromeOptions = webdriver.ChromeOptions()
driver = webdriver.Chrome('C:\\Users\\Root\\Downloads\\chromedriver.exe')

driver.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')
sleep(0.5)

username = driver.find_element_by_name('session_key')

username.send_keys(parameters.linkedin_username)
sleep(0.5)

password = driver.find_element_by_name('session_password')
password.send_keys(parameters.linkedin_password)
sleep(0.5)

sign_in_button = driver.find_element_by_xpath('//button[@class="btn__primary--large from__button--floating"]')
sign_in_button.click()

driver.get('https://www.linkedin.com/in/kate-yun-yi-wang-054977127/?originalSubdomain=hk')

loadmore_skills=driver.find_element_by_xpath('//button[@class="pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid"]')

actions = ActionChains(driver)
actions.move_to_element(loadmore_skills).perform()
#actions.move_to_element_with_offset(loadmore_skills, 0, 0).perform()
loadmore_skills.click()

AzyCrw4282 · Accepted Answer · 2020-04-21T17:11:28.987

After playing around with it, I seem to have figured out where the problem is stemming from. The error

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//button[@class="pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid"]"} (Session info: chrome=81.0.4044.113)

always correctly states the problem its encountering and as such it's not able to find the element. The possible causes of this include:

Element not present at the time of execution
Dynamically generated
content Conflicting names

In your case, it was the second point. As the content that is displayed is loaded dynamically as you scroll down. So When it first loads your profile the skills sections aren't actually present in the DOM. So to solve this, you simply have to scroll to the section so that it gets applied in the DOM.

This line is the trick here. It will position it to the correct panel and thus loading and applying the data to the DOM.

driver.execute_script("window.scrollTo(0, 1800)")

Here's my code (Please change it as necessary)

from time import sleep

# import parameters
from selenium.webdriver.common.action_chains import ActionChains
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait

ChromeOptions = webdriver.ChromeOptions()
driver = webdriver.Chrome('../chromedriver.exe')

driver.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')
sleep(0.5)

username = driver.find_element_by_name('session_key')

username.send_keys('')
sleep(0.5)

password = driver.find_element_by_name('session_password')
password.send_keys('')
sleep(0.5)

sign_in_button = driver.find_element_by_xpath('//button[@class="btn__primary--large from__button--floating"]')
sign_in_button.click()


driver.get('https://www.linkedin.com/in/kate-yun-yi-wang-054977127/?originalSubdomain=hk')

sleep(3)
# driver.execute_script("window.scrollTo(0, 1800)")
sleep(3)
loadmore_skills=driver.find_element_by_xpath('//button[@class="pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid"]')


actions = ActionChains(driver)
actions.move_to_element(loadmore_skills).perform()
#actions.move_to_element_with_offset(loadmore_skills, 0, 0).perform()
loadmore_skills.click()

Output

Update

In concerns to your newer problem, you need to implement a continuous scroll method that would enable you to dynamically update the skills section. This requires a lot of change and should ideally be asked as a another question.

I have also found a simple solution by setting the scroll to the correct threshold. For y=3200 seems to work fine for all the profiles I've checked including yours, mine and few others.

driver.execute_script("window.scrollTo(0, 3200)")

Thank you for this comprehensive answer! If the skills section is at a different coordinate on a different Linkedin profile, would the line ```driver.execute_script("window.scrollTo(0, 1800)")``` no longer work? My end goal is to make a for loop that runs this through multiple pages, so hoping for a scroll that would work no matter the Linkedin url — Yu Na, Apr 21 '20 at 16:00
Sadly no. In such cases, what you can do is perform continuous scroll until the element is found. — AzyCrw4282, Apr 21 '20 at 16:01
If this answer has helped then feel free to accept the answer :). If you do struggle to implement the continuous scroll then ask another question and @ me here. Thanks. — AzyCrw4282, Apr 21 '20 at 16:03
Any progress?? It would be great if you can give me an update. A side note, it would be great if you give the whole context of your problem on the initial post rather than incrementally building up the problem. — AzyCrw4282, Apr 21 '20 at 16:54
Thank you for your patience. This line worked ```driver.execute_script("window.scrollTo(0, 1800)")```, but ```driver.execute_script("window.scrollTo(0, 3200)")``` this line did not, as it scrolled to the very bottom of the page for me. I'll start another question, tried to implement a few continuous scroll options with no luck — Yu Na, Apr 21 '20 at 17:41
Attached is the link. Thanks so much for your continued help: https://stackoverflow.com/questions/61349971/how-to-continously-scroll-down-a-page-until-an-element-is-located-python-seleni — Yu Na, Apr 21 '20 at 17:58
Great, I will look into it and answer at the earliest. Also, if you feel I deserve the bounty feel free to award it by clicking the bounty sign near the up vote section. thanks — AzyCrw4282, Apr 21 '20 at 18:01

score 0 · Answer 2 · answered Apr 19 '20 at 04:12

0

If the button is not visible on the page at the time of loading then use the until method to delay the execution

try:
    myElem = WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.ID, 'IdOfMyElement')))
    print "Button is rdy!"
except TimeoutException:
    print "Loading took too much time!"

Example is taken from here

To get the exact location of the element, you can use the following method to do so.

element = driver.find_element_by_id('some_id')
element.location_once_scrolled_into_view

This actually intends to return you the coordinates (x, y) of the element on-page, but also scroll down right to target element. You can then use the coordinates to make a click on the button. You can read more on that here.

answered Apr 19 '20 at 04:12

AzyCrw4282

7,222
5
19
35

thanks for the reply. I tried both of these solutions to no avail. The first solution I received the Timeout Exception printed message. The second solution, I received the same error: ```NoSuchElementException: Message: no such element```. I know that my xpath is correct because I've used it line by line in ipython. – Yu Na Apr 19 '20 at 18:02
Very trivial error!!! This happens because when you automate it, it is not signed in and so it is often presented with either a LinkedIn register modal or a brief section of your linked profile. Try to visit your profile url in an incognito mode and you would be presented with either of the said cases. As such it does not contain the `skills section` and hence this gives the `no element found` error. – AzyCrw4282 Apr 20 '20 at 06:10
Ah, I should have included that in my code, but yes, I am logged in! I left it out for brevity. The element is not found despite being logged in – Yu Na Apr 20 '20 at 18:38

SeleniumUser · Answer 3 · 2020-04-21T23:45:35.757

You are getting NoSuchElementException error when the locators (i.e. id / xpath/name/class_name/css selectors etc) we mentioned in the selenium program code is unable to find the web element on the web page.

How to resolve NoSuchElementException:

Apply WebDriverWait : allow webdriver to wait for a specific time
Try catch block

so before performing action on webelement you need to take web element into view, I have removed unwated code and also avoided use of hardcoded waits as its not good practice to deal with synchronization issue. Also while clicking on show more button you have to scroll down otherwise it will not work.

from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains


driver = webdriver.Chrome(executable_path="path of chromedriver.exe")

driver.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')
driver.maximize_window()

WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.NAME, "session_key"))).send_keys("email id")
WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.NAME, "session_password"))).send_keys("password ")
WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.XPATH, "//button[@class='btn__primary--large from__button--floating']"))).click()

driver.get("https://www.linkedin.com/in/kate-yun-yi-wang-054977127/?originalSubdomain=hk")
driver.maximize_window()

driver.execute_script("scroll(0, 250);")
buttonClick = WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.XPATH, "//span[text()='Show more']")))
ActionChains(driver).move_to_element(buttonClick).click().perform()

Output:

Dynamically generated element -NoSuchElementException: Message: no such element: Unable to locate element?

3 Answers3

Linked