Unable to click on element even after finding it- Python seleneium

Question

I have been pained by a load more button for a while. I'm looking to create a loop where I click "load more" on the skills section of Linkedin pages. However, this button is just not consistently being clicked.

I was under the impression that the issue was that the element was not visible on the page. So, I have a segmented scroll, which continues moving down the page until the element is found. But what's baffling is that even though the page is now moving to the right place, the element is not being clicked. No error is being thrown.

I've tried nearly every version of the element location (xpath, class name, css selector, full xpath). Why would the button not be clicked, if it is visible on the page?

Relevant Code:

##log into Linkedin

linkedin_urls=['https://www.linkedin.com/in/julie-migliacci-revent/']

ChromeOptions = webdriver.ChromeOptions()
driver = webdriver.Chrome('C:\\Users\\Root\\Downloads\\chromedriver.exe')

driver.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')
driver.maximize_window()

WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.NAME, "session_key"))).send_keys("EMAIL")
WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.NAME, "session_password"))).send_keys("PASSWORD")
WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.XPATH, "//button[@class='btn__primary--large from__button--floating']"))).click()


linkedin_urls=['https://www.linkedin.com/in/julie-migliacci-revent/', 'https://www.linkedin.com/in/kathleen-meyers-126a7931']


for linkedin_url in linkedin_urls:
   driver.get(linkedin_url)

   looking_for_element = True
   ldmore = ''

   while looking_for_element:
       elements = driver.find_elements_by_xpath('/html/body/div[7]/div[3]/div/div/div/div/div[2]/main/div[2]/div[6]/div/section/div[2]/button/span[1]')

       if len(elements) > 0:
          ldmore = elements[0]
          ldmore.click()
          looking_for_element = False
       else:
           global_copyright = driver.find_elements_by_css_selector('#globalfooter-copyright')

           if len(global_copyright) > 0:
               looking_for_element = False
           else:
               body = driver.find_element_by_css_selector('body')
               sleep(5)
               body.send_keys(Keys.PAGE_DOWN)

I've not seen a discussion on SO about element issues when the underlying solution is not visibility. The code is designed to stop once the element is located -- and is performing this correctly. But it just isn't clicking on the element. I'm not sure why this is.

Locations I've tried:

absolute xpath:
driver.find_element_by_xpath('/html/body/div[7]/div[3]/div/div/div/div/div[2]/main/div[2]/div[6]/div/section/div[2]/button/span[1]').click()

relative xpath: 
//span[contains(text(),'Show more')]

class name: 
pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid" aria-controls="skill-categories-expanded

css: 
body.render-mode-BIGPIPE.nav-v2.theme.theme--classic.ember-application.boot-complete.icons-loaded:nth-child(2) div.application-outlet:nth-child(77) div.authentication-outlet:nth-child(3) div.extended div.body div.pv-profile-wrapper.pv-profile-wrapper--below-nav div.self-focused.ember-view div.pv-content.profile-view-grid.neptune-grid.two-column.ghost-animate-in main.core-rail div.profile-detail div.pv-deferred-area.ember-view:nth-child(6) div.pv-deferred-area__content section.pv-profile-section.pv-skill-categories-section.artdeco-container-card.ember-view div.ember-view > button.pv-profile-section__card-action-bar.pv-skills-section__additional-skills.artdeco-container-card-action-bar.artdeco-button.artdeco-button--tertiary.artdeco-button--3.artdeco-button--fluid

UPDATE: Tried the JS force, it clicked! But threw up the error: selenium.common.exceptions.WebDriverException: Message: unknown error: Cannot read property 'click' of null

if len(elements) > 0:
    ldmore = elements[0]
    ldmorebtn = driver.find_element_by_xpath('/html/body/div[7]/div[3]/div/div/div/div/div[2]/main/div[2]/div[6]/div/section/div[2]/button/span[1]').click()
    #driver.execute_script("arguments[0].checked = true;", ldmore)
    driver.execute_script("arguments[0].click();", ldmore)

Have you read [this](https://stackoverflow.com/questions/6101461/how-to-force-selenium-webdriver-to-click-on-element-which-is-not-currently-visib) I had a similair problem and had to resort to using pure JS to force a click — Umar.H, Apr 28 '20 at 01:43
did you try to click `button` instead of `button/span` ? Normally `span` is not clickabled, only `button` is clickabled. — furas, Apr 28 '20 at 02:32
Hi Datanovice, neat advice. I added ```driver.execute_script("arguments[0].click();", ldmore)```. It clicked, but threw up an error: ```selenium.common.exceptions.WebDriverException: Message: unknown error: Cannot read property 'click' of null```. Did you receive this too? — Yu Na, Apr 28 '20 at 02:35
You should use relative XPath of the button (see below) as @furas has suggested. You should first scroll until `
Recommandations
` is visible (with js scrollIntoView). Then use (after importing the proper libraries) : `WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,"//button[@data-control-name="skill_details"]"))).click()` — E.Wiest, Apr 28 '20 at 11:32
Hi @E. Wiest, I changed it as you suggested and it does click sometimes, but strangely getting the message that my click is being intercepted in other instances of iterating over profiles. (```error: Message: element click intercepted:```). I wonder why it only works sometimes. The page is still landing on the correct element, though. The click is the issue — Yu Na, Apr 28 '20 at 20:05
@Datanovice I managed to adapt the java script force click to this situation, if you want to post as an answer, I will accept — Yu Na, May 06 '20 at 16:57

score 1 · Accepted Answer · answered May 06 '20 at 17:16

The suggestion by @Datanovice to use javascript to force a click worked like a charm. Initially, when I had tried to adapt the solution, I received the error selenium.common.exceptions.WebDriverException: Message: unknown error: Cannot read property 'click' of null.

This error was because I had been using EC.element_to_be_clickable. Instead, when I paired the java method with EC.visibility_of_element_located, the click consistently worked.

The code:

ldmore = WebDriverWait(driver, 30).until(EC.visibility_of_element_located((By.XPATH,'xpath')))
driver.execute_script("arguments[0].click();", ldmore)

Unable to click on element even after finding it- Python seleneium

Recommandations

1 Answers1