Not clickable at point - SELENIUM - PYTHON

Question

Im trying to expand the arrow as below from https://www.xpi.com.br/investimentos/fundos-de-investimento/lista/#/

I'm usingo the code below:

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait as wait
from selenium.webdriver.support import expected_conditions as EC
from time import sleep

url = 'https://www.xpi.com.br/investimentos/fundos-de-investimento/lista/#/'

driver = webdriver.Chrome(options=options)
driver.get(url)
sleep(1)


expandir = driver.find_elements_by_class_name("sly-row")[-4]
expandir.click()
sleep(4)

expandir_fundo = wait(driver, 5).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div[class='arrow-details']")))
expandir_fundo.click()

And i'm getting the error: TimeoutException: Message

I tried to use the code below too:

expandir_fundo = driver.find_element_by_xpath('//*[@id="investment-funds"]/div/div/div[2]/article/article/section[2]/div[1]/div[10]')
expandir_fundo.click()

and got the error: ElementClickInterceptedException: Message: element click intercepted: Element ... is not clickable at point (1479, 8).

find below part of HTML:

<div class="funds-table-row sc-jAaTju kdqiDh sc-ckVGcZ drRPta">
  <div class="fund-name sc-jKJlTe hZlCDP" title="Bahia AM Maraú Advisory FIC de FIM" style="cursor:pointer;"><div>Bahia AM Maraú Advisory FIC de F...</div><p class="sc-brqgnPfbcFSC">Multimercado</p></div>
  <div class="morningstar sc-jKJlTe hZlCDP">-</div>
  <div class="minimal-initial-investment sc-jKJlTe hZlCDP">20.000</div>
  <div class="administration-rate sc-jKJlTe hZlCDP">1,90</div>
  <div class="redemption-quotation sc-jKJlTe hZlCDP"><div>D+30<p class="sc-brqgnP fbcFSC" style="font-size: 12px; color: rgb(24, 25, 26);">(Dias Corridos)</p></div></div>
  <div class="redemption-settlement sc-jKJlTe hZlCDP"><div>D+1<p class="sc-brqgnP fbcFSC" style="font-size: 12px; color: rgb(24, 25, 26);">(Dias Úteis)</p></div></div>
  <div class="risk sc-jKJlTe hZlCDP"><span class="badge-suitability color-neutral-dark-pure sc-jWBwVP hvQuvX" style="background-color: rgb(215, 123, 10);">8<span><strong>Perfil Médio</strong><br>A nova pontuação de risco leva em consideração critérios de risco, mercado e liquidez. Para saber mais, <a href="https://conteudos.xpi.com.br/guia-de-investimentos/relatorios/pontos-de-risco/" target="_blank" title="Clique aqui para saber mais sobre a nova pontuação de risco">clique aqui</a>.</span></span></div>
  <div class="profitability sc-jKJlTe hZlCDP"><div class="sc-kEYyzF lnwNVR"></div><div class="sc-kkGfuU jBBLoV"><div class="sc-jKJlTe hZlCDP">0,92</div><div class="sc-jKJlTe hZlCDP">0,48</div><div class="sc-jKJlTe hZlCDP">5,03</div></div></div><div class="invest-button sc-jKJlTe hZlCDP"><button class="xp__button xp__button--small" data-wa="fundos-de-investimento; listagem - investir; investir Bahia AM Maraú Advisory FIC de FIM">Investir</button></div>
  <div class="arrow-details sc-jKJlTe hZlCDP"><i type="arrow" data-wa="" class="eab2eu-0 eUjuAo"></i></div></div>

The HTML "arrow" is:

<div class="arrow-details sc-jKJlTe hZlCDP">
  <i type="arrow" data-wa="" class="eab2eu-0 eUjuAo">
    ::before
  </i>
</div>

It could be a couple things, but looking at the webpage that loads the first thing that comes to mind is that there is a big header on the page. That means when your scraper loads the page, it can't see the element unless you scroll to it. Try adding a line that scrolls to the element before clicking it. This should provide a good example: https://stackoverflow.com/questions/41744368/scrolling-to-element-using-webdriver — TomM, Aug 15 '21 at 21:27
If that doesn't work, update the post and people will know not to attempt that solution. — TomM, Aug 15 '21 at 21:28

YaDav MaNish · Answer 1 · 2021-08-16T04:28:24.493

You can check the below lines of code

option = Options()
#Disable the notification popUp
option.add_argument("--disable-notifications")
driver = webdriver.Chrome(r"ChromeDriverPath",chrome_options=option)

driver.get("https://www.xpi.com.br/investimentos/fundos-de-investimento/lista/#/")

#Clicked on the cookie, which is under the shadow-DOM So used execute_script(), Also used sleep() before clicking on cookies because at some point it throws the JS Error Shadow-DOM null.
sleep(5)
cookie_btn = driver.execute_script('return document.querySelector("#cookies-policy-container").shadowRoot.querySelector("soma-context > cookies-policy-disclaimer > div > soma-card > div > div:nth-child(2) > soma-button").shadowRoot.querySelector("button")')
cookie_btn.click()

#There can be a multiple way to scroll, Below is one of them
driver.execute_script("window.scrollBy(0,700)")

#There are multiple rows which have expand button so used the index of the XPath if you want to click on multiple try to use loop
expandir_fundo = WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, "((//*[@class='eab2eu-0 eUjuAo'])[1])")))
expandir_fundo.click()

import

from time import sleep
from selenium import  webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.chrome.options import Options

score 1 · Answer 2 · answered Aug 16 '21 at 10:02

While @YaDav MaNish answer seems to be working, I would rather use customized query Selector to click on accept cookies button, not generated by browser.

cookie_btn = driver.execute_script('return document.querySelector("#cookies-policy-container").shadowRoot.querySelector("soma-context soma-card soma-button").shadowRoot.querySelector("button")')
cookie_btn.click()

should work.

score 0 · Accepted Answer · answered Aug 16 '21 at 23:56

Your replies and the cookies code, gave me an ideia to usa the execute_script, to solve my problem.

Find below My code This first page I only open the URL, remove the cookies and expand all information.

# Setting options
options = Options()
options.add_argument('--window-size=1900,1000')
options.add_argument("--disable-notifications")

# Opening Website
url = 'https://www.xpi.com.br/investimentos/fundos-de-investimento/lista/#/'
driver = webdriver.Chrome(r"chromedriver", options=options)
driver.get(url)

# Removing Cookies
cookie_btn = driver.execute_script('return document.querySelector("#cookies-policy-container").'\
'shadowRoot.querySelector("soma-context > cookies-policy-disclaimer > div > soma-card > div > '\
'div:nth-child(2) > soma-button").shadowRoot.querySelector("button")')
cookie_btn.click()

# Expanding the whole list
expandir = driver.find_elements_by_class_name("sly-row")[-4]
expandir.click()
sleep(4)

# Creating content
page_content = driver.page_source
site = BeautifulSoup(page_content, 'html.parser')

This second part I used driver.execute_sript to open the arrow as I went getting information.

fundos = site.find_all('div',class_='funds-table-row')

for cont, fundo in enumerate(fundos):
    nome = fundo.find('div', class_='fund-name')['title']
    driver.execute_script(f"return document.getElementsByClassName('arrow-details'){[cont + 1]}.click()")
    page_detalhe = driver.page_source
    site2 = BeautifulSoup(page_detalhe, 'html.parser')
    details= site2.find('section', class_='has-documents').find_all('div')
    tax_id = details[2].get_text()

Thanks for everyone for help.

Not clickable at point - SELENIUM - PYTHON

3 Answers3