why does my selenium code work sometimes yes, sometimes no?

Question

I have little experience with selenium and would like to understand why my code sometimes finds the necessary elements, and sometimes it returns an error message saying that it did not find anything. Here is a print of one of the errors that appears, usually it informs that the element was not found, or that it is not clickable. erro

my code, i know it's messed up because i've tried several different ways to locate elements, all of them work, once yes, another time no. and so i'm in the eternal loop.

from selenium import webdriver
import undetected_chromedriver as uc  
from selenium.webdriver.common.by import By    #importa localizadores By
from selenium.webdriver.common.keys import Keys    #importa input para envio de dados
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time   #biblioteca de tempo

driver = uc.Chrome()   #driver alterado para 'undetected'

driver.get('[https://www.bcb.gov.br/estabilidadefinanceira/scrdata][2]')
time.sleep(10)

#seleção do tipo do cliente
todos = driver.find_element(By.XPATH, "//div/div[1]/div[1]/div/div[1]/div[2]/ng-select/div/div/div[3]/input")
todos.click()

op2 = driver.find_element(By.ID, "ab2af0cfd2af-2")
op2.click()

#seleção label cnae e adição de serie
time.sleep(2)
cnae = driver.find_element(By.XPATH, "//div/div[3]/bcb-graficoscr/div/div[1]/div[1]/div/div[2]/div[1]/ng-select")
cnae.click()

#passo 1: seleção cnae pj
wait = WebDriverWait(driver, 2)
sel = driver.find_element(By.XPATH, "//div/div[1]/div[1]/div/div[2]/div[1]/ng-select/ng-dropdown-panel/div/div[2]/div[10]")
wait.until(EC.element_to_be_clickable((sel)))
sel.click()

#click botão add
wait1 = WebDriverWait(driver, 2)
btn = driver.find_element(By.XPATH, "//*[contains(text(), 'Adicionar série')]")
wait1.until(EC.element_to_be_clickable((btn)))
btn.click()

time.sleep(3)
#passo 2:
wait1 = WebDriverWait(driver, 2)
sel1 = driver.find_element(By.XPATH, "/html/body/app-root/app-root/div/div/main/dynamic-comp/div/div[3]/bcb-graficoscr/div/div[1]/div[1]/div/div[2]/div[1]/ng-select/ng-dropdown-panel/div/div[2]/div[11]")
wait1.until(EC.element_to_be_clickable((sel1)))
sel.click()


time.sleep(10000)
driver.quit()

I already tried to locate elements by ID, which exist in my HTML but still selenium does not recognize, so I did everything by XPATH. I have also looked for ways to shorten the XPATH to avoid breaks, but again, once it works and once it doesn't. The search elements are static in the HTML do not change.

the url of the site here: https://www.bcb.gov.br/estabilidadefinanceira/scrdata

my intention is to automatically click on "tipo do cliente" --> select PJ --> click on CNAE --> select one activity at a time, related to PJ. (you click on the PJ activity you want, then click on the insert series button and repeat the action of selecting activity)

anyone who knows how to solve it, I am very grateful ;)

Answer 1

This error message...

...indicates that NoSuchElementException was raised when searching for the elements with the given locator strategy

---

Solution

The desired element is a dynamic element, so ideally to click on the element with text as Pessoas jurídicas - PJ from the dropdown you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:

• Code block:

  driver.get(url='https://www.bcb.gov.br/estabilidadefinanceira/scrdata')
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.btn.btn-primary.btn-accept"))).click()
  driver.execute_script("scroll(0, 250);") # Scroll Down
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//label[text()='Tipo de cliente']//following::ng-select[1]//div[@class='ng-input']"))).click()
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//ng-dropdown-panel[@aria-label='Options list']//span[text()='Pessoas jurídicas - PJ']"))).click()
  

• Note: You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  

• Browser snapshot:

---

References

You can find a couple of relevant discussions on NoSuchElementException in:

• selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element while trying to click Next button with selenium
• selenium in python : NoSuchElementException: Message: no such element: Unable to locate element

Answer 2

A Python Selenium framework such as https://github.com/seleniumbase/SeleniumBase can make your code more readable by using a special :contains() selector that makes your selectors understandable.

pip install -U seleniumbase then run the following script with python:

from seleniumbase import SB

with SB() as sb:
    sb.open("https://www.bcb.gov.br/estabilidadefinanceira/scrdata")
    sb.highlight("h1")
    sb.highlight('h5:contains("Filtros")')
    sb.sleep(2)
    sb.highlight_click('label:contains("Tipo de cliente") + ng-select')
    sb.click('span:contains("Pessoas jurídicas - PJ")')
    sb.highlight_click('label:contains("Modalidade") + ng-select')
    sb.click('span:contains("PJ - Capital de giro")')
    sb.highlight_click('label:contains("UF") + ng-select')
    sb.click('span:contains("Amazonas")')
    sb.highlight_click('label:contains("CNAE ou Ocupação") + ng-select')
    sb.click('span:contains("PJ - Artes, cultura, esporte e recreação")')
    sb.highlight_click('label:contains("Porte dos clientes") + ng-select')
    sb.click('span:contains("PJ - Grande")')
    sb.highlight_click('label:contains("Origem de recursos") + ng-select')
    sb.click('span:contains("Com destinação específica")')
    sb.highlight_click('label:contains("Indexador da operação") + ng-select')
    sb.click('span:contains("Índices de preços")')
    sb.click('button:contains("Adicionar série")')
    sb.sleep(5)

Adjust sleep() / timeouts as needed.