Python / Selenium - Comments in the code apparently doing difference in triggering NoSuchElementException

Question

Working on removing boring stuff from my job. Below you find the Python's Selenium library being used to access the website containing German companies' registers. It does the following:

1. opens the Handelsregister website,
2. looks for the "normal search" button and presses it,
3. looks for the field where company's name has to be input,
4. input chosen company's name,
5. changes the search manner to "contain the exact name of the company" and presses the "find" button,
6. presses the "CD" button (CD is a type of register).

Then the site requires you to login in order to download the register, so the last thing you should see is the login page. I show you the error first, if I do it at the bottom it shows up as part of the code.

Traceback (most recent call last): File "C:/Users/adria/.PyCharmCE2019.1/config/scratches/Handelsregister_login_downloadCD.py", line 28, in choose_class_CD_0 = chrome.find_element(by=By.CLASS_NAME, value='RegPortErg') File "C:\Users\adria\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 978, in find_element 'value': value})['value'] File "C:\Users\adria\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute self.error_handler.check_response(response) File "C:\Users\adria\Anaconda3\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".RegPortErg"} (Session info: chrome=78.0.3904.108)

Now, how is it possible that the following code works, while it doesn't once I remove the #-commented strings, giving away the above error?

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys

chrome = webdriver.Chrome(executable_path=r"C:\Users\...\Desktop\chromedriver_win32\chromedriver.exe")
chrome.get('https://www.handelsregister.de/')
assert "Register" in chrome.title

normal_search = chrome.find_element(by=By.ID, value='main.search')
normal_search.send_keys(Keys.ENTER)
# input_company_name_0 = chrome.find_element_by_id('container')
# input_company_name_1 = chrome.find_element_by_id('inhalt')
# input_company_name_2 = chrome.find_element_by_id('suchparameterForm')
input_company_name = chrome.find_element_by_name('schlagwoerter')
input_company_name.clear()
input_company_name.send_keys('Sparkasse Dortmund')
keywords = chrome.find_element_by_id('schlagwortOption3')
keywords.send_keys(Keys.SPACE)
find_button = chrome.find_element_by_id('submitBtn')
find_button.send_keys(Keys.ENTER)
choose_class_CD_0 = chrome.find_element(by=By.CLASS_NAME, value='RegPortErg')
# choose_class_CD_1 = chrome.find_element(by=By.CLASS_NAME, value='RegPortErg_RandRechts')
choose_CD = chrome.find_element(by=By.LINK_TEXT, value="CD")
choose_CD.send_keys(Keys.ENTER)

Answer 1

It seems a bit weird to me that comments would affect your code, unless there's some inherent issue with the parser you are using to run Python.

You are receiving a NoSuchElementException -- it may be possible that the element you are looking for on the page has not fully loaded before you attempt to locate it. This would result in the exception you are seeing.

To fix this, I would add some WebDriverWait into your code to wait on elements before you attempt to locate them.

Here's how I would refactor your code:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys


chrome = webdriver.Chrome()
chrome.get('https://www.handelsregister.de/')

assert "Register" in chrome.title

# wait on & locate 'normal search'
normal_search = WebDriverWait(chrome, 10).until(EC.presence_of_element_located((By.ID, "main.search")))
normal_search.send_keys(Keys.ENTER)

# wait on & input 'company name'
input_company_name = WebDriverWait(chrome, 10).until(EC.presence_of_element_located((By.NAME, "schlagwoerter")))
input_company_name.clear()
input_company_name.send_keys('Sparkasse Dortmund')

# select the radio button
chrome.find_element_by_id('schlagwortOption3').click()

# find and click submit button
chrome.find_element_by_id('submitBtn').click()

# wait for this element to exist before locating it
choose_class_CD_0 = WebDriverWait(chrome, 10).until(EC.presence_of_element_located((By.CLASS_NAME, "RegPortErg")))
choose_CD = chrome.find_element(by=By.LINK_TEXT, value="CD")
choose_CD.send_keys(Keys.ENTER)

chrome.close()
chrome.quit()

The issue I noticed when running your code was that form on 'Normal Search' page was never getting submitted, and we were not getting taken to the next page at all. I noticed the "Company or Keywords" field was not actually receiving any text input, and the radio button for "contain the exact name of the company" was also not getting ticked.

I fixed these issues by adding a WebDriverWait on the "Company or Keywords" field, because that's the first element on the page that we interact with. The send_keys operation was probably happening before the element was fully loaded, resulting in blank key send.

I also noticed you were using KEYS.Space and KEYS.Enter to simulate clicking elements -- while this may work for this website, there's nothing wrong with using a plain Selenium .click() for scenarios like this. I updated that in your code as well.

I left comments throughout the code to keep track of what I changed and I hope you find them helpful. I successfully ran the above code sample provided and did not receive the NoSuchElementException. The "Normal Search" form was successfully submitted, and took me to the next page, where the RegPortErg was successfully located.

Hopefully this WebDriverWait will help you continue this script without issue.

Answer 2

To open the site https://www.handelsregister.de/ and perform the mentioned steps to reach to the Login Page you need to induce WebDriverWait for the element_to_be_clickable() and you can use the following Locator Strategies:

• Code Block:

  from selenium import webdriver
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  
  options = webdriver.ChromeOptions() 
  options.add_argument("start-maximized")
  options.add_experimental_option("excludeSwitches", ["enable-automation"])
  options.add_experimental_option('useAutomationExtension', False)
  driver = webdriver.Chrome(options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
  driver.get("https://www.handelsregister.de/rp_web/welcome.do")
  WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, "//ul[@class='hinterlegt']/li/a[text()='Normal search']"))).click()
  WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, "//textarea[@name='schlagwoerter']"))).send_keys("Sparkasse Dortmund")
  driver.find_element_by_xpath("//label[@for='schlagwortOption3']").click()
  driver.find_element_by_xpath("//input[@id='submitBtn']").click()
  WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, "//td[@class='RegPortErg_RandRechts']//a[text()='CD']"))).click()
  

• Browser Snapshot: