NoSuchElementException with Selenium

Question

I have been working on developing a bot called Bot1 to scrape data from a specific website. For this purpose, I am utilizing the Selenium framework. However, I encountered an issue where the bot fails to locate a particular element on the website during its initial run, but successfully locates the same element in subsequent runs.

I kindly request you to review my code and provide any insights or suggestions.

def scrape_products(self, product_name):
    try:
        # Navigate to the site
        self.browser.get("https://www.example.com/")

        # Find the search box by id, enter the product name and hit enter.
        search_box = self.browser.find_element(By.ID, "exampleid")
        search_box.send_keys(product_name)
        search_box.send_keys(Keys.RETURN)

        # Scrape the products information as web elements.
        products = WebDriverWait(self.browser, 10).until(EC.presence_of_all_elements_located(
            (By.CSS_SELECTOR, "[some-selector='some-value']"))
        )


        # List of products after extracting the html representation from web elements.
        products_data = []
        for product in products:
            product_html = product.get_attribute('outerHTML')
            products_data.append(product_html)
        return products_data

    except Exception as e:
        
        return "Not Found"


def run(self, product_name):
    product_list = self.scrape_products(product_name)

    if product_list:
        self.database.create_table(product_name)
        for product in product_list:
            self.database.insert_product(product_name, product)

b1 = Bot1()
for prod in ['item1', 'item2']:
    b1.run(prod)

When it first runs with item1 it can't locate this element -> search_box = self.browser.find_element(By.ID, "exampleid") and returns this error selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="exampleid"]"} but the subsequent iterations work correctly, meaning from item2 onward (if adding more items into the list) works correctly. It can find the element, it can scrape and can do other operations.

Could you please tell me what did I do wrong?

I have tried using different methods, such as:

search_box = WebDriverWait(self.browser, 10).until(
    EC.presence_of_element_located((By.ID, "exampleid"))
)

search_box = WebDriverWait(self.browser, 10).until(
    EC.visibility_of_element_located((By.ID, "exampleid"))
)

search_box = WebDriverWait(self.browser, 10).until(
    EC.element_to_be_clickable((By.ID, "exampleid"))
)

Find the search box using XPath

search_box = driver.find_element_by_xpath("//input[@id='exampleid']")

Find the search box using CSS selector

search_box = driver.find_element(By.CSS_SELECTOR, "input#exampleid")

Updated: The html of the search box from the site:

<input type="text" id="exampleid" value="" name="field-keywords" autocomplete="off" placeholder="Search Example" class="nav-input nav-progressive-attribute" dir="auto" tabindex="0" aria-label="Search Example" spellcheck="false">

For one thing, returning "Not Found" in case of an exception is **terrible**. — John Gordon, Jun 09 '23 at 19:55
Hi @JohnGordon It's not finalized, I just used Not Found for not getting any error. I will handle such case with logging and other things. Could you please help me finding the actual wrong I've done? — Imtiaz Ahmed, Jun 09 '23 at 20:02
Update the question with the text based HTML of the search box — undetected Selenium, Jun 09 '23 at 20:49
Sorry, I didn't get it. I'm trying to select the search box using id. @undetectedSelenium — Imtiaz Ahmed, Jun 09 '23 at 20:53
Hi @undetectedSelenium I've updated the question with the html of the search box. — Imtiaz Ahmed, Jun 09 '23 at 21:03

score 0 · Answer 1 · answered Jun 09 '23 at 21:11

Once the url is accessed for the first time:

self.browser.get("https://www.example.com/")

Just after the initial pageload allow some delay for the page elements to render before executing the tasks inducing WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:

Using CSS_SELECTOR:

search_box = WebDriverWait(self.browser, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#exampleid[aria-label='Search Example'][placeholder='Search Example']")))

Using XPATH:

search_box = WebDriverWait(self.browser, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@id='exampleid' and @aria-label='Search Example'][@placeholder='Search Example']")))

Note: You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

I tried using delay before trying to access the search box, this resulted in the same Exception. Or Are you suggesting I should initialize the browser with self.browswer.get("https://www.example.com/") in the constructor method. And then after initializing the class delay for a moment and then execute the first run, like this: `b1 = Bot1() time.delay(5) #asume I have initialized the browser with link in the constructor for prod in ['item1', 'item2']: b1.run(prod)` — Imtiaz Ahmed, Jun 09 '23 at 21:19
I tried allowing delay before executing the search_box line. It still gives error. Tried using both the delays and methods you provided in the answer. still same error. — Imtiaz Ahmed, Jun 09 '23 at 21:32

NoSuchElementException with Selenium

1 Answers1