Why selenium's click method is returning NoSuchElementException?

Question

During scraping a javascript base website(link), I've stuck with a click function which is not producing the desired result. I've pasted the code which should return the data under "Fits the following cars" drop down menu but unexpectedly it's just printing the except statement's message. I'm able to scrape all the other data from the same code. Should I add few more lines to get the data hidden under that drop down, if yes then what lines to add.

def product(self,response):
    while True:
        try:
            drop=self.driver.find_element_by_xpath('//*[@id="toggleMakeModelArrow"]')
            self.logger.info('Sleep for 3 sec.')
            sleep(3)
            drop.click()
            sel=Selector(text=self.driver.page_source)
            drop_down=sel.xpath('//*[@id="CachedItemDispaly_make_model_div"]/select/option/text()').extract()
            for i in range(len(drop_down)):
                print drop_down[i]+"||"

        except NoSuchElementException:
            self.logger.info('No more Fits the following cars to load..')
            break

Possible duplicate of [Selenium "selenium.common.exceptions.NoSuchElementException" when using Chrome](https://stackoverflow.com/questions/47993443/selenium-selenium-common-exceptions-nosuchelementexception-when-using-chrome) — undetected Selenium, Apr 27 '18 at 07:48

score 0 · Answer 1 · answered Apr 27 '18 at 21:50

The reason why you keep receiving NoSuchElementException is that the dropdown is being inserted dynamically into DOM tree so you need to wait for some time until it is available. The best way to implement wait timeout is to use wait API provided by python-selenium.

Here is a working code where time.sleep being used as a temporary solution.

from selenium.webdriver.support.ui import Select
from selenium import webdriver
import time


driver = webdriver.Chrome()
driver.get("https://portal.orio.com/webapp/wcs/stores/servlet/ProductDisplay?urlRequestType=Base&productId=218044&catalogId=10051&categoryId=146003&errorViewName=ProductDisplayErrorView&urlLangId=-150&langId=-150&top_category=146001&parent_category_rn=146001&storeId=11901")

drop=driver.find_element_by_xpath('//*[@id="toggleMakeModelArrow"]')

time.sleep(5)
drop.click()
time.sleep(5)
select = Select(driver.find_element_by_class_name('orioVehicleMakeModeltypeInfo'))

for select_option in select.options:
    print(select_option.text)

Why selenium's click method is returning NoSuchElementException?

1 Answers1