Selenium. Unable to locate element from the html website

Question

Here's the link of the website I'm trying to scrape (I'm training for the moment, nothing fancy):

link

Here's my script, he's quite long but nothing too complicated :

from selenium import webdriver



if __name__ == "__main__":
    print("Web Scraping application started")

    PATH = "driver\chromedriver.exe"


    options = webdriver.ChromeOptions() 
    options.add_argument("--disable-gpu")
    options.add_argument("--window-size=1200,900")
    options.add_argument('enable-logging')

    driver = webdriver.Chrome(options=options, executable_path=PATH)

    driver.get('https://fr.hotels.com/')
    driver.maximize_window()

    destination_location_element = driver.find_element_by_id("qf-0q-destination")
    check_in_date_element = driver.find_element_by_id("qf-0q-localised-check-in")
    check_out_date_element = driver.find_element_by_id("qf-0q-localised-check-out")


    search_button_element = driver.find_element_by_xpath('//*[@id="hds-marquee"]/div[2]/div[1]/div/form/div[4]/button')

    print('Printing type of search_button_element')
    print(type(search_button_element))

    destination_location_element.send_keys('Paris')
    check_in_date_element.clear()
    check_in_date_element.send_keys("29/05/2021")
    check_out_date_element.clear()
    check_out_date_element.send_keys("30/05/2021")



    close_date_window = driver.find_element_by_xpath('/html/body/div[7]/div[4]/button')

    print('Printing type of close_date_window')
    print(type(close_date_window))

    close_date_window[0].click()

    search_button_element.click()

    time.sleep(10)

    hotels = driver.find_element_by_class_name('hotel-wrap')

    print("\n")

    i = 1

    for hotel in hotels:

        try:
            print(hotel.find_element_by_xpath('//*[@id="listings"]/ol/li['+str(i)+']/article/section/div/h3/a').text)
            print(hotel.find_element_by_xpath('//*[@id="listings"]/ol/li[' + str(i) + ']/article/section/div/address/span').text)

        except Exception as ex:
            print(ex)
            print('Failed to extract data from element.')

        i = i +1

        print('\n')

    driver.close()

    print('Web Scraping application completed')

And here's the error I get :

  File "hotelscom.py", line 21, in <module>
    destination_location_element = driver.find_element_by_id("qf-0q-destination")
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="qf-0q-destination"]"}
  (Session info: chrome=90.0.4430.85)

Any idea how to fix that ? I don't understand why it get me this error because in the html code, there is this syntax. But i guess I'm wrong.

Answer 1

You have multiple problems with your code and the site.

SITE PROBLEMS

1 The site is located on multiple servers and different servers have different html code. I do not know if it depends on location or not.

2 The version I have solution for has few serious bugs (or maybe those are features). Among them:

• When you press Enter it starts hotels search when a date field is opened and and you just want to close this date field. So, it is a problem to close input fields in a traditional way.
• clear() of Selenium does not work as it is supposed to work.

BUGS IN YOUR CODE

1 You are defining window size in options and you are maximizing the window immediately after site is opened. Use only one option

2 You are entering dates like "29/05/2021", but sites recognises formats only like: "05/30/2021". It is a big difference

3 You are not using any waits and they are extremely important.

4 Your locators are wrong and unstable. Even locators with id did not always work for me because if you will make a search, there are two elements for some of them. So I replaces them with css selectors.

Please note that my solution works only for an old version of site. If you want to a specific version to be opened you will need either:

• Get the site by a direct ip address, like driver.get('site ip address')
• Implement a strategy in your framework which recognises which site version is opened and applies inputs depending on it.

SOLUTION

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait


if __name__ == "__main__":
    print("Web Scraping application started")
    options = webdriver.ChromeOptions()
    options.add_argument("--disable-gpu")
    options.add_argument("--window-size=1200,900")
    options.add_argument('enable-logging')

    driver = webdriver.Chrome(options=options, executable_path='/snap/bin/chromium.chromedriver')

    driver.get('https://fr.hotels.com/')
    wait = WebDriverWait(driver, 15)

    wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#qf-0q-destination")))
    destination_location_element = driver.find_element_by_css_selector("#qf-0q-destination")
    destination_location_element.send_keys('Paris, France')
    wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".widget-autosuggest.widget-autosuggest-visible table tr")))
    destination_location_element.send_keys(Keys.TAB)  # workaround to close destination field
    driver.find_element_by_css_selector(".widget-query-sub-title").click()
    wait.until(EC.invisibility_of_element_located((By.CSS_SELECTOR, ".widget-query-group.widget-query-destination [aria-expanded=true]")))

    wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#qf-0q-localised-check-in")))
    check_in_date_element = driver.find_element_by_css_selector("#qf-0q-localised-check-in")
    check_in_date_element.send_keys(Keys.CONTROL, 'a')  # workaround to replace clear() method
    check_in_date_element.send_keys(Keys.DELETE)  # workaround to replace clear() method
    # check_in_date_element.click()
    check_in_date_element.send_keys("05/30/2021")

    # wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "#qf-0q-localised-check-out")))
    check_out_date_element = driver.find_element_by_id("qf-0q-localised-check-out")
    check_out_date_element.click()
    check_out_date_element.send_keys(Keys.CONTROL, 'a')
    check_out_date_element.send_keys(Keys.DELETE)
    check_out_date_element.send_keys("05/31/2021")
    driver.find_element_by_css_selector(".widget-query-sub-title").click()  # workaround to close end date
    wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#hds-marquee button"))).click()

Spent on this few hours, the task seemed just interesting for me. It works for this UI:

The code can still be optimised. It's up to you.

UPDATE:

I found out that the site has at least three home pages with three different Destination and other fields locators. The easiest workaround that came into my mind is something like this:

try:
    element = driver.find_element_by_css_selector("#qf-0q-destination")
    if element.is_displayed():
        wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#qf-0q-destination")))
        destination_location_element = driver.find_element_by_css_selector("#qf-0q-destination")
        print("making input to Destination field of site 1")
        destination_location_element.send_keys('Paris, France')
        # input following data
except:
    print("Page 1 not found")
try:
    element = driver.find_element_by_css_selector("input[name=q-destination-srs7]")
    if element.is_displayed():
        wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[name=q-destination-srs7]")))
        destination_location_element = driver.find_element_by_css_selector("input[name=q-destination-srs7]")
        print("making input to Destination field of site 2")
        destination_location_element.send_keys('Paris, France')
        # input following data
except:
    print("Page 2 is not found")
try:
    element = driver.find_element_by_css_selector("form[method=GET]>div>._1yFrqc")
    if element.is_displayed():
        wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "form[method=GET]>div>._1yFrqc")))
        destination_location_element = driver.find_element_by_css_selector("form[method=GET]>div>._1yFrqc")
        print("making input to Destination field of site 3")
        destination_location_element.send_keys('Paris, France')
        # input following data
except:
    print("Page 3 is not found")

But the best solution would be to have a direct access to a specific server that has only one version available.

Please also note that if you access the site by a direct link for France: https://fr.hotels.com/?pos=HCOM_FR&locale=fr_FR your input dates will be as you initially specified, for example 30/05/2021.

Answer 2

Try this

driver.find_element_by_xpath(".//div[contains(@class,'destination')]/input[@name='q-destination']")

Also please add wait after you maximize the window

Answer 3

You are missing a wait / sleep before finding the element.
So, just add this:

element = WebDriverWait(driver, 20).until(
EC.element_to_be_clickable((By.ID, "qf-0q-destination")))
element.click()

to use this you will have to use the following imports:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as E