How to locate the dropdown select element and select an option within a complex website using Selenium and Python

Question

I have a very complex website which I am trying to test with Selenium. But when I try to get the XPath, I get like this for example.

//*[@id="datatable1595356931082"]/div[1]/div[2]/table/tbody/tr[3]/td[2]/div/select/option[8]

Absolute XPath:

/html/body/div[4]/div[2]/div[2]/div/div[2]/div[2]/div/div/div[1]/div[2]/table/tbody/tr[3]/td[2]/div/select/option[8]

in Selenium I tried with absolute path like

driver.find_element_by_xpath("/html/body/div[4]/div[2]/div[2]/div/div[2]/div[2]/div/div/div[1]/div[2]/")

and it tries to error out saying it is unable to find XPath. The datatable seems to create a dynamic number during runtime. What here I am trying to do is to select a drop down which looks like this

<div role="columnheader" class="webix_hcell webix_ss_filter">
<select>
<option value=""></option>
<option value="A">A</option>
<option value="B">B</option>
.
.
.
</select>
</div>

I have also tried

driver.find_element_by_class_name('webix_hcell webix_ss_filter')

But that too errors out

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".webix_hcell webix_ss_filter"}

For the above one

selenium.common.exceptions.InvalidSelectorException: Message: invalid selector: Unable to locate an element with the xpath expression /html/body/div[4]/div[2]/div[2]/div/div[2]/div[2]/div/div/div[1]/div[2]/ because of the following error:
SyntaxError: Failed to execute 'evaluate' on 'Document': The string '/html/body/div[4]/div[2]/div[2]/div/div[2]/div[2]/div/div/div[1]/div[2]/' is not a valid XPath expression.
  (Session info: chrome=84.0.4147.89)

score 0 · Accepted Answer · answered Jul 22 '20 at 04:27

This won't work, because *_by_class_name is only for single class name:

driver.find_element_by_class_name('webix_hcell webix_ss_filter')

Try using *_by_css_selector:

element = driver.find_element_by_css_selector('div.webix_hcell.webix_ss_filter select')

score 0 · Answer 2 · answered Jul 22 '20 at 09:15

There are a couple of things you need to consider as follows:

From your first attempt the WebElement appears as a dynamic element due to presence of the id attribute value as datatable1595356931082, which possibly will change on every access or periodically:
```
//*[@id="datatable1595356931082"]/div[1]/div[2]/table/tbody/tr[3]/td[2]/div/select/option[8]
```
In your second attempt you have used an absolute xpath. As the website is dynamic, elements will be repositioned on every access or periodically:
```
/html/body/div[4]/div[2]/div[2]/div/div[2]/div[2]/div/div/div[1]/div[2]/table/tbody/tr[3]/td[2]/div/select/option[8]
```
In your third attempt the xpath ends with / which isn't desired. Hence you face InvalidSelectorException
In your forth attempt you have passed multiple classes through driver.find_element_by_class_name('webix_hcell webix_ss_filter') where as driver.find_element_by_class_name() accepts only one classname as an argument.

You can find a detailed discussion in Invalid selector: Compound class names not permitted error using Selenium

Finally, it's a <select> node, so you need to use the Select class.

You can find a detailed discussion in How to select a drop-down menu value with Selenium using Python?

Solution

The relevant text based HTML would have helped us to construct a canonical answer. However as per the HTML provided to click on the option with text as A you can use the following either of the following Locator Strategies:

Using XPATH and select_by_visible_text():

select = Select(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[@class='webix_hcell webix_ss_filter' and @role='columnheader']//following::select[1]"))))
select.select_by_visible_text('A')

Using CSS_SELECTOR and select_by_value():

select = Select(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "div.webix_hcell.webix_ss_filter[role='columnheader'] +select"))))
select.select_by_value('A')

Note : You have to add the following imports :

from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

How to locate the dropdown select element and select an option within a complex website using Selenium and Python

2 Answers2

Solution