Selenium unable to fetch the select results

Question

I am new to selenium and trying to get the values of the select tag from a site (link in code), but whenever I try to get the select tag it returns an empty field.

My Code


from selenium import webdriver
from selenium.webdriver.support.ui import Select

driver = webdriver.Chrome('chromedriver',chrome_options=chrome_options)

url = 'https://lifeinsurance.adityabirlacapital.com/about-us/public-disclosure'
driver.get(url)

select = Select(driver.find_element_by_id("select-public-year"))

print([option.text for option in select.options])
# Output: ['']

When I looked at the page source it shows me the following:

<select Name="selectPublicYear" class="demo-default select selectized selectPublicYear" id="select-public-year" name="SelectedYear">
  <option value="2020-2021">2020-2021</option>
  <option value="2019-2020">2019-2020</option>
  <option value="2018-2019">2018-2019</option>
  <option value="2017-2018">2017-2018</option>
  <option value="2016-2017">2016-2017</option>
  <option value="2015-2016">2015-2016</option>
  <option value="2014-2015">2014-2015</option>
  <option value="2013-2014">2013-2014</option>
  <option value="2012-2013">2012-2013</option>
  <option value="2011-2012">2011-2012</option>
  <option value="2010-2011">2010-2011</option>
  <option value="2009-2010">2009-2010</option>
  <option value="2008-2009">2008-2009</option>
  <option value="2007-2008">2007-2008</option>
  <option value="2006-2007">2006-2007</option>
  <option value="2005-2006">2005-2006</option>
</select>

Any hints or help will be appreciated.

Probably those options are not loaded to DOM unless you expand the list. So try to all clicking that select to expand it and then fetch the options again. — Alexey R., Feb 11 '21 at 17:22
`AttributeError: 'Select' object has no attribute 'click'` It gave the following error — Shaikh Abuzar, Feb 11 '21 at 17:37
Try something like first `driver.find_element_by_id("select-public-year").click()` and then `select = Select(driver.find_element_by_id("select-public-year"))` — Alexey R., Feb 11 '21 at 17:40
`ElementNotInteractableException: Message: element not interactable (Session info: headless chrome=87.0.4280.66)` It gave this error, for `driver.find_element_by_id("select-public-year").click()` — Shaikh Abuzar, Feb 11 '21 at 17:43
The page may be using bootstrap or something similar so what the user will be interactive with is another set of web elements altogether. That `select` would be hidden if so. — , Feb 11 '21 at 17:47

KunduK · Answer 1 · 2021-02-11T18:29:57.223

0

This seems synchronization issue. I would suggest use WebDriverWait() and wait for element_to_be_clickable()

select = Select(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.ID,"select-public-year"))))

print([option.text for option in select.options])

You need to import below library

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

Snapshot tested using java since I don't have python in working machine.

edited Feb 11 '21 at 18:29

answered Feb 11 '21 at 17:56

KunduK

32,888
5
17
41

`TimeoutException: Message:` It gave the following error – Shaikh Abuzar Feb 11 '21 at 18:11
@ShaikhAbuzar : You may some other issue similar code I have run with java and its working perfectly. Please see the attachment. – KunduK Feb 11 '21 at 18:29
Please have a look at the link I have created a colab notebook: [Colab Link](https://colab.research.google.com/drive/1NO7zmrQeVTxwlZyfXASsgBiuAskdmfk4?usp=sharing) – Shaikh Abuzar Feb 11 '21 at 18:37

score 0 · Accepted Answer · answered Feb 11 '21 at 18:49

The <select> tag is having the attribute style="display: none;" so that's not your desired element.

To extract the texts e.g. 2020-2021, from all of the <class="option"> using Selenium and python you have to induce WebDriverWait for visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

Using CSS_SELECTOR:

driver.get('https://lifeinsurance.adityabirlacapital.com/about-us/public-disclosure')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.selectize-input.items.full.has-options.has-items"))).click()
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.selectize-dropdown-content div.option")))])

Using XPATH:

driver.get('https://lifeinsurance.adityabirlacapital.com/about-us/public-disclosure')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[@class='selectize-input items full has-options has-items']"))).click()
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='selectize-dropdown-content']//div[contains(@class, 'option')]")))])

Console Output:

['2020-2021', '2019-2020', '2018-2019', '2017-2018', '2016-2017', '2015-2016', '2014-2015', '2013-2014', '2012-2013', '2011-2012', '2010-2011', '2009-2010', '2008-2009', '2007-2008', '2006-2007', '2005-2006']

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Thanks and worked like a charm. I wanted to know now if I wish to select a year and a quarter and see the results of the selection and then download all the files that are shown. Can you share some references on how should I go about it? — Shaikh Abuzar, Feb 11 '21 at 19:10
@ShaikhAbuzar A lot would depend on how the elements to download all the files are defined. If they are static, you can click in a loop. But if they contains `__doposback` you have tweak the clicking strategy a bit. — undetected Selenium, Feb 11 '21 at 19:13
Could you please refer me to some good materials on static download approach. — Shaikh Abuzar, Feb 11 '21 at 19:24
@ShaikhAbuzar See [this](https://stackoverflow.com/questions/45097302/download-and-save-multiple-csv-files-using-selenium-and-python-from-popup/45098422#45098422) discussion. — undetected Selenium, Feb 11 '21 at 19:28
How can I get the values of the quarter section after the year now — Shaikh Abuzar, Feb 11 '21 at 19:37
@ShaikhAbuzar Sounds like a different issue all together. Can you raise a new question as per your new requirement? Stackoverflow contributors will be happy to help you out. — undetected Selenium, Feb 11 '21 at 19:59

Selenium unable to fetch the select results

2 Answers2