Selenium Driver unable to locate an element within a scroll bar list

Question

I am web scraping a URL https://wonder.cdc.gov/controller/datarequest/D149.

Within that page I am clicking on http://wonder.cdc.gov/natality-expanded-current.html and then I am clicking on I Agree button. In Section 10 there is a scroll bar list of Year, for which I am using below code

wait.until(ExpectedConditions.presence_of_element_located((By.XPATH,'//select[@id="SD149.V20"]//option[@value="2016"]'))).click()

However its not able to select the element. Please help on same.

Answer 1

Your code is working just fine, i.e. it is actually selecting value 2016 from dropdown. However what I notice when you click on 2016 is, it selects 2016 and 2021 both(see below).

So, to deselect 2021 and keep only 2016 selected, use the below code:

# use below line to deselect 2021
wait.until(EC.presence_of_element_located((By.XPATH,'//select[@id="SD149.V20"]//option[@value="2021"]'))).click()
# use below line to select 2016
wait.until(EC.presence_of_element_located((By.XPATH,'//select[@id="SD149.V20"]//option[@value="2016"]'))).click()

Result after adding the above code:

FYI : Another way to select the dropdown values in Selenium:

You can use selenium's Select class to handle dropdowns, see the code below:

from selenium.webdriver.support.ui import Select

select = Select(driver.find_element(By.ID, "SD149.V20"))
select.select_by_value("2016")

Answer 2

The desired element:

<option value="2016">&nbsp;2016&nbsp;</option>

is an <option> tag within it's parent <select> tag.

---

This usecase

Ideally to select an <option> from an html-select you need to use the Select() method inducing WebDriverWait for the visibility_of_element_located() and you can use the following Locator Strategies:

• Code block:

  driver.get("https://wonder.cdc.gov/controller/datarequest/D149")
  WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.w-sys-message p a[href*='natality']"))).click()
  WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[name='action-I Agree']"))).click()
  driver.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//td[@class='hd' and contains(., 'Select delivery characteristics')]"))))
  Select(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "select[id='SD149.V20']")))).deselect_all()
  Select(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "select[id='SD149.V20']")))).select_by_value("2016")
  

• Note : You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  

• Browser snapshot:

Note: The option with text 2016 does get selected but you need to scroll a bit up to verify it manually.