Selenium Driver Web Scraping is unable to get the drop-down element when multiple windows open

Question

I am web scraping a URL https://ephtracking.cdc.gov/DataExplorer/ using the below code

options = webdriver.ChromeOptions() 
options.headless = False

options.add_argument("window-size=1920,1080")
options.page_load_strategy = 'none'
options.add_argument("--enable-javascript")
options.add_argument('--ignore-certificate-errors') 

driver = Chrome(options=options, service=chrome_service) 

driver.get(url)
wait= WebDriverWait(driver,20)

step1Content_click = wait.until(ExpectedConditions.presence_of_element_located((
      By.XPATH,'//select[@id="contentArea"]//option[text()="Tornadoes"]')))
driver.execute_script("arguments[0].click();", step1Content_click)

However, it's unable to select the option Tornadoes from the drop-down list of the Step1 Content -> Select Content Area.

Answer 1

To select the select-options with text Tornadoes from the html-select tag you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:

• Using CSS_SELECTOR:

  driver.get("https://ephtracking.cdc.gov/DataExplorer/")
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "select#contentArea[data-qp-data-name='contentAreas']"))).click()
  Select(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "select#contentArea[data-qp-data-name='contentAreas']")))).select_by_visible_text("Tornadoes")
  

• Using XPATH:

  driver.get("https://ephtracking.cdc.gov/DataExplorer/")
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//select[@id='contentArea' and @data-qp-data-name='contentAreas']"))).click()
  Select(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//select[@id='contentArea' and @data-qp-data-name='contentAreas']")))).select_by_value('39')
  

• Note : You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  from selenium.webdriver.support.ui import Select
  

• Browser snapshot:

---

References

You can find a couple of relevant detailed discussions in:

• How to select an option from the dropdown menu using Selenium and Python

Answer 2

Check the working code below:

import time

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as ExpectedConditions
from selenium.webdriver.support.ui import Select

options = webdriver.ChromeOptions()
options.add_argument("window-size=1920,1080")
options.page_load_strategy = 'none'
options.add_argument("--enable-javascript")
options.add_argument('--ignore-certificate-errors')

driver = webdriver.Chrome(options=options)
driver.maximize_window()

driver.get("https://ephtracking.cdc.gov/DataExplorer/")
wait= WebDriverWait(driver,60)
select = Select(wait.until(ExpectedConditions.visibility_of_element_located((By.ID, "contentArea"))))
select.select_by_visible_text("Tornadoes")
time.sleep(30)

Result: