Selenium Web Scraping (Implementing Filters)

Question

I am trying to scrape Federal reserve's press releases on https://www.federalreserve.gov/newsevents/pressreleases.htm, but have to impose some filters before I can actually scrape stuff.

In particular, have to do the following:

1. Select Monetary Policy under Filters (it works thanks to the suggestions here)
2. Specify a date range under Date (from 01/01/2010 to 12/31/2021)
3. Finally, click on Submit button once both the filters are imposed.

For 2., I tried the following but it returned a NoSuchElementException.

 date_input = driver.find_element_by_xpath('//*[@id="content"]/div[2]/div/div[1]/form/div[2]/div/div[1]/input')
 date_input.click()                      # Focus input field
 date_input.send_keys(Keys.CONTROL, "a") # Select all pre-existing text/input value
 date_input.send_keys(Keys.BACKSPACE)    # Remove that text
 date_input.send_keys("01012010")        # Add desired text/set input value

For 3., I modified the code from here and tried the following:

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.LINK_TEXT, "Submit"))).click() #submit the filter

and

WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH ,"//*[@id='content']/div[2]/div/div[1]/form/div[5]/a"))).click()

but both don't work.

Any ideas? Thanks!

Answer 1

The element with text as Submit is an <a> tag

<a href="#">
    "Submit"
    <span class="icon icon__sm icon--right icon-more"></span>
</a>

---

Solution

To implement the filters and click on the Submit element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:

driver.get("https://www.federalreserve.gov/newsevents/pressreleases.htm")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@aria-label='Checkbox for Monetary Policy']"))).click()
driver.find_element(By.XPATH, "//input[@ng-model='datepicker.startDate']").send_keys("01/01/2010")
driver.find_element(By.XPATH, "//input[@ng-model='datepicker.endDate']").send_keys("12/31/2021")
driver.find_element(By.XPATH, "//a[contains(., 'Submit')]").click()

Note: You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC