Selenium Scraping: Clicking on the Next Button

Question

I am trying to scrape Federal reserve's press releases on https://www.federalreserve.gov/newsevents/pressreleases.htm, and to scrape documents from previous years, I need to move onto the next page by clicking on Next button at the bottom of the page.

I have tried a few things, but all of them return a Message: no such element: Unable to locate element: error, and I can't figure out the issue:

#Attempt 1
    driver.find_element_by_css_selector('.pagination-next ng-scope').click()
    WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".pagination-next ng-scope"))).click()

#Attempt 2
    driver.find_element_by_xpath("//*[@id='article']/ul[2]/li[7]/a").click()

#Attempt 3
    element=WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH ,"//*[@id='article']/ul[2]/li[7]/a")))
    element.click()

Could anyone help me with where I am going wrong? For ease, the HTML for the Next button is:

<li ng-if="::directionLinks" ng-class="{disabled: noNext()||ngDisabled}" class="pagination-next ng-scope"><a href="" ng-click="selectPage(page + 1, $event)" ng-disabled="noNext()||ngDisabled" uib-tabindex-toggle="" class="ng-binding">Next</a></li>
     <a href="" ng-click="selectPage(page + 1, $event)" ng-disabled="noNext()||ngDisabled" uib-tabindex-toggle="" class="ng-binding">Next</a>

Thanks

Can you try this code perhaps: `WebDriverWait(driver, 10).until( EC.element_to_be_clickable((By.XPATH, "//*[@id='article']/ul[2]/li[7]/a")) ).click()` — helloworld, Apr 29 '22 at 21:51

undetected Selenium · Answer 1 · 2022-04-29T22:05:03.490

0

The element with text as Next is an <a> tag and is an Angular element.

<a href="" ng-click="selectPage(page + 1, $event)" ng-disabled="noNext()||ngDisabled" uib-tabindex-toggle="" class="ng-binding">Next</a>

Solution

To click element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:

Using LINK_TEXT:

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.LINK_TEXT, "Next"))).click()

Using CSS_SELECTOR:

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "li.pagination-next.ng-scope a.ng-binding"))).click()

Using XPATH:

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[text()='Next']"))).click()

Note: You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

edited Apr 29 '22 at 22:05

answered Apr 29 '22 at 21:53

undetected Selenium

183,867
41
278
352

It works, thanks! Two follow-up questions: how do I click on "Submit" after filtering by Monetary Policy? I modified your solutions above and tried `(By.LINK_TEXT, "Submit")` and `(By.XPATH ,"//*[@id='content']/div[2]/div/div[1]/form/div[5]/a")` but both of these don't work. Any ideas? – AYA Apr 30 '22 at 02:40
I am also looking for specifying the dates (on the same page). Particularly, I want to restrict observations from 01/01/2010 to 12/31/2021. Sorry, I am new to Selenium, and it is a bit confusing. Thanks! – AYA Apr 30 '22 at 02:52
@AYA Sounds like e new requirement all together. Feel free to raise a new question as per your new requirement. – undetected Selenium Apr 30 '22 at 04:39
I posted it here `https://stackoverflow.com/questions/72071384/selenium-web-scraping-implementing-filters` – AYA Apr 30 '22 at 19:05

Selenium Scraping: Clicking on the Next Button

1 Answers1

Solution

Linked