Clicking Next Page Using Selenium

Question

I have tried a number of ways, from xpath to link text to get Selenium to locate the "Next Page" button, and then click until the last page, but just couldn't get this to work.

This is the error that I have using xpath:

no such element: Unable to locate element: {"method":"xpath","selector":"//li[@class="next"]/a"} (Session info: chrome=79.0.3945.88)

Code:

import requests, bs4, time, selenium  #import libraries
from selenium import webdriver
from selenium.webdriver.support.select import Select

driver=webdriver.Chrome()
driver.get('https://egov2.manchesternh.gov/Click2GovTX/accountsearch.html')

select=Select(driver.find_element_by_id('searchMethod'))
select.select_by_value('2')

streetName=driver.find_element_by_id('addressName')
time.sleep(1)
streetName.clear()
streetName.send_keys("A")
streetName.send_keys(u'\ue007')

url=driver.current_url
print(url)
driver.get(url)
nxt=driver.find_element_by_xpath('//li[@class="next"]/a')
nxt.click() 
time.sleep(1)

joonghyup cha · Answer 1 · 2019-12-28T03:07:55.940

1

As you can see in the next picture, there are two elements in the page that are identical to the xpath syntax you defined.

In order to get to the second value, you need to define the xpath from the parent div and nav

import requests, bs4, time, selenium  #import libraries
from selenium import webdriver
from selenium.webdriver.support.select import Select

driver=webdriver.Chrome()
driver.get('https://egov2.manchesternh.gov/Click2GovTX/accountsearch.html')

select=Select(driver.find_element_by_id('searchMethod'))
select.select_by_value('2')

streetName=driver.find_element_by_id('addressName')
time.sleep(1)
streetName.clear()
streetName.send_keys("A")
streetName.send_keys(u'\ue007')

nxt=driver.find_element_by_xpath('//div[@class="panel-body"]/nav[2]//li[@class="next"]/a')
nxt.click() 
time.sleep(1)

edited Dec 28 '19 at 03:07

answered Dec 28 '19 at 02:43

joonghyup cha

624
3
12

Ok that makes sense. How did you find out that there are two elements in the page that are identical the xpath that I defined? Did you check for duplications as the first step? Just wanted to see how I can avoid the same problem going forward. – Fatchoco Dec 28 '19 at 03:05
and you verify your `xpath` by this way. https://yizeng.me/2014/03/23/evaluate-and-validate-xpath-css-selectors-in-chrome-developer-tools/ please read `From Console panel` section – joonghyup cha Dec 28 '19 at 03:10

score 0 · Answer 2 · answered Dec 29 '19 at 22:08

To get Selenium to locate the Next Page » and then click until the last page you need to induce WebDriverWait for the element_to_be_clickable() and you can use the following Locator Strategies:

Code Block:

    from selenium import webdriver
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support import expected_conditions as EC
    from selenium.webdriver.support.ui import Select
    from selenium.common.exceptions import TimeoutException

    options = webdriver.ChromeOptions() 
    options.add_argument("start-maximized")
    options.add_experimental_option("excludeSwitches", ["enable-automation"])
    options.add_experimental_option('useAutomationExtension', False)
    driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
    driver.get('https://egov2.manchesternh.gov/Click2GovTX/accountsearch.html')
    select = Select(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "select#searchMethod"))))
    select.select_by_value('2')
    streetName = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input.form-control[name='addressName']")))
    streetName.send_keys("A")
    streetName.send_keys(u'\ue007')
    while True:
        try:
            WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//p//following::nav[2]//ul//li[@class='next']//a"))).click()
            print("Clicked on  Next Page »")
        except TimeoutException:
            print("No more Next Page »")
            break
    driver.quit()

Console Output:

    Clicked on  Next Page »
    Clicked on  Next Page »
    Clicked on  Next Page »
    Clicked on  Next Page »
    Clicked on  Next Page »
    .
    .
    .
    No more Next Page »

Clicking Next Page Using Selenium

2 Answers2