Python: selenium click next button

Question

I am using Python Selenium with ChromeDriver, all are up to date. I am trying click on "Next" button until the last page.

I have tried myself but after few page clicks the script breaks or stops from clicking further. I have made some edits and lost the partial working code.

Here is the html code.

html at the beginning of the page:

<div class="pagination" total="2098" limit="20" offset="1" view="products">
  <ul>
    <li class="disabled page">First
    </li>
    <li class="disabled page">Prev
    </li>
    <li key="1" class="pageLink digital current page">1
    </li>
    <li key="2" class="pageLink digital page">2
    </li>
    <li key="3" class="pageLink digital page">3
    </li>
    <li key="2" class="pageLink page">Next
    </li>
    <li key="105" class="pageLink page">Last
    </li>
  </ul>
</div>

html at the last page:

<div class="pagination" total="6866" limit="20" offset="344" view="products"><ul><li key="1" class="pageLink page">First</li><li key="343" class="pageLink page">Prev</li><li key="342" class="pageLink digital page">342</li><li key="343" class="pageLink digital page">343</li><li key="344" class="pageLink digital current page">344</li><li class="disabled page">Next</li><li class="disabled page">Last</li></ul></div>

Edit:

Python code I tried.

while True:    
next_page_btn = None
next_page_btn = browser.find_elements_by_xpath("/html[1]/body[1]/div[1]/div[2]/div[2]/div[6]/div[1]/div[1]/div[3]/div[1]/div[1]/ul[1]/li[6]")
if len(next_page_btn) < 1:
    print("No more pages left")
    break
else:
    
    element = WebDriverWait(browser, 10).until( 
    EC.presence_of_element_located((By.xpath, "/html[1]/body[1]/div[1]/div[2]/div[2]/div[6]/div[1]/div[1]/div[3]/div[1]/div[1]/ul[1]/li[6]")) 
    )
    element.click()

EDIT 2: Below code is what I am using so far it is working fine except one issue. Even after the last page it keep continuing loading the last page without end. How do we stop when it reaches at the end.

while True:
time.sleep(5)
#wait for pagination to show 
EC.presence_of_element_located((By.XPATH, "//div[contains(@class, 'pagination')]")) 
next_page_btn = browser.find_elements_by_xpath("//div[contains(@class, 'pagination')]//li[contains(text(), 'Next')]")
if len(next_page_btn) < 1:
    print("No more pages left")
    break
else:
    WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.XPATH, "//li[.='Next']"))).click()

This might be helpful [wait](https://stackoverflow.com/questions/26566799/wait-until-page-is-loaded-with-selenium-webdriver-for-python) — Ice Bear, Dec 15 '20 at 16:55
@DMart there are still pages to load but script stops working halfway or next button shows as disabled even though there are still pages to process. — ACE, Dec 15 '20 at 17:32
I have made an edit, Please refer to the Last page html I believe it may help to determine we have reached the last page. — ACE, Dec 16 '20 at 07:01

score 0 · Answer 1 · answered Dec 15 '20 at 16:50

0

Normally the pagination state (the page number) is stored in the URL as a query string, You can simply use a counter and loop through the URLs without worrying about targeting the element correctly

answered Dec 15 '20 at 16:50

Yuvraj Singh

105
6

DMart · Answer 2 · 2020-12-16T04:31:57.453

0

I would use this xpath to identify the next button:

//div[contains(@class, 'pagination')]//li[contains(text(), 'Next')]

while True:   
    #wait for pagination to show 
    EC.presence_of_element_located((By.XPATH, "//div[contains(@class, 'pagination')]")) 
    )
next_page_btn = browser.find_elements_by_xpath("//div[contains(@class, 'pagination')]//li[contains(text(), 'Next')]")
if len(next_page_btn) < 1:
    print("No more pages left")
    break
else:
    element.click()

Or you could simplify by doing try catch:

while True:
    EC.presence_of_element_located((By.XPATH, "//div[contains(@class, 'pagination')]"))
    try:
        driver.find_element_by_xpath("//div[contains(@class, 'pagination')]//li[contains(text(), 'Next')]").click()
    except NoSuchElementException:
        print("Element not found")
        break

edited Dec 16 '20 at 04:31

answered Dec 15 '20 at 17:32

DMart

2,401
1
14
19

I am getting following error for the first part of the code when I try to execute the code. `code EC.presence_of_element_located((By.xpath, "//div[contains(@class, 'pagination')]")) AttributeError: type object 'By' has no attribute 'xpath'` By the way, " ) " at 6th line shows error so I removed it. Please confirm if I need to remove it. – ACE Dec 15 '20 at 18:20
Second part of the code is also giving me error `code While True: ^ SyntaxError: invalid syntax ` – ACE Dec 15 '20 at 18:33
Looks like i had some syntax errors. Sorry! Try again. – DMart Dec 16 '20 at 04:31

score -1 · Answer 3 · answered Dec 15 '20 at 17:12

-1

try using "find_elements_by_link_text" instead of xpath

answered Dec 15 '20 at 17:12

Abdoullah al-hajj

45
9

Search by link text is not applicable to `li` nodes – JaSON Dec 15 '20 at 17:37

score -1 · Answer 4 · answered Dec 15 '20 at 17:36

Try this code to click Next until no pages left

browser.get(URL)

while True:
    ... <scraping code>...
    try:
        WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.XPATH, "//li[.='Next']"))).click()
    except:
        print('No more pages to load')
        break

Python: selenium click next button

4 Answers4