python - How to click "next" in Selenium until it's no longer available?

Question

I would like to scrape some data on a page and extract table from each page to a dataframe, then click on the "next" button and scrape some data on the second page until it's no longer available. How can I make it with this html structure:

<div class="dataTables_paginate paging_bootstrap pagination">
 <ul>
  <li class="active"><a href="javascript:void(0);">First</a></li>
  <li class="active"><a href="javascript:void(0);">Prev</a></li>
  <li p="1" class="active" onclick="loadDataByPage(1);">
   <a href="javascript:void(0);" style="color:#0D88CB">1</a>
  </li>
  <li p="2" class="inactive" onclick="loadDataByPage(2);">
   <a href="javascript:void(0);">2</a>
  </li>
  <li p="3" class="inactive" onclick="loadDataByPage(3);">
   <a href="javascript:void(0);">3</a>
  </li>
  <li p="2" class="inactive" onclick="loadDataByPage(2);">
   <a href="javascript:void(0);" style="color:#0D88CB">Next</a>
  </li>
  <li onclick="loadDataByPage(490);" class="inactive">
   <a href="javascript:void(0);" style="color:#0D88CB">Last</a>
  </li>                
 </ul>
</div>

can you please provide link of page? To click on link next you can use `driver.find_element(By.XPATH, '//a[contains(text(), "Next")]').click()` and to do it until it's no longer available just use while loop in which you put `try: driver.find_element(By.XPATH, '//a[contains(text(), "Next")]').click()` and if it's there than it will succeed, but if not then it won't. — ands, Jun 27 '16 at 11:13

score 1 · Answer 1 · answered Jun 27 '16 at 13:47

You should use looping as below :-

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 10)

while True:
# scrape some data on a page and extract table

# click next link
try:
    element = wait.until(EC.visibility_of_element_located((By.XPATH, "//a[contains(text(), 'Next')]")))
    element.click()
except TimeoutException:
    break

Hope it will help you..:)

python - How to click "next" in Selenium until it's no longer available?

1 Answers1

Linked