7

This is a follow up question from my first question, and I am trying to scrape a website and have Selenium click on next (until it cant be clicked) and collect the results as well.

This is the html tag from the website with the button:

<div class="pager results-display">
  <span class="action-btn prev inactive" data-page="1">Prev</span>
  <span class="action-btn inactive" data-page="1">1</span>  # Currently in page 1 thus inactive
  <span class="action-btn" data-page="2">2</span>
  <span class="action-btn" data-page="3">3</span> 
  <span class="action-btn" data-page="4">4</span>
  <span class="action-btn" data-page="5">5</span>
  <span class="action-btn" data-page="6">6</span>
  <span class="action-btn" data-page="7">7</span>
  <span class="action-btn" data-page="8">8</span> 
  <span class="action-btn" data-page="9">9</span>
  <span class="action-btn" data-page="10">10</span>
  <span class="action-btn" data-page="11">11</span> 
  <span class="action-btn" data-page="12">12</span>
  <span class="action-btn" data-page="13">13</span>
  <span class="action-btn" data-page="14">14</span>
  <span class="action-btn next" data-page="2">Next</span>
</div>
<div class="no-results-display hidden" style="display: none;">
                    No companies matched your search. Try again.
                </div>

I've tried this code:

elm = driver.find_element_by_name('pager results-display')
elm.click()

And I've also checked this question out, but still wasn't able to solve it.

Any thoughts?

Guy
  • 46,488
  • 10
  • 44
  • 88
jake wong
  • 4,909
  • 12
  • 42
  • 85

1 Answers1

10

pager results-display are actually two classes that belongs to the button's parent element. They aren't name attribute, and find_element_by_name gets only one name anyway. Try

elm = driver.find_element_by_class_name('next')
elm.click()

Notice that you might have to relocate the button each time if the DOM has changed after the click.

To click in loop as long as the button is active you can check if the buuton has class inactive

while True:
    elm = driver.find_element_by_class_name('next')
    if 'inactive' in elm.get_attribute('class'):
        break;
    elm.click()
Guy
  • 46,488
  • 10
  • 44
  • 88
  • Hello, this works. But how can I structure this into a loop that clicks on next until it ends? From what I observed, when the page is at the last page, `next` is still there, but the `html code` changes to `Next` – jake wong Mar 27 '16 at 07:45
  • @jakewong I added example in my answer. – Guy Mar 27 '16 at 07:56
  • Thank you very much. This gives me what I'm looking for :) – jake wong Mar 27 '16 at 08:11