Python selenium click element while displayed

Question

I'm scraping a dynamic page that requires the user to click a "Load More Results" button several times in order to get all data. Is there a better way to approach the task of clicking on an element while displayed?

def clickon(xpath):
    try:
        element = driver.find_element_by_xpath(xpath)
    except:
        print "RETRYING CLICKON() for %s" % (xpath)
        time.sleep(1)
        clickon(xpath)
    else:
        element.click()
        time.sleep(3)

def click_element_while_displayed(xpath):
    element = driver.find_element_by_xpath(xpath)
    try:
        while element.is_displayed():
            clickon(xpath)
    except:
        pass

score 4 · Accepted Answer · edited May 23 '17 at 11:48

I suspect you are asking this question because the current solution is slow. This is mostly because you have these hardcoded time.sleep() delays which wait for more than they usually should. To tackle this problem I'd start using Explicit Waits - initialize an endless loop and break it once selenium stopped waiting for the button to be clickable:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

wait = WebDriverWait(driver, 10)

while True:
   try:
       element = wait.until(EC.element_to_be_clickable((By.XPATH, xpath)))
       element.click()
   except TimeoutException:
       break  # cannot click the button anymore

   # TODO: wait for the results of the click action

Now the last TODO part is also important - here I suggest you wait for a specific condition that would indicate that click resulted into something on a page - for instance, more results were loaded. For example, you can use a custom Expected Condition similar to this one.

Python selenium click element while displayed

1 Answers1