0

I am scraping roughly 1000 urls using selenium, and I am very close to having it work. Each url has a "load more" button that I continuously click until a Stale Element exception is thrown, which is passed. The scrape works great until a random video ad covers the button. I thought using XPATH to locate the button would solve the issue, but it does not. I know using java to execute the script will solve the issue, but I am not sure how to use on a looping basis. Below is what I have at the moment.

    if clicks >= 1:
        webdriver.ActionChains(driver).send_keys(Keys.ESCAPE).perform()
        sleep(1)
        try:
            while EC.element_to_be_clickable((By.XPATH, '/ html / body / div[2] / div / div / div[4] / div[4] / div / div / button')):
                button = driver.find_elements_by_xpath("/ html / body / div[2] / div / div / div[4] / div[4] / div / div / button")
                button[0].click()
                sleep(2)
                if not button.is_enabled():
                    print('ad is covering button, hopefully this works')
                    wait2 = WebDriverWait(driver, 10)
                    wait2.until(EC.element_to_be_clickable(By.XPATH("/ html / body / div[2] / div / div / div[4] / div[4] / div / div / button").click()))
                    
        except StaleElementReferenceException:
            pass

    else:
        print('under 10 reviews')
        pass

my hope was the if not line would save the day... but I am not sure if I am setting it correctly. I have also found this link may be helpful: Selenium-Debugging: Element is not clickable at point (X,Y) , but I do not know how to continuously click using the execute script feature. This link may also be helpful as well: WebDriver click() vs JavaScript click() Any help would be appreciated.

ColeGulledge
  • 393
  • 1
  • 2
  • 12

1 Answers1

1

You could simply rework it into this. Pretty sure you need a timeout except as well. In case it doesn't find it.

wait = WebDriverWait(driver, 10)
while True:
    try:
        elem=wait.until(EC.element_to_be_clickable((By.XPATH,"//button[.='load more']")))
        driver.execute_script("arguments[0].click();", elem)
    except StaleElementReferenceException:
        break
    except TimeOutException:
        break
Arundeep Chohan
  • 9,779
  • 5
  • 15
  • 32
  • Arundeep! Great to hear from you again. You have once again saved me. Also, if you'd like to edit your answer. I threw in a except StaleElementReferenceException:break, except TimeOutException:break, and it runs fine. Thanks again! – ColeGulledge Mar 11 '21 at 16:36