1

I am trying to scrape off-campus apartment names and addresses for an apartment listing website for universities. My current obstacle is clicking a "Load More" Button in selenium that can appear x number of times per search result. I am new to webscraping, and I was wondering if someone could help me click this button until it is gone, so I then can gather all of the other information using beautifulsoup.

Website: https://www.rentcollegepads.com/off-campus-housing/mississippi-state/search

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
BQuist
  • 171
  • 7

2 Answers2

2

To click on the element Load More you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:

  • Using CSS_SELECTOR:

    WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.btn.btn-success.btn-block > i.fa.fa-refresh"))).click()

  • Using XPATH:

    WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[@class='btn btn-success btn-block' and contains(., 'Load More')]"))).click()

  • Note : You have to add the following imports :

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
  • Hi thanks for this! I used the XPATH option, and it works, but it only click the button once. In this scenario, I need it to click it 2 times, but in other scenarios I might need it to click 10 times for example. Would you know how to do that? – BQuist Aug 04 '22 at 14:36
  • You have got it all, the `xpath` to `click` with `WebDriverWait` and _and it works_. You can loop it as many times you want, not onlt 2 times but 10, 50, 100 times. If you need help for the loop, feel free to raise a new question with your new requirement. – undetected Selenium Aug 04 '22 at 14:40
0

I used another thread (How to loop a python action until a button is clickable (selenium)) and changed it a bit as I had a similar task recently:

you need to add:

x_path="//button[@class='btn btn-success btn-block' and contains(., 'Load More')]"
while True:
    try:
        time.sleep(5)
        ActionChains(driver).move_to_element(driver.find_element(By.XPATH, xpath)).perform()

        if driver.find_element(By.XPATH, x_path).is_enabled():
            driver.find_element(By.XPATH, x_path).click()
            print('clicked')
        else:
            driver.refresh()
    except Exception as e:
        print(f'{e}')
        break
Bilyana
  • 1
  • 1