0

I'm trying to extract all ads from this page https://www.centris.ca/en/properties~for-rent?view=Thumbnail and then click on them to extract data.

For some reason, I can only extract the first entry despite using "find_elements".

Here is my code:

from selenium import webdriver
driver = webdriver.Chrome()
driver.get("https://www.centris.ca/en/properties~for-rent?view=Thumbnail")

list = driver.find_elements_by_xpath("//div[contains(@class, 'thumbnailItem col-12 col-sm-6 col-md-4 col-lg-3')]")
print(list)
for l in list:
    l.click()

What is the proper way to extract all elements with xpath (or other methods)?

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
Lucien S.
  • 5,123
  • 10
  • 52
  • 88

1 Answers1

1

To extract the href attribute of all the ads from this page https://www.centris.ca/en/properties~for-rent?view=Thumbnail you need to induce WebDriverWait for the visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

  • Using CSS_SELECTOR:

    driver.get("https://www.centris.ca/en/properties~for-rent?view=Thumbnail")
print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div[data-id='templateThumbnailItem'] a")))])

  • Using XPATH:

    driver.get("https://www.centris.ca/en/properties~for-rent?view=Thumbnail")
print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@data-id='templateThumbnailItem']//a")))])

  • Console Output:

    ['https://www.centris.ca/en/lofts-studios~for-rent~quebec-la-cite-limoilou/22499659?view=Summary', 'https://www.centris.ca/en/lofts-studios~for-rent~quebec-la-cite-limoilou/22499659?view=Summary', 'https://www.centris.ca/en/lofts-studios~for-rent~quebec-la-cite-limoilou/22499659?view=Summary', 'https://www.centris.ca/en/lofts-studios~for-rent~quebec-la-cite-limoilou/22499659?view=Summary', 'https://www.centris.ca/en/lofts-studios~for-rent~quebec-la-cite-limoilou/22499659?view=Summary', None, 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/12049555?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/12049555?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/12049555?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/12049555?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/12049555?view=Summary', None, 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-le-sud-ouest/23940814?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-le-sud-ouest/23940814?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-le-sud-ouest/23940814?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-le-sud-ouest/23940814?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-le-sud-ouest/23940814?view=Summary', None, 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/27006991?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/27006991?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/27006991?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/27006991?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/27006991?view=Summary', None, 'https://www.centris.ca/en/condos-apartments~for-rent~levis-desjardins/19288399?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~levis-desjardins/19288399?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~levis-desjardins/19288399?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~levis-desjardins/19288399?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~levis-desjardins/19288399?view=Summary', None, 'https://www.centris.ca/en/condos-apartments~for-rent~mont-tremblant/17619353?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~mont-tremblant/17619353?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~mont-tremblant/17619353?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~mont-tremblant/17619353?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~mont-tremblant/17619353?view=Summary', None, 'https://www.centris.ca/en/condos-apartments~for-rent~quebec-sainte-foy-sillery-cap-rouge/21825148?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~quebec-sainte-foy-sillery-cap-rouge/21825148?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~quebec-sainte-foy-sillery-cap-rouge/21825148?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~quebec-sainte-foy-sillery-cap-rouge/21825148?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~quebec-sainte-foy-sillery-cap-rouge/21825148?view=Summary', None, 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/21424641?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/21424641?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/21424641?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/21424641?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/21424641?view=Summary', None, 'https://www.centris.ca/en/condos~for-sale~bromont/26233302?view=Summary', 'https://www.centris.ca/en/condos~for-sale~bromont/26233302?view=Summary', 'https://www.centris.ca/en/condos~for-sale~bromont/26233302?view=Summary', 'https://www.centris.ca/en/condos~for-sale~bromont/26233302?view=Summary', 'https://www.centris.ca/en/condos~for-sale~bromont/26233302?view=Summary', None, 'https://www.centris.ca/en/houses~for-rent~westmount/17444501?view=Summary', 'https://www.centris.ca/en/houses~for-rent~westmount/17444501?view=Summary', 'https://www.centris.ca/en/houses~for-rent~westmount/17444501?view=Summary', 'https://www.centris.ca/en/houses~for-rent~westmount/17444501?view=Summary', 'https://www.centris.ca/en/houses~for-rent~westmount/17444501?view=Summary', None, 'https://www.centris.ca/en/condos-apartments~for-rent~laval-chomedey/19011333?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~laval-chomedey/19011333?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~laval-chomedey/19011333?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~laval-chomedey/19011333?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~laval-chomedey/19011333?view=Summary', None, 'https://www.centris.ca/en/condos-apartments~for-rent~brossard/20981895?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~brossard/20981895?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~brossard/20981895?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~brossard/20981895?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~brossard/20981895?view=Summary', None, 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-le-sud-ouest/11477282?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-le-sud-ouest/11477282?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-le-sud-ouest/11477282?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-le-sud-ouest/11477282?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-le-sud-ouest/11477282?view=Summary', None, 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/12730891?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/12730891?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/12730891?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/12730891?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/12730891?view=Summary', None, 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/14530697?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/14530697?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/14530697?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/14530697?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/14530697?view=Summary', None, 'https://www.centris.ca/en/houses~for-rent~blainville/20565674?view=Summary', 'https://www.centris.ca/en/houses~for-rent~blainville/20565674?view=Summary', 'https://www.centris.ca/en/houses~for-rent~blainville/20565674?view=Summary', 'https://www.centris.ca/en/houses~for-rent~blainville/20565674?view=Summary', 'https://www.centris.ca/en/houses~for-rent~blainville/20565674?view=Summary', None, 'https://www.centris.ca/en/condos-apartments~for-rent~quebec-la-cite-limoilou/17234044?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~quebec-la-cite-limoilou/17234044?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~quebec-la-cite-limoilou/17234044?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~quebec-la-cite-limoilou/17234044?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~quebec-la-cite-limoilou/17234044?view=Summary', None, 'https://www.centris.ca/en/houses~for-sale~montreal-ville-marie/17988959?view=Summary', 'https://www.centris.ca/en/houses~for-sale~montreal-ville-marie/17988959?view=Summary', 'https://www.centris.ca/en/houses~for-sale~montreal-ville-marie/17988959?view=Summary', 'https://www.centris.ca/en/houses~for-sale~montreal-ville-marie/17988959?view=Summary', 'https://www.centris.ca/en/houses~for-sale~montreal-ville-marie/17988959?view=Summary', None, 'https://www.centris.ca/en/condos-apartments~for-rent~sainte-julie/14344430?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~sainte-julie/14344430?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~sainte-julie/14344430?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~sainte-julie/14344430?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~sainte-julie/14344430?view=Summary', None, 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/28728702?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/28728702?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/28728702?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/28728702?view=Summary', 'https://www.centris.ca/en/condos-apartments~for-rent~montreal-ville-marie/28728702?view=Summary', None]

  • Note : You have to add the following imports :

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352