0

This is the first time I'm making a website scraper and I'm relatively new to programming in general. So I'm trying to get the HREF links for all the subpages on this site:. But when I ran the code, instead of getting all the different HREF links, I keep getting the same one (the first one) 30 times. I'm sure I'm making a silly mistake here, but can't quite figure out what it is.

So here's my code:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC



PATH='C:\Program Files (x86)\chromedriver.exe'
driver = webdriver.Chrome(PATH)
driver.implicitly_wait(20)
driver.get("https://iamovers.mobilityex.com/#/search?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C")

WebDriverWait(driver, 15).until(EC.element_to_be_clickable((By.CLASS_NAME, 'cc-btn cc-dismiss'))).click()

list_links=[]

left_component = driver.find_element(By.ID, 'left')
for company_box in left_component.find_elements(By.CSS_SELECTOR, 'div[ng-repeat="sp in sr track by $index"'):
    url_div = company_box.find_element(By.CLASS_NAME, 'col-md-6')
    url_a = url_div.find_element(By.TAG_NAME, 'a')
    url = url_a.get_attribute('href')
    print(url)
bad_coder
  • 11,289
  • 20
  • 44
  • 72
JayLil
  • 11

1 Answers1

0

To print the value of all the href attributes you have to induce WebDriverWait for the visibility_of_all_elements_located() and you can use the following locator strategy:

  • Using CSS_SELECTOR:

    driver.get('https://iamovers.mobilityex.com/#/search?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C')
print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "#left div[ng-repeat='sp in sr track by $index'] .col-md-6 a")))])

  • Console Output:

    ['https://iamovers.mobilityex.com/#/search/service-providers/63?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', 'https://iamovers.mobilityex.com/customer/images/63_PCG_PCG.pdf', 'https://iamovers.mobilityex.com/customer/images/63_ISO%2014000_Screen%20Shot%202021-09-07%20at%208.20.44%20AM.png', 'https://www.fidi.org/find-fidi-affiliate/aspire-mobility-group', None, 'https://iamovers.mobilityex.com/customer/images/63_BAR_Screen%20Shot%202020-07-14%20at%203.01.11%20PM.png', 'https://iamovers.mobilityex.com/customer/images/63_DMF_Screen%20Shot%202022-03-01%20at%209.24.06%20AM.png', 'https://www.fidi.org/find-fidi-affiliate/aspire-mobility-group', 'https://iamovers.mobilityex.com/customer/images/LACMA_cert.png', None, 'https://www.worldwideerc.org/member-directory/aspire-mobility-group', 'https://www.omnimoving.com/memberfinder/profile/aspire-mobility-group_107/', 'https://www.eura-relocation.com/members/aspire-mobility-group', 'https://iamovers.mobilityex.com/#/search/service-providers/1068?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', None, None, 'https://fedemac.com/member/aktieselskabet-move-it/', 'https://iamovers.mobilityex.com/#/search/service-providers/1244?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', 'https://iamovers.mobilityex.com/customer/images/1244_PCG_Screen%20Shot%202018-10-08%20at%208.03.24%20AM.png', 'https://iamovers.mobilityex.com/customer/images/1244_ISO%209000_Screen%20Shot%202021-01-27%20at%208.56.05%20AM.png', None, 'https://iamovers.mobilityex.com/customer/images/Euromovers_cert.png', 'https://iamovers.mobilityex.com/customer/images/PAIMA_cert.png', None, 'https://iamovers.mobilityex.com/customer/images/1244_ATVL_Atlas.pdf', 'https://iamovers.mobilityex.com/#/search/service-providers/38?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', 'https://iamovers.mobilityex.com/customer/images/38_ISO%209000_Screen%20Shot%202020-09-15%20at%2011.57.29%20AM.png', 'https://iamovers.mobilityex.com/customer/images/38_ISO%2014000_Screen%20Shot%202020-09-15%20at%2011.52.02%20AM.png', 'https://www.fidi.org/find-fidi-affiliate/hertling-gmbh-co-kg', None, 'https://www.fidi.org/find-fidi-affiliate/hertling-gmbh-co-kg', 'https://iamovers.mobilityex.com/customer/images/38_OMA_Screen%20Shot%202022-03-15%20at%2010.22.48%20AM.png', None, 'https://iamovers.mobilityex.com/customer/images/38_AMO_156-131275-Hertling%20GmbH%20&%20Co.%20KG%20ZERTIFIKAT.pdf', 'https://www.eura-relocation.com/members/hertling-gmbh-co-kg', 'https://iamovers.mobilityex.com/#/search/service-providers/865?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', 'https://iamovers.mobilityex.com/customer/images/865_ISO%209000_Screen%20Shot%202019-11-29%20at%207.55.11%20AM.png', None, None, None, 'https://iamovers.mobilityex.com/customer/images/865_AMO_AMO%CC%88%20Zertifikat%20bis%2031-12-2022.pdf', 'https://iamovers.mobilityex.com/#/search/service-providers/14021?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', 'https://iamovers.mobilityex.com/customer/images/14021_PCG_PCG%20document.pdf', None, 'https://iamovers.mobilityex.com/customer/images/14021_DMF_DMF%20Certifikat%202022.pdf', 'https://iamovers.mobilityex.com/customer/images/Harmony_cert.png', None, 'https://iamovers.mobilityex.com/customer/images/14021_FEDEMAC_Screen%20Shot%202021-02-25%20at%201.51.00%20PM.png', 'https://iamovers.mobilityex.com/#/search/service-providers/661?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', 'https://iamovers.mobilityex.com/customer/images/661_PCG_Screen%20Shot%202020-11-24%20at%2011.30.45%20AM.png', None, None, 'https://iamovers.mobilityex.com/customer/images/661_CONFERN_Screen%20Shot%202022-01-13%20at%2011.04.19%20AM.png', 'https://iamovers.mobilityex.com/#/search/service-providers/1367?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', 'https://iamovers.mobilityex.com/customer/images/1367_PCG_PCG.pdf', None, None, 'https://iamovers.mobilityex.com/customer/images/1367_CONFERN_Screen%20Shot%202022-01-13%20at%2011.26.18%20AM.png', 'https://iamovers.mobilityex.com/#/search/service-providers/33?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', 'https://iamovers.mobilityex.com/customer/images/33_PCG_Screen%20Shot%202020-11-05%20at%2010.52.35%20AM.png', 'https://iamovers.mobilityex.com/customer/images/33_ISO%209000_A.23280_ENG_GLC_Flytt_H%C3%A4ngcert_9001.pdf', 'https://iamovers.mobilityex.com/customer/images/33_ISO%2014000_A.23280_ENG_GLC_Flytt_H%C3%A4ngcert_9001.pdf', None, None, 'https://iamovers.mobilityex.com/#/search/service-providers/14876?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', 'https://iamovers.mobilityex.com/customer/images/14876_PCG_Screen%20Shot%202021-06-07%20at%208.54.04%20AM.png', None, None, 'https://iamovers.mobilityex.com/#/search/service-providers/612?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', 'https://iamovers.mobilityex.com/customer/images/612_PCG_Screen%20Shot%202021-01-21%20at%207.51.14%20AM.png', None, None, 'https://iamovers.mobilityex.com/#/search/service-providers/886?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', 'https://iamovers.mobilityex.com/customer/images/886_PCG_PCG.pdf', 'https://www.fidi.org/find-fidi-affiliate/ito-mobeltransport-gmbh', None, 'https://www.fidi.org/find-fidi-affiliate/ito-mobeltransport-gmbh', None, 'https://iamovers.mobilityex.com/#/search/service-providers/1458?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', 'https://iamovers.mobilityex.com/customer/images/1458_PCG_Acknowledgement.pdf', 'https://iamovers.mobilityex.com/customer/images/1458_ISO%209000_Screen%20Shot%202020-04-01%20at%208.59.44%20AM.png', 'https://www.fidi.org/find-fidi-affiliate/brauns-international', None, 'https://www.fidi.org/find-fidi-affiliate/brauns-international', None, 'https://fedemac.com/member/brauns-international-moving-services-gmbh/', 'https://iamovers.mobilityex.com/customer/images/1458_AMO_2022%20AMO%CC%88%20173-141010-Brauns%20International%20Moving%20Services%20GmbH%20ZERTIFIKAT.pdf', 'https://iamovers.mobilityex.com/#/search/service-providers/2692?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', 'https://iamovers.mobilityex.com/customer/images/2692_AEO_2016-01-19%20AEO-Bewilligung.pdf', None, 'https://iamovers.mobilityex.com/customer/images/2692_PCG_SKMBT_42318091310490.pdf', 'https://iamovers.mobilityex.com/customer/images/2692_ISO%209000_Wilhelm%20Rosebrock%20Germany%209001%202018.pdf', 'https://iamovers.mobilityex.com/customer/images/2692_ISO%2014000_Wilhelm%20Rosebrock%20Germany%2014001%202018.pdf', 'https://iamovers.mobilityex.com/customer/images/2692_ISO%2027000_Wilhelm%20Rosebrock%20GmbH%2027001%202016.pdf', None, 'https://iamovers.mobilityex.com/customer/images/ILN_Validation_Cert.pdf', 'https://iamovers.mobilityex.com/customer/images/Eurovan_cert.png', 'https://iamovers.mobilityex.com/customer/images/ISA_Validation_Cert.pdf', 'https://iamovers.mobilityex.com/customer/images/LACMA_cert.png', None, 'https://iamovers.mobilityex.com/customer/images/2692_FIATA_Screen%20Shot%202021-05-13%20at%2012.37.31%20PM.png', 'https://iamovers.mobilityex.com/customer/images/2692_CONFERN_Screen%20Shot%202022-01-13%20at%2011.06.52%20AM.png', 'https://iamovers.mobilityex.com/customer/images/2692_AMO_Screen%20Shot%202021-02-23%20at%209.22.01%20AM.png', 'https://www.iata.org/en/publications/directories/cargolink/directory/wilhelm-rosebrock-gmbh--co.-kg/7113/', 'https://iamovers.mobilityex.com/#/search/service-providers/530?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', 'https://iamovers.mobilityex.com/customer/images/530_AEO_Mail0400.pdf', None, 'https://iamovers.mobilityex.com/customer/images/530_PCG_PCG.pdf', None, 'https://iamovers.mobilityex.com/customer/images/ILN_Validation_Cert.pdf', 'https://iamovers.mobilityex.com/customer/images/ISA_Validation_Cert.pdf', 'https://www.wcaworld.com/directory/members/59295', 'https://iamovers.mobilityex.com/customer/images/PAIMA_cert.png', None, 'https://iamovers.mobilityex.com/customer/images/530_ARA_Screen%20Shot%202021-01-07%20at%209.17.04%20AM.png', 'https://iamovers.mobilityex.com/customer/images/530_AMO_174-141027-Carl%20Hartmann%20GmbH%20&%20Co.%20KG%20%C3%9Cberseespedition%20ZERTIFIKAT.pdf', 'https://iamovers.mobilityex.com/#/search/service-providers/9461?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', 'https://iamovers.mobilityex.com/customer/images/9461_PCG_PCG.pdf', None, None, 'https://iamovers.mobilityex.com/#/search/service-providers/14729?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', None, None, 'https://iamovers.mobilityex.com/#/search/service-providers/3336?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', 'https://iamovers.mobilityex.com/customer/images/3336_PCG_Screen%20Shot%202021-10-25%20at%208.59.50%20AM.png', None, None, 'https://iamovers.mobilityex.com/#/search/service-providers/689?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', 'https://iamovers.mobilityex.com/customer/images/689_PCG_Screen%20Shot%202020-09-22%20at%2010.33.10%20AM.png', None, 'https://iamovers.mobilityex.com/customer/images/Harmony_cert.png', 'https://www.ipata.org/member/111', None, 'https://www.eura-relocation.com/members/move-one-relocations-5', 'https://iamovers.mobilityex.com/#/search/service-providers/361?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', 'https://www.fidi.org/find-fidi-affiliate/universal-express', None, 'https://iamovers.mobilityex.com/customer/images/361_BAR_Screen%20Shot%202020-07-15%20at%209.45.29%20AM.png', 'https://iamovers.mobilityex.com/customer/images/ILN_Validation_Cert.pdf', 'https://www.fidi.org/find-fidi-affiliate/universal-express', 'https://iamovers.mobilityex.com/customer/images/361_OMA_Screen%20Shot%202020-04-15%20at%2011.38.18%20AM.png', None, 'https://www.omnimoving.com/memberfinder/profile/universal-express-relocations-sp-z-o-o_109/', 'https://iamovers.mobilityex.com/#/search/service-providers/14764?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', None, None, 'https://iamovers.mobilityex.com/#/search/service-providers/14641?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', 'https://iamovers.mobilityex.com/customer/images/14641_PCG_Acknowledgement%20Statement%20INDYGO%20MS.pdf', None, None, 'https://fedemac.com/member/indygo-ms/', 'https://iamovers.mobilityex.com/#/search/service-providers/1237?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', 'https://iamovers.mobilityex.com/customer/images/1237_PCG_Screen%20Shot%202021-03-22%20at%2010.54.27%20AM.png', None, None, 'https://iamovers.mobilityex.com/customer/images/1237_FEDEMAC_Screen%20Shot%202021-03-22%20at%2010.56.08%20AM.png', 'https://iamovers.mobilityex.com/#/search/service-providers/1683?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', None, None, 'https://iamovers.mobilityex.com/#/search/service-providers/11755?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', 'https://iamovers.mobilityex.com/customer/images/11755_PCG_Screen%20Shot%202020-04-03%20at%207.59.09%20AM.png', None, 'https://iamovers.mobilityex.com/customer/images/Euromovers_cert.png', None, 'https://fedemac.com/member/wilhelm-rosebrock-poland/', 'https://iamovers.mobilityex.com/#/search/service-providers/1007?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', 'https://iamovers.mobilityex.com/customer/images/1007_PCG_Screen%20Shot%202019-01-30%20at%206.38.55%20AM.png', 'https://www.fidi.org/find-fidi-affiliate/santa-fe-relocation-warsaw', None, 'https://www.fidi.org/find-fidi-affiliate/santa-fe-relocation-warsaw', None, 'https://iamovers.mobilityex.com/#/search/service-providers/5186?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', 'https://iamovers.mobilityex.com/customer/images/5186_PCG_Screen%20Shot%202020-09-22%20at%2010.33.10%20AM.png', None, None, 'https://www.eura-relocation.com/members/move-one-relocations-8', 'https://iamovers.mobilityex.com/#/search/service-providers/901?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', 'https://iamovers.mobilityex.com/customer/images/901_PCG_Screen%20Shot%202019-01-30%20at%206.38.55%20AM.png', 'https://www.fidi.org/find-fidi-affiliate/santa-fe-relocation-prague', None, 'https://www.fidi.org/find-fidi-affiliate/santa-fe-relocation-prague', None, 'https://iamovers.mobilityex.com/#/search/service-providers/745?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', 'https://iamovers.mobilityex.com/customer/images/745_PCG_PCG.pdf', None, None, 'https://iamovers.mobilityex.com/#/search/service-providers/11534?loc=Europe&lat=54.5259614&lng=15.2551187&range=50&assocs=800&fvw=c&ctry=undefined%5C', 'https://iamovers.mobilityex.com/customer/images/11534_ISO%209000_Screen%20Shot%202021-02-12%20at%209.40.03%20AM.png', None, 'https://iamovers.mobilityex.com/customer/images/Euromovers_cert.png', None]

  • Note : You have to add the following imports :

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352