I am new to selenium and trying to work on a project where I need to scrape the URL's from a page.
The source is:- https://www.autofurnish.com/audi-car-accessories
I wanted to scrape the data to get the URLs of these products. I am able to complete it but facing an issue with the scrolling part. I need to scrape all the URLs of all the products on this page. This is a huge page having a lot of results.
What I tried:-
1.
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
I tried this code, but it's just scrolling down till the end and all the products aren't loading.
2.
data = driver.find_elements(By.XPATH,"//h2[@class='product-title']//a")
for i in data:
driver.execute_script("arguments[0].scrollIntoView();", i)
items = [] last_height = driver.execute_script("return document.body.scrollHeight") item_targetcount = 1000 while item_targetcount > len(items): driver.execute_script("window.scrollTo(0,document.body.scrollHeight);") time.sleep(2) # giving time to website to load new_height = driver.execute_script("return document.body.scrollHeight") if new_height == last_height: break last_height = new_height
Tried to take help from:- How to scroll down in Python Selenium step by step Scrolling to element using webdriver? Tried watching a few youtube videos still unable to fix this.
My main code to scrape other details is:-
prod_details = []
for i in models:
driver.find_element(By.XPATH,"//span[@aria-labelledby='select2-brand-container']").click()
time.sleep(2)
driver.find_element(By.XPATH,"//input[@class='select2-search__field']").send_keys(i)
driver.find_element(By.XPATH,"//input[@class='select2-search__field']").send_keys(Keys.ENTER)
driver.find_element(By.XPATH,"//div[@class='btnred sbv-link sbv-inactive']").click()
time.sleep(3)
prod = driver.find_elements(By.XPATH,"//h2[@class='product-title']//a")
for i in prod:
prod_details.append(i.get_attribute("href"))
driver.get('https://www.autofurnish.com/')
time.sleep(2)
Still unable to load the page completely and get all the outputs.
