I have the following problem when webscraping a shein page.
In general the problem is with attributes and in relation to - AttributeError: 'WebDriver' object has no attribute 'find_elements_by_xpath' -
If any of you have faced this problem, I would like to know your suggestions for a solution.
from selenium import webdriver
import time
import pandas as pd
# Create a new instance of the Firefox driver
driver = webdriver.Firefox()
# Navigate to the Shein website
driver.get("https://www.shein.com.mx/style/Women-Clothing-sc-001121425.html?ici=mx_tab01navbar04&src_module=topcat&src_tab_page_id=page_select_class1686667964514&src_identifier=fc%3DWomen%60sc%3DROPA%60tc%3D0%60oc%3D0%60ps%3Dtab01navbar04%60jc%3DitemPicking_001121425&srctype=category&userpath=category-ROPA")
# Wait for the page to load
time.sleep(2)
# Close the cookie consent popup if it appears
try:
close_button = driver.find_element_by_class_name("cookie-close-btn")
close_button.click()
except:
pass
# Create lists to store the extracted data
product_names = []
prices = []
urls = []
# Extract the product names, prices, and URLs from the homepage
product_elements = driver.find_elements_by_xpath(".//div[@class='product-list.j-expose__product-list.j-product-list-info.j-da-event-box']")
for product_element in product_elements:
name_element = product_element.find_element_by_xpath(".//div[@class='S-product-item__name']")
price_element = product_element.find_element_by_xpath(".//span[@class='normal-price-ctn__sale-prices']")
url_element = product_element.find_element_by_xpath(".//a[@class='S-product-item__link']")
name = name_element.text
price = price_element.text
url = url_element.get_attribute("href")
product_names.append(name)
prices.append(price)
urls.append(url)
# Create a pandas DataFrame from the extracted data
data = {
"Product Name": product_names,
"Price": prices,
"URL": urls
}
df = pd.DataFrame(data)
# Save the DataFrame to an Excel file
df.to_excel("shein_products.xlsx", index=False)
# Close the browser
driver.quit()
I get the following error:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In[32], line 27
24 urls = []
26 # Extract the product names, prices, and URLs from the homepage
---> 27 product_elements = driver.find_elements_by_xpath(".//div[@class='product-list.j-expose__product-list.j-product-list-info.j-da-event-box']")
28 for product_element in product_elements:
29 name_element = product_element.find_element_by_xpath(".//div[@class='S-product-item__name']")
AttributeError: 'WebDriver' object has no attribute 'find_elements_by_xpath'
I am hoping to be able to correctly get the price for each product identified on the Shein page.
Can you provide me with any suggestions to solve this problem?
Greetings, Alexis