0

I have the following problem when webscraping a shein page.

In general the problem is with attributes and in relation to - AttributeError: 'WebDriver' object has no attribute 'find_elements_by_xpath' -

If any of you have faced this problem, I would like to know your suggestions for a solution.

from selenium import webdriver
import time
import pandas as pd

# Create a new instance of the Firefox driver
driver = webdriver.Firefox()

# Navigate to the Shein website
driver.get("https://www.shein.com.mx/style/Women-Clothing-sc-001121425.html?ici=mx_tab01navbar04&src_module=topcat&src_tab_page_id=page_select_class1686667964514&src_identifier=fc%3DWomen%60sc%3DROPA%60tc%3D0%60oc%3D0%60ps%3Dtab01navbar04%60jc%3DitemPicking_001121425&srctype=category&userpath=category-ROPA")

# Wait for the page to load
time.sleep(2)

# Close the cookie consent popup if it appears
try:
    close_button = driver.find_element_by_class_name("cookie-close-btn")
    close_button.click()
except:
    pass

# Create lists to store the extracted data
product_names = []
prices = []
urls = []

# Extract the product names, prices, and URLs from the homepage
product_elements = driver.find_elements_by_xpath(".//div[@class='product-list.j-expose__product-list.j-product-list-info.j-da-event-box']")
for product_element in product_elements:
    name_element = product_element.find_element_by_xpath(".//div[@class='S-product-item__name']")
    price_element = product_element.find_element_by_xpath(".//span[@class='normal-price-ctn__sale-prices']")
    url_element = product_element.find_element_by_xpath(".//a[@class='S-product-item__link']")
    
    name = name_element.text
    price = price_element.text
    url = url_element.get_attribute("href")
    
    product_names.append(name)
    prices.append(price)
    urls.append(url)

# Create a pandas DataFrame from the extracted data
data = {
    "Product Name": product_names,
    "Price": prices,
    "URL": urls
}
df = pd.DataFrame(data)

# Save the DataFrame to an Excel file
df.to_excel("shein_products.xlsx", index=False)

# Close the browser
driver.quit()

I get the following error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[32], line 27
     24 urls = []
     26 # Extract the product names, prices, and URLs from the homepage
---> 27 product_elements = driver.find_elements_by_xpath(".//div[@class='product-list.j-expose__product-list.j-product-list-info.j-da-event-box']")
     28 for product_element in product_elements:
     29     name_element = product_element.find_element_by_xpath(".//div[@class='S-product-item__name']")

AttributeError: 'WebDriver' object has no attribute 'find_elements_by_xpath'

I am hoping to be able to correctly get the price for each product identified on the Shein page.

Can you provide me with any suggestions to solve this problem?

Greetings, Alexis

  • 1
    Selenium has made `find_element_by_xxx` deprecated. So the new version, you have to write like shown here: https://selenium-python.readthedocs.io/locating-elements.html – Yubo Jun 15 '23 at 01:01

0 Answers0