1

How to get the get data from span using Selenium Python.

Code trials:

import time
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By

class Amazon():

    def amazon_data(self):

        driver = webdriver.Chrome(executable_path=ChromeDriverManager().install())
        driver.get(url="https://www.amazon.com/")
        input_search = driver.find_element(By.XPATH,"//input[@id='twotabsearchtextbox']")
        input_search.send_keys("smart phones under 100000")
        time.sleep(1)
        search_button = driver.find_element(By.XPATH,"//input[@id='nav-search-submit-button']")
        search_button.click()

        time.sleep(3)

        
d = driver.find_elements(By.CLASS_NAME,"a-size-medium a-color-base a-text-normal")
print(len(d))


object = Amazon()
object.amazon_data()



d = driver.find_elements(By.CLASS_NAME,"a-size-medium a-color-base a-text-normal")
print(len(d))

This is the line where I'm stuck and I am not able to fetch the class from website can anyboy help me to sortout please.

enter image description here

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
Shoaib Shoukat
  • 11
  • 2

1 Answers1

0

To extract and print the texts from all of the <span> you need to induce WebDriverWait for visibility_of_all_elements_located() and using List Comprehension you can use either of the following Locator Strategies:

  • Using CSS_SELECTOR and get_attribute("innerHTML"):

    driver.get("https://www.amazon.com/")
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'input#twotabsearchtextbox'))).send_keys("smart phones under 100000" +Keys.RETURN)
print([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "span.a-size-medium.a-color-base.a-text-normal")))])

  • Using XPATH and text attribute:

    driver.get("https://www.amazon.com/")
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'input#twotabsearchtextbox'))).send_keys("smart phones under 100000" +Keys.RETURN)
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//span[@class='a-size-medium a-color-base a-text-normal']")))])

  • Console Output:

    ['OnePlus Nord N20 5G | Android Smart Phone | 6.43" AMOLED Display| 6+128GB | U.S. Unlocked | 4500 mAh Battery | 33W Fast Charging | Blue Smoke', 'SAMSUNG Galaxy A03s Cell Phone, Factory Unlocked Android Smartphone, 32GB, Triple Lens Camera, Infinity Display Screen, Long Battery Life, Expandable Storage, US Version, Blue', 'Google Pixel 6 – 5G Android Phone - Unlocked Smartphone with Wide and Ultrawide Lens - 128GB - Stormy Black', 'SAMSUNG Galaxy S20 FE 5G Cell Phone, Factory Unlocked Android Smartphone, 128GB, Pro Grade Camera, 30X Space Zoom, Night Mode, US Version, Cloud Lavender', 'SAMSUNG Galaxy Z Flip 3 5G Cell Phone, Factory Unlocked Android Smartphone, 256GB, Flex Mode, Super Steady Camera, Ultra Compact, US Version, Phantom Black', 'SAMSUNG Galaxy Z Flip 4 Cell Phone, Factory Unlocked Android Smartphone, 256GB, Flex Mode, Hands Free Camera, Compact, Foldable Design, Informative Cover Screen, US Version, Bora Purple', 'Ulefone Rugged Smartphone, 13200mAh Large Battery, Power Armor 13 8GB + 256GB Android 11 FHD+ 6.81", Octa-core 48MP Quad Camera, NFC OTG Wireless Reverse Charging, IP68 Waterproof Unlocked Cell Phone', '5.72in HD Full Screen Smart Phone, Note30 Plus Unlocked Smartphones, Dual Cards Dual Standby, 512MB+4GB Cell Phones, for Android 4.4.2, Support WiFi +BT+FM (Violet)', 'Apple iPhone 11 Smart Battery Case with Wireless Charging - Black', 'SAMSUNG Galaxy S21 Ultra 5G Factory Unlocked Android Cell Phone 256GB US Version Smartphone Pro-Grade Camera 8K Video 108MP High Res, Phantom Black', 'Apple iPhone 11 Pro Smart Battery Case with Wireless Charging - White', 'Auto Face Tracking Phone Holder, 360 Smart Selfie Tripod Mount Fast Motion Moving Tracker for Smartphone GoPro DJI Action Insta360 Camera SLR Mirrorless Video Vlog Live Streaming Accessories', 'E-YIIVIIL LCD Screen OLED Display Compatible with Google Pixel 5A 5G 2021 G1F8F 6.34" inch LCD Screen Display Digitizer Assembly with Tools', 'Apple iPhone 11 Pro Max Smart Battery Case with Wireless Charging - White', 'Unlocked Cell Phone ，Y99 Android Smartphone, 5.75-inch IPS Full-Screen，Dual SIM Card，1GB RAM 8GB ROM，Only Supports Dual SIM Card Frequency Band of 3GWCDMA ：850/2100MHZ（Blue）', 'Blackview Mobile Phone A55,4G Dual SIM Unlocked Phones,Android 11 3GB+16GB/TF 128GB Smart Phone Unlocked,Face ID,3 Card Slots,4780mAh Capacity Battery, T-Mobile Unlocked Phone 6.5" HD+Large Screen', 'Apple iPhone 11 Pro Max Smart Battery Case with Wireless Charging - Black', 'LMDAMZ Heavy Duty Case for Samsung Galaxy A22(5G), Military Grade Dual Layer Hybrid Cushioning Rubber Bumper Protector with Slide Camera Protector, Built-in 360 Degree Swivel Stand -Rose Gold', 'Google Pixel 6 Pro - 5G Android Phone - Unlocked Smartphone with Advanced Pixel Camera and Telephoto Lens - 256GB - Stormy Black', 'ROG Phone Gaming Smartphone ZS600KL-S845-8G512G - 6” FHD+ 2160x1080 90Hz Display - Qualcomm Snapdragon 845 - 8GB RAM - 512GB Storage - LTE Unlocked Dual SIM Gaming Phone - US Warranty', 'andobil Phone Mount for Car (Bumpy Roads Friendly) Cell Phone Holder Car — Easy Clamp Hands-Free Universal — Fit for Dashboard-Windshield-Vent iPhone 14 13 12 Pro Max, Samsung Galaxy S22 All Phones', 'JIASEN G14s 6.82inch HD Screen Smartphone,Pro MAX Unlocked Smartphones,4G RAM+128G ROM Dual Sim Cellphones,Android Smartphones 24+48MP,Fingerprinr Lock &amp; Face ID(White)', 'WV LeisureMaster M12 5G Unlocked Cell Phone,4GB RAM+128GB ROM Dual Sim Cellphone, Android Smartphone 48MP+72MP Dual Camera, 7300mAh Battery 7.3-inch HD+ Screen, Face Recognition', 'Car Phone Mount,Auto-Clamping Smart Sensor 10W Qi Fast Charging Car Front Windshield Dashboard Air Vent Phone Holder Compatible with iPhone12/12 pro max/Samsung S20/note 20 All 4.7-6.7 Inch Smartphone']

  • Note : You have to add the following imports :

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Outro

Link to useful documentation:

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352