Selenium only loading half of the elements found on the page

Question

"I am fairly new to selenium, i am trying to get all the records on a single page of ebay, but seems like webdriver is only loading half to the instances of the required element, i am using time.sleep and webdriver.wait both to load the complete page but its still showing me only 24 of the records when the actual page has 48, i am also scrolling to the end of the page so all the elements can load but still its only showing me 24 instances, here is my code."

{
# creating a random user agent
    ua = UserAgent()
    user_agent = ua.random


# setting up browser
    chrome_options = webdriver.ChromeOptions()
    chrome_options.add_argument("user-agent={user_agent}")
    chrome_options.add_argument("start-maximized")
    chrome_options.add_argument('--headless')

# Creating driver

    driver = webdriver.Chrome(executable_path="path to chrome driver", chrome_options=chrome_options)
    driver.get("https://www.ebay.com/b/Laptops-Netbooks/175672/bn_1648276")
    time.sleep(5)

# Scrolling to the end of the page
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    WebDriverWait(driver, 30).until(EC.visibility_of_all_elements_located((By.CLASS_NAME, 's-item')))

# Looking for required elements
    req_ele = driver.find_elements_by_class_name("s-item")
    for ele in req_ele:
        print(ele)

    driver.quit()
}

"You can see the actual page has 48 records but this code only prints out 24 selenium objects"

Print the length of `req_ele = driver.find_elements_by_class_name("s-item")` and see if it is getting all 48 records. — Tek Nath Acharya, Dec 04 '19 at 12:52

undetected Selenium · Accepted Answer · 2019-12-04T13:10:06.803

To print the alt attribute of all the 48 items on the webpage https://www.ebay.com/b/Laptops-Netbooks/175672/bn_1648276 you need to induce WebDriverWait for the visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

Using CSS_SELECTOR:

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get("https://www.ebay.com/b/Laptops-Netbooks/175672/bn_1648276")
print([my_elem.get_attribute("alt") for my_elem in WebDriverWait(driver, 5).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "section.b-module.b-list.b-listing.srp-list.b-display--landscape img")))])

Using XPATH:

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get("https://www.ebay.com/b/Laptops-Netbooks/175672/bn_1648276")
print([my_elem.get_attribute("alt") for my_elem in WebDriverWait(driver, 5).until(EC.visibility_of_all_elements_located((By.XPATH, "//section[@class='b-module b-list b-listing srp-list b-display--landscape']//img")))])

Console Output:

['Apple Macbook Pro 15 inch 500 GB Mid 2012 ', 'Google Chrome Pixelbook C0A i7-7Y75 16GB RAM 512GB SSD - Discounted 3444659', 'DELL W21C 3.10GHz 8GB 1TB', 'Dell Latitude E5550 15.6" Intel core i7 5600U 2.6 GHz 8GB 320GB HDD Web cam  ', 'Asus ASUS Vivobook Flip 11.6 HD2-in-1 ConvertibleTouchscreen Laptop, Intel', 'macbook air 2015 13 inch 8gb', 'Apple MacBook Pro 17" Laptop - Working Condition', 'MacBook Pro 13 Inch', 'Apple MacBook Pro A1278\xa0 13.3" Laptop -(April, 2010) Great for School', 'ASUS Computers L402SA Portable Lightweight Laptop PC, Intel Dual Core', 'Dell Vostro 3560 15.5" Laptop Windows 7 Pro 320gb HD', 'MacBook Pro 15" (2016) 500GB SSD 16GB RAM Intel Quad-Core i7 2.7GHz', 'HP Pavilion 15", Used, great condition', '13.3" Fujitsu Lifebook T900 160GB HD Windows 7 Professional', 'Apple MacBook Air 4,2 Core i5-2557M CPU 1.70GHz 4GB 128GB SSD', 'Apple MacBook Pro A1398 Core i7-4980HQ@2.8GHz 16GB 0HD Boots w/GeForce GT 750M', 'ThinkPad X230', 'Apple Macbook Pro 2019 bundle', 'APPLE Macbook Pro 15.4" Retina Core i7 4th 16GB Intel Iris MJLQ2LL/A 2015 READ', 'Sony Vaio Gaming Laptop 2.9GHz Intel i7 12GB RAM Windows 10 Pro 64bit 600GB HDD', 'SAMSUNG CHROMEBOOK PLUS', "Asus G731GU-BI7N9 17.3'' FHD Gaming Laptop i7-9750H 16GB 512GB SSD GTX 1660 Ti", 'Apple MacBook A1278 13.3" Laptop - MB466LL/A (October, 2008)', 'Apple MacBook Pro 15" Laptop with Touchbar and Touch ID, 256GB ', 'MacBook Air 11" 2011', 'HP Pavilion Laptop DV6-2110ea', '2015 MacBook Pro - 13.3" retina display - 8GB RAM- 128GB ', 'Lenovo X1 Carbon ThinkPad i7-4600U 2.69GHz 256GB SSD Pro Dock DU9033S1 Keyboard', "Apple MacBook 12'' 512 GB Gold Laptop - MMGL2LL/A", '??2017 Macbook Pro - 15 Inch W/ Touchbar - Space Gray & Accessories??', 'LENOVO THINKPAD T400 2767-DR8', 'Apple PowerBook G4 15.2" Laptop (10.5) 2005', 'Dell Inspiron 15 7567 15.6" (8GB) - GAMING LAPTOP', 'Sony VAIO VGN-UX280P ', 'Apple MacBook 12" Retina A1534-2746 (2015, Space Gray)', 'Dell Inspiron 15-3521 15.6" Laptop 4GB, Intel processor, WIN 10 ', 'ASUS ROG STRIX GL552VW-DH71 15.6in. (1TB, Intel Core i7 6th Gen., 2.6GHz, 16GB)', 'HP Pavilion x360 11.6 TOUCHSCREEN  Win8 RT3290', 'Apple MacBook Pro 13.3" (128GB SSD, Intel Core i5 8th Gen., 3.90 GHz, 8GB) 2019', 'Acer v3-571-53234g - used - out of box', 'Dell Inspiron 11 3000 Series 4GB RAM - Windows 8', 'Late 2017 MacBook Pro 13 Retina with Apple Care Warranty- Pristine Condition', 'IBM Thinkpad 390e Laptop Intel Celetron 300Mhz SEE NOTES', 'Dell Precision 5510 P56F Laptop Intel Core i7-6820HQ 2.7Ghz 8GB SEE NOTES', 'Sony Vaio P90HS P Series UMPC Intel Z540 1.87GHz 60GB HDD 2GB RAM WINDOWS 10', 'one laptop per child (OLTP)', 'HP Spectre x360 15.6" 4K OLED - Intel Core i7 - 16GB RAM - 1TB SSD (15-df0070nr)', 'HP Pavilion DV 6500 Laptop']

Your solution works like a charm for single element but this way i have to explicitly wait for each element, isn't there a way i can wait for the whole dom to load and then create a beautifulsoup obj of it? — Shahzaib Zubair, Dec 06 '19 at 07:24
@ShahzaibZubair Even waiting for the _...whole dom to load..._ won't ensure desired elements are **present** / **visible** / **interactable**. However I would suggest you to raise a new question as per your new requirement so we can understand your usecase in a better way. — undetected Selenium, Dec 06 '19 at 07:35

score 0 · Answer 2 · answered Dec 04 '19 at 18:00

0

The problem is in your wait:

WebDriverWait(driver, 30).until(EC.visibility_of_all_elements_located((By.CLASS_NAME, 's-item')))

You're not waiting for the new elements you're just waiting for all the elements that have that class name are visible. So if the new 24 haven't been attached to the dom yet, the wait doesn't care.

You need to wait for the count to be more than 24 AND for them to be visible. A couple of potential solutions: https://sqa.stackexchange.com/questions/7018/how-to-wait-until-the-count-of-a-web-element-changes-in-webdriver or https://huddle.eurostarsoftwaretesting.com/how-to-selenium-expected-conditions/

answered Dec 04 '19 at 18:00

DMart

2,401
1
14
19

these are the solutions on java, i am looking for something on python – Shahzaib Zubair Dec 06 '19 at 07:25
The techniques are the same, just in a different language. Count the elements, make the action, wait for count to increase. – DMart Dec 06 '19 at 18:16

Selenium only loading half of the elements found on the page

2 Answers2