trip advisor scraper using selenium, unable to get all images having same class name using find_elements_by_classname selenium

Question

The problem I'm facing is I'm using

driver.find_elements_by_classname("a_classname_common_to_all_images_in_tripadvisor_hotels")

However, on running script each time I'm getting less as well as different outcomes. For instance, sometimes it scrapes first 5 out of 30 on the page, sometimes 4/30 and so on. I'm scraping images from this link:
https://www.tripadvisor.in/Hotels-g304551-New_Delhi_National_Capital_Territory_of_Delhi-Hotels.html

images = driver.find_elements_by_class_name("_1a4WY7aS")

I am able to find all names of the hotels using class_name method, however with images it's variable. Any help is appreciated, thanks :)

Arundeep Chohan · Answer 1 · 2020-09-30T07:59:14.390

From How can I scroll a web page using selenium webdriver in python?

SCROLL_PAUSE_TIME = 0.5

# Get scroll height
last_height = driver.execute_script("return document.body.scrollHeight")

while True:
    # Scroll down to bottom
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

    # Wait to load page
    time.sleep(SCROLL_PAUSE_TIME)

    # Calculate new scroll height and compare with last scroll height
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

Inducing a webdriver wait to load all your elements.

images = WebDriverWait(driver, 30).until(EC.presence_of_all_elements_located((By.CLASS_NAME, "_1a4WY7aS")))

Import

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait

score 0 · Answer 2 · edited Sep 30 '20 at 08:25

When it scrapes only 5 of the images present means only 5 images were loaded. You should do 2 things to get every image on the page.

Scroll down to the end of the page: You can do this by selecting the body element and then sending the down keys.

from selenium.webdriver.common.keys import Keys
import time

for _ in range(10):
    driver.find_element_by_tag_name("body").send_keys(Keys.PAGE_DOWN)
    time.sleep(0.2)

After scrolling, wait for the elements to be present

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait

WebDriverWait(driver,10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,
 "img._1a4WY7aS")))

trip advisor scraper using selenium, unable to get all images having same class name using find_elements_by_classname selenium

2 Answers2