Web-scraping (the class name changes)

Question

I am using selenium to download some images for my project!

For downloading images, I use the following command lines:

# Finding elements of images by class name
image_lm = prd.find_element_by_class_name('main')

# The URL to the image
image_url = image_lm.get_attribute('src')

Then, using image_url, I download the images.

The problem is that, after several tries, the class name changes to "main-lazy".

I can manually change the "main" to "main-lazy". is there any way to do it by code.

I am looking for a way to tell the code that either finds the class name of "main-lazy" is the class name of "main" is not available!

You could try getting the class "main" first. If it doesn't exist, then try "main-lazy". — Code-Apprentice, Nov 20 '19 at 17:09

score 1 · Answer 1 · answered Nov 20 '19 at 17:13

1

Why not try/except ?

try:
   image_lm = prd.find_element_by_class_name('main')
except Exception as e:
   print("changing to main_lazy \n"+e)
   image_lm = prd.find_element_by_class_name('main_lazy')

answered Nov 20 '19 at 17:13

Vignesh SP

451
6
15

score 1 · Answer 2 · answered Nov 20 '19 at 23:25

1

If the only two variations are "main" and "main-lazy" then you can try using:

By.XPath("//[contains(@class,'main')]

Sorry this is C# variation but I am sure you can figure out the Python equivalent.

answered Nov 20 '19 at 23:25

ratsstack

1,012
4
13
32

score 1 · Answer 3 · answered Nov 21 '19 at 03:12

1

You can use css Or syntax of

image_lm = prd.find_element_by_css_selector('.main, .main-lazy')

answered Nov 21 '19 at 03:12

QHarr

83,427
12
54
101

score 1 · Answer 4 · answered Nov 21 '19 at 08:17

ClassName as main-lazy indicates the elements are loaded through lazy-loading. In such cases you have to induce WebDriverWait and you can club up combine check for both the elements using xpath through a lambda expression as follows:

Using class_name 1:

image_lm = WebDriverWait(driver, 20).until(lambda x: (x.find_element_by_class_name("main"), x.find_element_by_class_name("main-lazy")))

Using class_name 2:

image_lm = WebDriverWait(driver,20).until(lambda driver: driver.find_element(By.CLASS_NAME,"main") and driver.find_element(By.CLASS_NAME,"main-lazy"))

As an alternative you can club up combine check for both the elements using the equivalent css-selectors as follows:

Using css_selector:

image_lm = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, ".main, .main-lazy")))

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

You can find a relevant discussion in selenium two xpath tests in one

Web-scraping (the class name changes)

4 Answers4