How to loop over elements with selenium

Question

I want to use selenium to loop over a few divs on a webpage and find the content of the divs

The basic setup of the webpage looks like this:

<html>
   <div data-testid="property-card">
       <div data-testid="title">content 1</div>
   </div>
   <div data-testid="property-card">
       <div data-testid="title">content 2</div>
   </div>
   <div data-testid="property-card">
       <div data-testid="title">content 3</div>
   </div>
</html>

and here is my code:

def report_results(self):
    hotel_boxes = self.find_elements(By.XPATH, '//div[@data-testid="property-card"]')
    for hotel in hotel_boxes:
        hotel_name = hotel.find_element(By.XPATH, '//div[@data-testid="title"]').get_attribute('innerHTML')
        print(hotel_name)

However, the problem is that this only prints "content 1" for three times. What am I doing wrong here?

score 3 · Answer 1 · answered Aug 17 '22 at 20:44

You are almost there, the only 1 thing you are missing is a dot . at the front of XPath expression.
It should be

hotel_name = hotel.find_element(By.XPATH, './/div[@data-testid="title"]').get_attribute('innerHTML')

When using '//div[@data-testid="title"]' XPath expression it will search for the matching locator from the top of the page until it finds the first match.
While when we have that dot . it means to start searching inside the current node i.e. inside the parent element hotel
So, your entire code can be:

def report_results(self):
    hotel_boxes = self.find_elements(By.XPATH, '//div[@data-testid="property-card"]')
    for hotel in hotel_boxes:
        hotel_name = hotel.find_element(By.XPATH, './/div[@data-testid="title"]').get_attribute('innerHTML')
        print(hotel_name)

undetected Selenium · Answer 2 · 2022-08-17T20:50:47.337

As per the given HTML:

<html>
   <div data-testid="property-card">
       <div data-testid="title">content 1</div>
   </div>
   <div data-testid="property-card">
       <div data-testid="title">content 2</div>
   </div>
   <div data-testid="property-card">
       <div data-testid="title">content 3</div>
   </div>
</html>

To print the innerText of the descendant <div> tags you can use list comprehension and you can use either of the following locator strategies:

Using CSS_SELECTOR and text attribute:

print([my_elem.text for my_elem in driver.find_elements(By.CSS_SELECTOR, "div[data-testid='property-card'] > [data-testid='title']")])

Using XPATH and .get_attribute('innerHTML'):

print([my_elem..get_attribute('innerHTML') for my_elem in driver.find_elements(By.XPATH, "//div[@data-testid='property-card']/div[@data-testid='title']")])

How to loop over elements with selenium

2 Answers2