Python - Selenium Webdriver find an element where all the div nodes have similar names

Question

I am trying to access a div where all divs have the same name. Let me explain. I am just starting out with selenium and python and I am trying to scrape a webpage just to learn. I ran into the following problem. I made the example html to show the buildup of the webpage. All the divs have the exact same class and title. Then there is the h1 tag for the item and the p tag for the color (which is a clickable link). I am trying to search a page when you give it certain instructions. Example: I am looking for a white racebike. I am able to find the bikes with the first line of code, but how do I find the right color within the racebike section? If I run the Python mentioned below I get an error message. Thanks in advance!

<!DOCTYPE html>
<html>
    <body>
        <div class=div title=example>
            <h1>racebike</h1>
            <p class='test'>black</p>
        </div>
        <div class=div title=example>
            <h1>racebike</h1>
            <p class='test'>white</p>
        </div>
        <div class=div title=example>
            <h1>racebike</h1>
            <p class='test'>yellow</p>
        </div>
        <div class=div title=example>
            <h1>citybike</h1>
            <p class='test'>yellow</p>
        </div>
        <div class=div title=example>
            <h1>citybike</h1>
            <p class='test'>green</p>
        </div>
    </body>
</html>

test = (self.driver.find_element_by_xpath("//*[contains(text(), racebike)]"))
test.self.driver.find_element_by_xpath(".//*[contains(text(), white)]").click

score 0 · Accepted Answer · answered Mar 23 '20 at 22:02

0

To locate/click() on the white racebike element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following xpath based Locator Strategies:

Using XPATH:

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//h1[text()='racebike']//following-sibling::p[@class='test' and text()='white']"))).click()

Using XPATH considering the parent <div>:

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[@class='div' and @title='example']/h1[text()='racebike']//following-sibling::p[@class='test' and text()='white']"))).click()

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

answered Mar 23 '20 at 22:02

undetected Selenium

183,867
41
278
352

Thanks for your response! I used the code but got this error: `code` Traceback (most recent call last): File "location of script", line 23, in PythonProj(category, item, color, size) File "location of script", line 16, in __init__ wait = WebDriverWait(self.driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//h1[text()='Race Bike']//following-sibling::p[text()='Red']"))).click() File "location of selenium wait.py", line 80, in until raise TimeoutException(message, screen, stacktrace) selenium.common.exceptions.TimeoutException: Message: `code` – Thijs Mar 24 '20 at 21:39
Sorry had problems with my last comment hopefully this one is clearer to read. `Traceback (most recent call last): File "location of script", line 23, in PythonProj(category, item, color, size) File "location of script", line 16, in __init__ wait = WebDriverWait(self.driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//h1[text()='Race Bike']//following-sibling::p[text()='Red']"))).click() File "location of selenium wait.py", line 80, in until raise TimeoutException(message, screen, stacktrace) selenium.common.exceptions.TimeoutException: Message: ` What am I doing wrong? – Thijs Mar 24 '20 at 21:46

SeleniumUser · Answer 2 · 2020-03-25T18:57:08.050

0

You can use same xpath which you tried in your solution. It might be possible server is taking too long to repond.

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

element = WebDriverWait(page, 10).until(EC.presence_of_element_located((By.XPATH, "//p[contains(@class, 'white')]")))
element.click()

for multiple bikes with whiite color

elements= WebDriverWait(driver, 30).until(EC.presence_of_all_elements_located((By.XPATH, "//p[contains(@class, 'white')]")))
for element in elements:
    element.click()

edited Mar 25 '20 at 18:57

answered Mar 23 '20 at 22:41

SeleniumUser

4,065
2
7
30

Thanks for your response! I think this would work in the example I gave on this page, but how would this work when there are multiple bikes with a white color? How do you find the right bike in the right color? – Thijs Mar 25 '20 at 09:21
check updated solution for a multiple bikes with whiite color – SeleniumUser Mar 25 '20 at 09:46
When I use this code I get a raise TimeoutException(message, screen, stacktrace) error, what is going wrong? – Thijs Mar 25 '20 at 18:09
I have increased wait please check now – SeleniumUser Mar 25 '20 at 18:57
I got the problem fixed, thank you so much, the code was structured in a weird way so that is why it didn't work. I ended up finding the right xpath using the parent and child methods. – Thijs Mar 26 '20 at 21:46

Python - Selenium Webdriver find an element where all the div nodes have similar names

2 Answers2