How to extract the text 154 from nested div as per the html provided through Selenium and Python

Question

I am new to Python web scraping. I am writing a program that tries to capture the number 154 of class named "stat answers col-3" by using selenium library in Python.

<div id="user-card" class="user-card">

            <div class="row col-content">

                <div class="col-right col-4">  
                    <div class="user-links">
                                <div class="user-stats">
        <div class="row">
            <div class="stat answers col-3">
                <span class="number">154</span>
                answers
            </div>

            <div class="stat questions col-3">
                <span class="number">44</span>
                questions
            </div>

        </div>
    </div>
    </div>
    </div>                    
    </div>

Here is my code:

from selenium import webdriver
driver = webdriver.Chrome(executable_path="/Users/username/Downloads/chromedriver")
driver.get("website address")
content = driver.find_element_by_xpath('//div[@id="user-card" and @class="stat answers col-3"]').__getattribute__("number")

However, I was not able to run the program. Could please anyone point where I am making mistake.

@Andersson This question have multiple issue apart from `.text`. First of all OP's locator is a bit off. Second possibly due to certain factor OP wants to use `getattribute()`. With those two points combined, clearly not a dup. Please have a relook. — undetected Selenium, Aug 10 '18 at 05:28
@DebanjanB , wrong locators should lead to `NoSuchElementException` which has nothing to do with *How to extract the text* issue. Also note that Python built-in `__getattribute__()` is something different than Selenium `get_attribute()`... — Andersson, Aug 10 '18 at 05:41
@Andersson Exactly. OP mentioned _I was not able to run the program_ but never mentioned `NoSuchElementException` or `InvalidSelector` or `wrong/no text`? So how can it be a dup? Additionally, usage of `get_attribute()` mustn't be discouraged as the function is _robust_, _proven_ and _effective_. — undetected Selenium, Aug 10 '18 at 05:47

score 1 · Answer 1 · edited Aug 10 '18 at 05:15

1

As per the given HTML, to extract the number 154 of class named "stat answers col-3" you can use either of the following solutions:

css_selector:

content = driver.find_element_by_css_selector("div.user-card#user-card div.answers span.number").get_attribute("innerHTML")

xpath:

content = driver.find_element_by_xpath("//div[@class='user-card' and @id='user-card']//div[@class='stat answers col-3']//span[@class='number']").get_attribute("innerHTML")

edited Aug 10 '18 at 05:15

Shivam Mishra

1,731
2
11
29

answered Aug 10 '18 at 04:33

undetected Selenium

183,867
41
278
352

1

Why not use ".text" ? – Shivam Mishra Aug 10 '18 at 05:12
1

@ShivamMishra Apparently both should work. The reason behind using using `get_attribute()` is to suggest minimal changes to preserve OP's current work and innovation. – undetected Selenium Aug 10 '18 at 05:18

How to extract the text 154 from nested div as per the html provided through Selenium and Python

1 Answers1