How to access HTML text using selenium/python?

Question

I have HTML tags as below:

<div class="mt-md common__EiReviewTextStyles__allowLineBreaks">
    <p class="strong">Pros</p>
    <p>This is the text to be captured</p></div>

I want to access the text " This is the text to be captured " using selenium on Python.

Any ideas on how this can be done?

There are a huge number of blogs and sites that will give you examples of web scraping. This question is not quite suitable for stackoverflow in its current form. I would suggest making an attempt at the problem and show your attempt if it doesnt work and people will help — Chris Doyle, Nov 28 '19 at 14:47
s = """

Pros

This is the text to be captured
""" You can try to use BeautifulSoup ```from bs4 import BeautifulSoup soup = BeautifulSoup(s) print(soup.text) ``` — Petr Matuska, Nov 28 '19 at 15:56
This gives me only 'Pros' as output. I want to get the next line too in my output — user3021495, Nov 28 '19 at 16:34
try with this xpath - this will return the text you need . //div/p[@class='strong']/following-sibling::p/text() — Sureshmani Kalirajan, Nov 28 '19 at 17:11

score 0 · Answer 1 · answered Nov 29 '19 at 03:20

Use .find_element_by_xpath, like bellow:

driver.find_element_by_xpath('//div[contains(@class, "mt-md")]//p[@class="strong"]//following-sibling::p[text()]').text

Or if you want achieve all text in your div, use .find_element_by_css_selector:

driver.find_element_by_css_selector('div.mt-md.common__EiReviewTextStyles__allowLineBreaks').text

score 0 · Answer 2 · answered Nov 29 '19 at 08:28

To extract the text within the second <p> tag i.e. This is the text to be captured you have to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:

Using CSS_SELECTOR and get_attribute():

print(WebDriverWait(browser, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.mt-md.common__EiReviewTextStyles__allowLineBreaks p:nth-of-type(2)"))).get_attribute("innerHTML"))

Using XPATH and text:

print(WebDriverWait(browser, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='mt-md common__EiReviewTextStyles__allowLineBreaks']//following-sibling::p[2]"))).text)

How to access HTML text using selenium/python?

2 Answers2