Selenium Webdriver How to scrape text?

Question

Here's the website code:

<div class="question-row clearfix " lang="en">
    <div class="qquestion qtext  ">
        five
    </div>
</div>

I'm trying to scrape the text (in this example, it's 'five'). Here's my current code:

qtitle = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//*[@id='prompt-row'div/div"))).get_attribute("text")

However when I run it, this error code comes up:

  File "C:\Users\lycop\AppData\Local\Programs\Python\Python38-32\lib\site-packages\selenium\webdriver\support\wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

That's all it says. Any help on solving this problem would be appreciated.

undetected Selenium · Accepted Answer · 2020-07-25T15:57:55.057

To print the text five you can use either of the following Locator Strategies:

Using css_selector and get_attribute():

print(driver.find_element_by_css_selector("div.qquestion.qtext").get_attribute("innerHTML"))

Using xpath and text attribute:

print(driver.find_element_by_xpath("//div[@class='qquestion qtext  ']").text)

Ideally, to print the innerText of an element you have to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following solutions:

Using CSS_SELECTOR:

print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "button.save[type='submit'][onclick]"))).get_attribute("innerHTML"))

Using XPATH:

print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//button[@class='save' and text()='save'][@type='submit' and @onclick]"))).text)

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

You can find a couple of relevant discussions on NoSuchElementException in:

Now it comes up with the error: "selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//div[@class='qquestion qtext']"} (Session info: chrome=84.0.4147.89)". I think this is because the page hasn't loaded entirely yet, which is why I used 'WebDriverWait'. How would I implement WebDriverWait into this as well? — Jack Preston, Jul 25 '20 at 15:57

Selenium Webdriver How to scrape text?

1 Answers1