How to extract the contents using Selenium and Python

Question

Considering the HTML:

I want to select the paragraphs to the left with Selenium. Tried my hand at class_name and id but got NoSuchElementException. Why am I getting this error? I mean the elements are clearly there then why isn't Selenium recognizing those? Methods I tried:

element = driver.find_element_by_xpath("//div[@id = 'mar-2019']//div[@class='report_data']").text

element = driver.find_element_by_id("mar-2019").text

element = driver.find_element_by_class_name("report_data").text

Where am I going wrong?

score 0 · Answer 1 · answered Jun 04 '20 at 09:32

To handle dynamic element induce WebDriverWait() and wait for visibility_of_element_located()

element=WebDriverWait(driver,10).until(EC.visibility_of_element_located((By.ID,"mar-2019"))).text

OR

element=WebDriverWait(driver,10).until(EC.visibility_of_element_located((By.CSS_SELECTOR,"div#mar-2019"))).text

You need to import following libraries.

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

score 0 · Answer 2 · answered Jun 04 '20 at 12:12

There are multiple child <p> elements within multiple parent/ancestor <div> elements. To extract the contents of the <p> elements within the parent <div id="mar-2019"> element you need to induce WebDriverWait for the visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

Using CSS_SELECTOR and get_attribute("innerHTML"):

print([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, ".data-row")))])

Using XPATH and text attribute:

print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@id='mar-2019']//div[@class='report_data']//p")))])

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Reference

You can find a couple of relevant discussions on NoSuchElementException in:

How to extract the contents using Selenium and Python

2 Answers2

Reference