Why does the xpath gives NosuchElementException?

Question

I am going to scrape the comments of facebook video using selenium using xpath but it gives exception of no element found.

In this picture there is div with text "Comments" inside this <div> there is second <div> with <ul> and inside which there is <li> lists I want to fetch the text of those lists.

Codetrials:

from selenium.webdriver.common.by import By
comments = driver.find_element(by=By.XPATH, value='//div/h2[text()="Comments"]/div[2]/ul/li')

Snapshot:

Answer 1

As per the snapshot:

the ancestor <div> of the <ul> isn't a descendant of h2[text()="Comments"]. Hence you see exception of no element found

---

Solution

To print the comments you have to induce WebDriverWait for visibility_of_all_elements_located() and you can usethe following locator strategy:

print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div/h2[contains(., 'Comments')]//following-sibling::div[2]/ul//li")))])

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC