Selenium - searching for an element inside iterated element

Question

I'm trying to scrape Facebook posts off a site of one band, but I get an error while searching in an iterated WebElement:

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"class name","selector":"userContent"}

The posts are found successfully, but the code breaks while searching for post_text_element. I tried searching by XPATH as well, but the result was same.

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By


SITE_URL = 'https://www.facebook.com/pg/mnagaazdorp/posts/'
POSTS_XPATH = "//*[contains(@class, '_4-u2') and contains(@class, '_4-u8')]"
POST_TEXT_CLASS = "userContent"
TIMEOUT = 1
CHROME_DRIVER_PATH = "C:\\Users\\tonda\\Documents\\chromedriver.exe"

browser = webdriver.Chrome(executable_path=CHROME_DRIVER_PATH)

browser.get(SITE_URL)

wait = WebDriverWait(browser, TIMEOUT)

posts = browser.find_elements_by_xpath(POSTS_XPATH)

for post in posts:
    post_text_element = post.find_element_by_class_name(POST_TEXT_CLASS)
    print(post_text_element.text)

browser.quit()

there is no element with `userContent` class name in '//*[contains(@class, '_4-u2') and contains(@class, '_4-u8')]' div. That's why you are getting that error. can you show us which text you are trying to extract, so that I can give you the code. — supputuri, Mar 12 '19 at 23:24
The facebook css is dynamic. You can't expect it to be the same twice. — pguardiario, Mar 13 '19 at 00:20

C. Peck · Accepted Answer · 2019-03-12T23:31:38.287

So, post is a WebElement member of the posts array. The syntax

post.find_element_by_class_name(POST_TEXT_CLASS)

will only return an element if there is a matching one that is a direct child of post. After clicking on your link and inspecting there aren't any direct children of post that have a class of userContent.

However, if you try the following, you should get an array of WebElements that consists of all divs below the xpath specified for POSTS_XPATH that have a class of userContent, which I believe is your goal. Let me know if the following helps:

posts = browser.find_elements_by_xpath("//*[contains(@class, '_4-u2') and contains(@class, '_4-u8')]//div[contains(@class, 'userContent')]")

Then you can simply loop through the array and print the text of each post if that is your goal.

for post in posts:
    print(post.text)

That works just fine, thank you very much. – Antonín Melenovský Mar 13 '19 at 17:26 — Antonín Melenovský, Mar 13 '19 at 17:26

Selenium - searching for an element inside iterated element

1 Answers1