0

I'm using Selenium with python to extract comments from a website.

Eventually I end up with a list of WebElement-objects, each corresponding to a single comment. I then use element.get_element_by_xpath(XPATH) to locate different informations inside the comment-Object, such as the name of the commenter, the amount of likes etc.

The comments are all structured exactly the same, I've checked this with element.get_attribute('outerHTML').

But still, the Xpath-expressions only capture the relevant informations only every tenth time or so. The comments that are captured nicely don't seem to differ in any way from the other comments.

Has anyone experienced a similar problem, and maybe found a solution?

Edit: I found the problem wasn't the Xpath-expressions, but the way I tried to get the data from the elements (I used the text-attribute). This post here has the answer to the question I was actually trying to ask: getText() method of selenium chrome driver sometimes returns an empty string

backendboi
  • 131
  • 1
  • 6
  • 1
    are you seeing any exceptions? I suspect stale element references and/or lack of webdriverwait. – pcalkins May 13 '20 at 18:54
  • 1
    @pcalkins Thank you for your reply! After some further digging, I found that I asked the wrong question. The Xpaths are alright, but the .text-Attribute (I used, but didn't mention here) did sometimes returned empty strings. This post here came to the right solution: https://stackoverflow.com/questions/20888592/gettext-method-of-selenium-chrome-driver-sometimes-returns-an-empty-string – backendboi May 14 '20 at 15:03

0 Answers0