0

What's the problem?

Im currently trying to scrape data from a subreddit (I am using the old-reddit chrome-extension that gives back the old look of reddit -> this way it's easier to scrape), but whenever I'm trying to get the results I get the error from this little bit of code:

xpath = "//a[@class='title may-blank loggedin ']"
element = driver.find_element_by_xpath(xpath)

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//a[@class='title may-blank loggedin ']"}

What did I try to fix the problem?

I already saw many posts with similar errors, often related with scraping the results before the page was loaded. I tried to fix that with:

time.sleep(20)

But still no diffrence.

The path is correct as well. I entered the same path on Chrome's console and it displayed correct results.

When I search for tag names, class names etc., I get correct results as well.

Thank you for your help in advance!!

Stack trace

Traceback (most recent call last):
  File "D:\Web_Dev\Projekte\cs50_project\test.py", line 68, in <module>
    main()
  File "D:\Web_Dev\Projekte\cs50_project\test.py", line 32, in main
    element = driver.find_element_by_xpath(xpath)
  File "C:\Users\User\AppData\Roaming\Python\Python39\site-packages\selenium\webdriver\remote\webdriver.py", line 394, in find_element_by_xpath
    return self.find_element(by=By.XPATH, value=xpath)
  File "C:\Users\User\AppData\Roaming\Python\Python39\site-packages\selenium\webdriver\remote\webdriver.py", line 976, in find_element
    return self.execute(Command.FIND_ELEMENT, {
  File "C:\Users\User\AppData\Roaming\Python\Python39\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "C:\Users\User\AppData\Roaming\Python\Python39\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response    
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//a[@class='title may-blank loggedin ']"}
  (Session info: chrome=88.0.4324.150)
Deepak Rai
  • 2,163
  • 3
  • 21
  • 36
CodingForFun
  • 21
  • 5
  • Try it with xpath = "//a[@class='title.may-blank.loggedin ']" or just xpath = "//a[@class='loggedin ']" – Frederick Feb 06 '21 at 11:48
  • If this doesnt work, could you give us the url? – Frederick Feb 06 '21 at 11:49
  • Yes it its https://www.reddit.com/search/?q=cs50 but I am using a chrome extension that changes the layout of the page https://chrome.google.com/webstore/detail/old-reddit-redirect/dneaehbmnbhcippjikoajpoabadpodje – CodingForFun Feb 06 '21 at 13:51
  • I am not sure if it works with the extention, I can just help you without the extention, so what to you need from the page? – Frederick Feb 06 '21 at 13:54
  • I have tried both of your proposals but neither works unfortunately, I am stil getting the same error – CodingForFun Feb 06 '21 at 13:54
  • What do you need from the page? The posts? Because I cant find anything using the classes above – Frederick Feb 06 '21 at 13:55
  • I want to get all the posts titles and I could do it with "find by class" but the problem is that im getting ads titles as well. That is why I want to use xpath so that I can access the siblings as well (the sibling has a little footnote that declares it an ad) – CodingForFun Feb 06 '21 at 14:06
  • Did you try this one: xpath = "//a[@class='title.may-blank.loggedin ']" – Frederick Feb 06 '21 at 14:16
  • It works now, tried it before but it didnt work but now its finally running, I have no idea how, but thanks a lot!! – CodingForFun Feb 06 '21 at 14:26

2 Answers2

1

i would try to multiply classes in xpath, like that:

    xpath = "//a[@class='title'][@class='may-blank'][@class='loggedin']"
    element = driver.find_element_by_xpath(xpath)

or like that:

    xpath = "//a[@class='title' and @class='may-blank' and @class='loggedin']"
    element = driver.find_element_by_xpath(xpath)
Daniel
  • 46
  • 3
0

What happened?

Calling this XPath "//a[@class='title may-blank loggedin ']" does not work for Selenium because the space is like a delimiter for classes in HTML so you are looking for a class which cant exsist.

How to solve it?

Solving this issue is very simple just put dots instead of delimiter, to show that you look for an element which has all three classes like this:

    xpath = "//a[@class='title.may-blank.loggedin']"
    element = driver.find_element_by_xpath(xpath)
Frederick
  • 450
  • 4
  • 22
  • IMO, `xpath = "//a[@class='title.may-blank.loggedin']"` isn't a valid xpath. – undetected Selenium Feb 06 '21 at 14:44
  • When I look at your profile it seems like you know a little bit about Selenium, so you are right but why does it work for him and what would be the right answer? – Frederick Feb 06 '21 at 15:21
  • Would this be the soloution? //div[contains(@class, 'class1') and contains(@class, 'class2')] got it from here https://stackoverflow.com/questions/3881044/how-to-get-html-elements-with-multiple-css-classes – Frederick Feb 06 '21 at 15:28