0

First of all if a similar topic occurred earlier I'm sorry but I couldn't find any problem like mine.

I would like to create a simple script which enters an e-mail website, log into my account and finds the amount of unread messages.

This is the part with logging in

from selenium import webdriver
from time import sleep

class sMailBot():
    def __init__(self):
        self.driver = webdriver.Chrome()

    def login(self):
        self.driver.get('website.com')

        sleep(2)

        btn_login = self.driver.find_element_by_xpath('//*[@id="username"]')
        btn_login.send_keys('my_username')

        btn_password = self.driver.find_element_by_xpath('//*[@id="password"]')
        btn_password.send_keys('my_password')

        btn_logintoaccount = self.driver.find_element_by_xpath('//*[@id="button"]')
        btn_logintoaccount.click()

        sleep(5)

It works really well. After logging into my mail account comments like driver.title or driver.current_url work.

Now I would like to scrape this part of html code:

<b>some_important_string_which_stores_the_amount_of_unread_mails</b>

I tried to do this using it's path

driver.find_element_by_xpath('//*[@id="MS_act1"]/span)

However it does not work. Moreover I can't find any other elements from this side.

I would like to highlight that I waiting even more than 10 seconds for the page to load.

The error which occurred

NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="MS_act1"]/span/b"}
  (Session info: chrome=80.0.3987.87)

As you asked I add some surrounding HTML code

<span style="float: right">
<b>some_important_string_which_stores_the_amount_of_unread_mails</b> 
</span>
DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
Hendrra
  • 682
  • 1
  • 8
  • 19
  • 1
    It might be in an ` – Guy Feb 10 '20 at 14:14
  • 1
    If you run the search from dev tools is it working ? – Marco Feb 10 '20 at 14:16
  • 1
    As Guy said, check if it's in an IFRAME. You said you are waiting 10s but in your code you are only waiting 5. 5s may not be enough. You should add a wait for 30s and see if that helps. Also, you please post more of the relevant HTML around the element you are looking for. Edit you question and add in the error message that is thrown when this fails. Thanks. – JeffC Feb 10 '20 at 14:39
  • @Marco No it doesn't work. – Hendrra Feb 11 '20 at 10:08
  • @JeffC I provided the things you asked. – Hendrra Feb 11 '20 at 10:08

3 Answers3

0

Please, don't use sleep, it's not a good choice for selenium.

Instead, use selenium waits:

driver = webdriver.Firefox()
driver.get("http://somedomain/url_that_delays_loading")
try:
    element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, "myDynamicElement"))
    )
finally:
    driver.quit()

https://selenium-python.readthedocs.io/waits.html

Ramon Medeiros
  • 2,272
  • 2
  • 24
  • 41
0

First of all I will avoid using sleep. You may try using WebDriverWait instead. This will pause the browser until a given condition is satisfied.

e.g. as follows

WebDriverWait(self.driver, 60).until(EC.presence_of_element_located((By.XPATH, "//button[text()='Login']")))

This will wait for 60 sec maximum for the element (button with text Login) to occur in the page.

HerculesOfHell
  • 117
  • 2
0

After logging into your mail account commands like driver.title and driver.current_url works but they are not part of the DOM Tree.

The relevant HTML would have helped us to construct a canonical answer. However to extract the desired text, you have to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:

  • Using CSS_SELECTOR and get_attribute("innerHTML"):

    print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "[id^='MS_act'] span>b"))).get_attribute("innerHTML"))

  • Using XPATH and text attribute:

    print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//*[starts-with(@id, 'MS_act')]//span/b"))).text)

  • Note : You have to add the following imports :

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352