Need Help Trying to identify the username section and pw section of a page (python selenium)

Question

I am essentially scraping chegg and I need help in trying to identify the css id= "emailForSignIn" I am trying to find a way to detect where the text bar for the login section is for the chegg login page

Below is the function implemented:

def signin():  # Only use this function if you are using new instances of your browser each time
    print('>>signing in!')
    browser.get('https://www.chegg.com/auth?action=login&redirect=https%3A%2F%2Fwww.chegg.com%2F')
    handle_captcha()

    time.sleep(2)
    email_elem = browser.find_element_by_id('emailForSignIn')
    for character in 'alondra_calderon@ymail.com':
        email_elem.send_keys(character)
        time.sleep(0.1)
    time.sleep(2)

    password_elem = browser.find_element_by_id('passwordForSignIn')
    for character in 'Blueyes97':
        password_elem.send_keys(character)
        time.sleep(0.1)
    time.sleep(2)

    browser.find_element_by_name('login').click()

    try:
        if WebDriverWait(browser, 5).until(EC.presence_of_element_located((By.XPATH, "/html/body/div[1]/div[3]/div[2]/div[2]/div/div[3]/div/oc-component/div/div/div/div[2]/div[1]/div[1]/div/form/div/div/div/div/div[3]/span"))):
            print('redirecting back to login')
            browser.get('https://www.chegg.com/auth?action=login')
            handle_captcha()
            signin()
            handle_captcha()
    except TimeoutException:
        pass

    if browser.find_element_by_tag_name('h1').text == 'Oops, we\'re sorry!':
        return [0]
    handle_captcha()

Below is the error:

PS C:\Users\Dami\Desktop\chegg_discord_bot-master>  & 'C:\Program Files (x86)\Python38-32\python.exe' 'c:\Users\Dami\.vscode\extensions\ms-python.python-2020.8.101144\pythonFiles\lib\python\debugpy\launcher' '58627' '--' 'c:\Users\Dami\Desktop\cheggDiscordBot.py'

DevTools listening on ws://127.0.0.1:58638/devtools/browser/837850c8-796e-44be-950f-49f667f48f0a
>>signing in!
Traceback (most recent call last):
  File "c:\Users\Dami\Desktop\cheggDiscordBot.py", line 288, in <module>
    signin()
  File "c:\Users\Dami\Desktop\cheggDiscordBot.py", line 257, in signin
    email_elem = browser.find_element_by_id('emailForSignIn')
  File "C:\Users\Dami\AppData\Roaming\Python\Python38\site-packages\selenium\webdriver\remote\webdriver.py", 
line 360, in find_element_by_id
    return self.find_element(by=By.ID, value=id_)
  File "C:\Users\Dami\AppData\Roaming\Python\Python38\site-packages\selenium\webdriver\remote\webdriver.py", 
line 976, in find_element
    return self.execute(Command.FIND_ELEMENT, {
  File "C:\Users\Dami\AppData\Roaming\Python\Python38\site-packages\selenium\webdriver\remote\webdriver.py", 
line 321, in execute
    self.error_handler.check_response(response)
  File "C:\Users\Dami\AppData\Roaming\Python\Python38\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="emailForSignIn"]"}
  (Session info: headless chrome=84.0.4147.125)

I am not sure if your captcha handle method is able to navigate to Email / Password enter page. As i can not see any issue with Locator. Please check if its possible that captcha is not handled properly. — rahul rai, Aug 17 '20 at 12:52

NarendraR · Answer 1 · 2020-08-18T05:56:00.393

Seems synchronization issue as below error indicates-

Unable to locate element: {"method":"css selector","selector":"[id="emailForSignIn"]"}

Introduce Explicit wait in your script and change your code as given below:

WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.ID, "emailForSignIn"))).send_keys("alondra_calderon@ymail.com")
WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.ID, "passwordForSignIn"))).send_keys("Blueyes97")
driver.find_element_by_name('login').click()

Note: No need to loop through the email and password Selenium automatically enters the text in same way

Import following packages for that:

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait

score 0 · Answer 2 · answered Aug 17 '20 at 14:38

To send a character sequence to the Email and Password field you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:

Using CSS_SELECTOR:

driver.get('https://www.chegg.com/auth?action=login&redirect=https%3A%2F%2Fwww.chegg.com%2F')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#emailForSignIn"))).send_keys("alondra_calderon@ymail.com")
driver.find_element_by_css_selector("input#passwordForSignIn").send_keys("passwordForSignIn")

Using XPATH:

driver.get('https://www.chegg.com/auth?action=login&redirect=https%3A%2F%2Fwww.chegg.com%2F')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@id='emailForSignIn']"))).send_keys("alondra_calderon@ymail.com")
driver.find_element_by_xpath("//input[@id='passwordForSignIn']").send_keys("passwordForSignIn")

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Browser Snapshot:

Need Help Trying to identify the username section and pw section of a page (python selenium)

2 Answers2