How to get HTML elements from dynamically generated page in Selenium?

Question

I'm trying to sign in my account in Figure eight

using selenium and python but i can't because there is no any HTML elements in the page only these lines exist.

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta content="ie=edge" http-equiv="x-ua-compatible">
    <title>Figure Eight Contributor Portal</title>
    <link href="/favicon.ico" rel="shortcut icon" />
    <link href="//fonts.googleapis.com/css?family=Lato:100,300,400,700,900" rel="stylesheet" />
    <link href="https://use.fontawesome.com/releases/v5.2.0/css/all.css" rel="stylesheet" />
  </head>
  <body>
    <div id="app">
    </div>
    <script src="/1.index.js?eca43d17e540e40a3231" type="text/javascript"></script>
    <script src="/index.js?eca43d17e540e40a3231" type="text/javascript"></script>
  </body>
</html>

Surprisingly when i use inspect element i can get the HTML elements. Searching for similar topic on stackoverflow and google but can't find any solution. My code trying to sign in is :

from selenium import webdriver
from selenium.webdriver.support.expected_conditions import presence_of_element_located
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver import Chrome

def sign_in(driver, url):
    driver.get(url)
    WebDriverWait(driver, 60).until(presence_of_element_located(driver.find_element_by_id('Email')))
    username = driver.find_element_by_id("Email")
    password = driver.find_element_by_id("Password")
    username.send_keys("##########")
    password.send_keys("@@@@@@@@@@")
    driver.find_element_by_name("Sign in").click()
    return True, "Success"

url = "https://contributors.figure-eight.work/login"
driver = Chrome()
result, message = sign_in(driver, url)
if result:
    print(message)

Please remember to mark the answer that solves your issue. See: https://stackoverflow.com/help/someone-answers — Jortega, Jan 04 '20 at 20:37
Your locators are not correct. There are no elements on the page with IDs of "Email" or "Password". You also should be waiting for visibility of and not presence of. Presence of means the element exists in the DOM but may or may not be interacted with. Waiting for visibility will ensure that it's present and ready to be interacted with. — JeffC, Jan 05 '20 at 00:22
@JeffC This statement _...Waiting for visibility will ensure that it's present and ready to be interacted with..._ is completely **wrong**, **incorrect** and **misleading**. _Visibility_ ensures _the expectation for checking that the element is present on the DOM of a page and visible. Visibility means that the elements are not only displayed but also has a height and width that is greater than 0_ but doesn't guarantees **clickability** / **interactability**. For **clickability** / **interactability** you need `element_to_be_clickable()` — undetected Selenium, Jan 05 '20 at 15:25
@DebanjanB 1. If you look at OP's code, the only thing he's using a wait on is the email address element. That doesn't need to be clicked, it only needs to be interacted with using sendkeys so waiting for visible is enough. 2.If you look at [the source](https://selenium.dev/selenium/docs/api/py/_modules/selenium/webdriver/support/expected_conditions.html#element_to_be_clickable) for wait for clickable, you will see that it waits for visible and not disabled. Not disabled only applies to INPUT elements so that check will be true on all other elements all the time anyway so is unnecessary. — JeffC, Jan 05 '20 at 16:08
@DebanjanB Also... wrong and incorrect mean the same thing and misleading is less severe than wrong. Before telling others they are wrong, you should probably make sure you are right. — JeffC, Jan 05 '20 at 16:14
@JeffC I think rather then criticizing OP's attempts you need to help them out. But now you are disappointing us with _...doesn't need to be clicked, it only needs to be interacted..._, do you still find them different? — undetected Selenium, Jan 05 '20 at 16:15
@DebanjanB I'm lost at what you are trying to accomplish here. I didn't criticize anything. You should read my comment again. Yes, clicked and interacted many times are different. If you send keys to an element, are you interacting with that element? — JeffC, Jan 05 '20 at 16:23

undetected Selenium · Accepted Answer · 2020-01-04T20:34:39.743

To send a character sequence to the Email and Password field you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:

Using CSS_SELECTOR:

driver.get('https://contributors.figure-eight.work/login')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[name='email']"))).send_keys("mohammed@awni.com")
driver.find_element_by_css_selector("input[name='password']").send_keys("mohammed@awni.com")
driver.find_element_by_css_selector("div.b-Login__ButtonHolder>a").click()

Using XPATH:

driver.get('https://contributors.figure-eight.work/login')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@name='email']"))).send_keys("mohammed@awni.com")
driver.find_element_by_xpath("//input[@name='password']").send_keys("mohammed@awni.com")
driver.find_element_by_xpath("//div[@class='b-Login__ButtonHolder']/a").click()

Note : You have to add the following imports:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Browser Snapshot:

score 1 · Answer 2 · answered Jan 04 '20 at 20:24

All the elements are present on the page and you can find those by inspecting any element on the page on chrome or firefox.

You can sign in using the code(Have added the xpaths for all the elements in the code below):

from selenium import webdriver
from selenium.webdriver.support.expected_conditions import 
presence_of_element_located
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver import Chrome



def sign_in(driver, url):
    driver.get(url)
    WebDriverWait(driver, 60).until(
    presence_of_element_located(
        driver.find_element_by_xpath("//input[@name='email']")))
    username = driver.find_element_by_xpath("//input[@name='email']")
    password = driver.find_element_by_xpath("//input[@name='password']")
    username.send_keys("##########")
    password.send_keys("@@@@@@@@@@")
    driver.find_element_by_xpath("//a[text()='Sign in']").click()
    return True, "Success"

url = "https://contributors.figure-eight.work/login"
driver = Chrome()
result, message = sign_in(driver, url)
if result:
    print(message)

There is an error in your code TypeError: find_element() argument after * must be a sequence, not WebElement. I had to replace WebDriverWait(driver,60).until(presence_of_element_located(driver.find_element_by_xpath("//input[@name='email']"))) with WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@name='email']"))).send_keys("######") to work. — mohammed awni, Jan 05 '20 at 10:54
@mohammedawni thanks for letting me know that, i might have missed that because i used your code and just replaced the xpaths in that. — Sameer Arora, Jan 05 '20 at 12:38

How to get HTML elements from dynamically generated page in Selenium?

2 Answers2