Not understanding how driver unable to locate element

Question

Here is the website: https://seekingalpha.com/news/3580581-docusign-eps-beats-0_02-beats-on-revenue

I am trying to webscrape the EPS, EPS beat, GEPS GEPS beat, and revenue revenue beat.

List1 = driver.find_element_by_xpath("""/html/body/div[2]/div[1]/div/main/div[2]/div[3]/div[2]/section[1]/div/div/div[3]/div/div/div[1]/ul/li[1]/text()[2]""")

This returns unable to locate element.

Also this does not work

List1 = driver.find_element_by_xpath("""/html/body/div[2]/div[1]/div/main/div[2]/div[3]/div[2]/section[1]/div/div/div[3]""")

I don't think it is due to the website being loaded. I put time.sleep to be 10 seconds and it also did not work. I am not sure how I navigate through the website to get to the list that has the information I want.

score 1 · Answer 1 · answered Dec 01 '20 at 00:36

Very strange issue. I was able to get the data with the following code:

driver = webdriver.Chrome(path_to_chrome_driver)
driver.get("https://seekingalpha.com/news/3580581-docusign-eps-beats-0_02-beats-on-revenue")
time.sleep(40)
driver.execute_script("window.scrollTo(0, 400)") 
time.sleep(20)
data = driver.find_elements_by_xpath('/html/body/div[2]/div[1]/div/main/div[2]/div[3]/div/section[1]/div/div/div[3]/div/div/div[1]/ul/li[1]')
for dat in data:
    print(dat.text)

It appears that the page does not fully load unless the user does some scrolling, perhaps for optimization purposes. Hopefully this works for you as well!

Thanks, the real problem in my case was that I forgot to switch tabs. I went onto seekingalpha website. Clicked on a link and did not switch to that new tab. Once I fixed it onto the correct tab then things worked again. — Ivan Pupo, Dec 01 '20 at 23:36

undetected Selenium · Accepted Answer · 2020-12-02T11:17:31.983

To print desired texts you have to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:

To print DocuSign (NASDAQ:DOCU): Q1 Non-GAAP EPS of $0.12 beats by $0.02; GAAP EPS of -$0.26 misses by $0.03.:

driver.get('https://seekingalpha.com/news/3580581-docusign-eps-beats-0_02-beats-on-revenue')
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@data-test-id='content-container']/ul/li"))).text)

To print Q1 Non-GAAP EPS of $0.12:

print(driver.execute_script('return arguments[0].childNodes[2].textContent;', WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@data-test-id='content-container']/ul/li")))).strip())

To print beats by $0.02:

print(driver.execute_script('return arguments[0].childNodes[3].textContent;', WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@data-test-id='content-container']/ul/li")))).strip())

To print GAAP EPS of -$0.26:

print(driver.execute_script('return arguments[0].childNodes[4].textContent;', WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@data-test-id='content-container']/ul/li")))).strip())

To print misses by $0.03:

print(driver.execute_script('return arguments[0].childNodes[5].textContent;', WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@data-test-id='content-container']/ul/li")))).strip())

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Not understanding how driver unable to locate element

2 Answers2