1

I am trying to use python, selenium and xpath to grab the text $0.180 in the following block of HTML,

<div class="wlp-values-div">
  <span class="wlp-last-price-span">$0.180</span>

This code is deep into a websites HTML and I tried to use the following Xpath to locate it:

import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

driver = webdriver.Chrome()
driver.get("https://smallcaps.com.au/stocks/?symbol=AWJ")
time.sleep(3)
webdriver.ActionChains(driver).send_keys(Keys.ESCAPE).perform()

ele = driver.find_element_by_xpath("//div[@class='wlp-values-div']/span[@class='wlp-last-price-span']")

But I receive the error:

NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//div[@class='wlp-values-div']/span[@class='wlp-last-price-span']"}

Any help would be greatly appreciated

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
lachlan
  • 13
  • 2

2 Answers2

1

The last price field is within nested <iframe> elements so you have to:

  • Induce WebDriverWait for the parent frame to be available and switch to it.

  • Induce WebDriverWait for the child frame to be available and switch to it.

  • Induce WebDriverWait for the desired element to be visible.

  • You can use either of the following Locator Strategies:

    • Using CSS_SELECTOR:

      driver.get('https://smallcaps.com.au/stocks/?symbol=AWJ')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "svg[data-name='close']"))).click()
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe#wl-summary")))
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe.wl-quote-frame")))
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "span.wlp-last-price-span"))).get_attribute("innerHTML"))
driver.quit()

    • Using XPATH:

      driver.get('https://smallcaps.com.au/stocks/?symbol=AWJ')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//*[name()='svg' and @data-name='close']"))).click()
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[@id='wl-summary']")))
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[@class='wl-quote-frame']")))
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//span[@class='wlp-last-price-span']"))).get_attribute("innerHTML"))
driver.quit()

  • Note : You have to add the following imports :

     from selenium.webdriver.support.ui import WebDriverWait
 from selenium.webdriver.common.by import By
 from selenium.webdriver.support import expected_conditions as EC

  • Console Output:

      $0.190

You can find a detailed relevant discussion in How To sign in to Applemusic With Python Using Chrome Driver With Selenium

Reference

You can find a couple of relevant discussions in:

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
  • 1
    wow thats incredibly clear! ive just had a chance to read this and cant thank you enough! The way you can breakdown a complex topic into such simple terms is a credit to your own abilities and knowledge – lachlan Feb 17 '21 at 09:36
0
<iframe class="wl-summary" id="wl-summary" src="https://cloud.weblink.com.au/clients/smallcaps/summary/summary.aspx?symbol=AWJ&amp;" width="100%" height="25" frameborder="0" scrolling="no" allowtransparency="true" style="height: 1077px;"></iframe>
<iframe class="wl-quote-frame" src="quoteFrame.aspx?symbol=AWJ&amp;" scrolling="no" frameborder="0" height="1" style="height: 177px;"></iframe>

It's in a double nested iframe so you have to switch to it. This also has the appropriate waits instead of using time.sleep() and hoping the elements load. It also clicks the popup to get out of the way and prints the spans text.

wait = WebDriverWait(driver, 10)
driver.get("https://smallcaps.com.au/stocks/?symbol=AWJ")
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, '#tve_editor > div'))).click()
wait.until(EC.frame_to_be_available_and_switch_to_it((By.CLASS_NAME, 'wl-summary')))
wait.until(EC.frame_to_be_available_and_switch_to_it((By.CLASS_NAME, 'wl-quote-frame')))
elem=wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, 'span.wlp-last-price-span')))
print(elem.text)

Import

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC
Arundeep Chohan
  • 9,779
  • 5
  • 15
  • 32