How do I grab a header from this page with either BS4 or Selenium?

Question

I am quite new at this and am challenging myself to my first useful project beyond tic tac toe games and text based choose your own adventure games. Im trying to write a script which will show when the base price of some houses increases past their current values then send a sms to me saying it has increased. For now, I just need help getting the price to print in pycharm using either BS4 or Selenium. so far I have gotten selenium to find and open the page. When I inspect the Base Price element on the page it comes back as :

> "<h3 ng-show="model.mod_basePrice" class="ng-binding"> Base Price
> $278,900</h3>"

. I have tried everything I can find to get it to display the header or price. Ive been at this all day long. Im missing something and Im getting nowhere.

import selenium
from selenium import webdriver

PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)

driver.get("https://www.legacyhomesal.com/pennington-freedom-series-richmond-ii")
price = driver.find_elements_by_class_name(name='ng-binding')
print(price)

I have tried each of these and more:

driver.find_elements_by_class_name(name='ng-binding')
driver.find_elements_by_class_name('ng-binding')
driver.find_element_by_class_name(name='ng-binding')
driver.find_element_by_class_name('ng-binding')
driver.find_element_by_xpath("//*[@id='page']/section[2]/main/div[2]/div/div[2]/aside/h3")

I created my own Xpath as well. When I inspect the base price element on the page, and find //h3[@class='ng-binding' it does highlight the correct element so at least I know the xpath works. I tried the following with no results as well:

price = driver.find_element_by_xpath(xpath="//h3[@class='ng-binding']")
print(price)

The result I get back with the above is:

C:\Users\kgood\AppData\Local\Microsoft\WindowsApps\python.exe "C:/Users/kgood/Desktop/Legacy R2 Price.py"
<selenium.webdriver.remote.webelement.WebElement (session="0c6f3462267a3648682858dc0fd0d3c4", element="fdddf0be-c2ca-4466-996f-caac9d31e269")>

Process finished with exit code 0

Try price.text. Should output Base Price $278,900. – Arundeep Chohan Dec 08 '20 at 05:38 — Arundeep Chohan, Dec 08 '20 at 05:38

score 0 · Answer 1 · answered Dec 08 '20 at 05:40

What you want is the following to print the innerhtml.

price = driver.find_element_by_xpath("//h3[@class='ng-binding']")
print(price.text)

Also use webdriver waits for a more stable find.

price=WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//h3[@class='ng-binding']")))

Import

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC

score 0 · Answer 2 · answered Dec 08 '20 at 22:51

To print the text Base Price $278,900 you can use either of the following Locator Strategies:

Using css_selector and get_attribute():

driver.get('https://www.legacyhomesal.com/pennington-freedom-series-richmond-ii')
print(driver.find_element_by_css_selector("h3.ng-binding[ng-show='model.mod_basePrice']").get_attribute("innerHTML"))

Using xpath and text attribute:

driver.get('https://www.legacyhomesal.com/pennington-freedom-series-richmond-ii')
print(driver.find_element_by_xpath("//h3[@class='ng-binding']").text)

Ideally you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:

Using CSS_SELECTOR and text attribute:

driver.get('https://www.legacyhomesal.com/pennington-freedom-series-richmond-ii')
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "h3.ng-binding[ng-show='model.mod_basePrice']"))).text)

Using XPATH and get_attribute():

driver.get('https://www.legacyhomesal.com/pennington-freedom-series-richmond-ii')
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//h3[@class='ng-binding']"))).get_attribute("innerHTML"))

Console Output:
```
Base Price $278,900
```

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

References

Link to useful documentation:

get_attribute() method Gets the given attribute or property of the element.
text attribute returns The text of the element.
Difference between text and innerHTML using Selenium

How do I grab a header from this page with either BS4 or Selenium?

2 Answers2

References