How to retrieve text from HTML to Python using Selenium

Question

Hi I know there's questions already pertaining to this but I do not have enough experience to figure it out. I am trying to write a simple script in Python that periodically checks the earliest available date on a DMV website. Where I live, the DMV is backed up for months and after just failing a drivers test - I want to snag the earliest available date when somebody cancels their appointment.

Anyways, here is the HTML I am trying to grab from:

          <div _ngcontent-glu-c19="" class="department-appointment-header">Earliest date:</div>
          <br _ngcontent-glu-c19="">
          <div _ngcontent-glu-c19="">
            Monday
          </div>
          <div _ngcontent-glu-c19="">
             May 31st
          </div>
        </div>

Now, I am trying to grab that May 31st date so I can compare it with an Earliest Date variable that continuously updates when there is a sooner date than the existing one. Eventually I will have Python notify me by text. I can't figure out how to retrieve the May 31st element and assign it to a string variable or list, so I can convert the month/day to an integer between 1 - 365.

Please I'm new to Selenium and I haven't touched Python in awhile, I'm quite rusty and all help would be appreciated. If you need more of the HTML code then let me know I'll add more, I just didnt want to fill this entire page.

Hi, are you still having trouble with question? – Arundeep Chohan Jan 09 '21 at 07:37 — Arundeep Chohan, Jan 09 '21 at 07:37

score 1 · Answer 1 · answered Jan 09 '21 at 19:23

To print the text May 31st you can use either of the following Locator Strategies:

Using xpath and class attribute:

print(driver.find_element(By.XPATH, "//div[@class='department-appointment-header']//following-sibling::div[2]").text)

Using xpath and textContext:

print(driver.find_element(By.XPATH, "//div[text()='Earliest date:']//following-sibling::div[2]").text)

Ideally you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:

Using XPATH and class attribute:

print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='department-appointment-header']//following-sibling::div[2]"))).text)

Using XPATH and textContext:

print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[text()='Earliest date:']//following-sibling::div[2]"))).text)

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

You can find a relevant discussion in How to retrieve the text of a WebElement using Selenium - Python

Mick · Answer 2 · 2021-01-09T06:08:35.430

0

try:

date = driver.find_element_by_xpath("//div[@_ngcontent-glu-c19='']).text

I'm not sure it'll work but it's worth a try

edited Jan 09 '21 at 06:08

answered Jan 09 '21 at 05:58

Mick

796
4
8

Not a valid XPath expression – anaonyoamosu Jan 09 '21 at 06:05
Sorry lol, single quotes instead of double – Mick Jan 09 '21 at 06:09
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//div[@_ngcontent-glu-c19=""]"} – anaonyoamosu Jan 09 '21 at 06:16
Hey I figured it out, I got it to print "May 31st" using date = driver.find_element_by_xpath("//div[2]/div/div/div[2]/div[3]").text – anaonyoamosu Jan 09 '21 at 06:20
Yea but xpaths like that should be avoided, they aren't reproducible and will break if anything changes – Mick Jan 09 '21 at 06:22
Okay fair point. I used the Selenium IDE to find that target however it gave me other options like a css locator - .department-appointment > div:nth-child(4) However, I couldn't get this to work in my code because I'm not sure what comes before the first . – anaonyoamosu Jan 09 '21 at 06:25
It's also not very reproducible, generally you want a more human xpath that will work if the website changes. Hold on, I almost got it – Mick Jan 09 '21 at 06:26
I got the parent element using element = driver.find_element_by_css_selector("div.department-appointment-header"). I'm still trying to isolate the children – Mick Jan 09 '21 at 06:32

How to retrieve text from HTML to Python using Selenium

2 Answers2