Webscraping an element with style="display: none;" using Python and Selenium Webdriver

Question

I want to scrape web site for scheduling medical appointments in hospital in my country, for this purpose I use this code:

from selenium import webdriver

d = webdriver.Chrome()
d.get("http://www.mojtermin.mk/map/spetsijalist?specialty=meditsinska_genetika&location=grad_skopje&clinic=jzu_uk_za_ginekologija_i_akusherstvo&resource=873745910")
d.maximize_window()
d.implicitly_wait(20)
d.find_element_by_id('cdk-describedby-message-0').text
termins = d.find_element_by_id(id_='cdk-describedby-message-0').text

print(termins)

When I run code, Chrome is open successfully and show website, but find_element_by_id, didn't return anything only empty string.

when I inspect element in chrome for this element I find this :

<div id="cdk-describedby-message-container" aria-hidden="true" style="display: none;"><div id="cdk-describedby-message-0">Нема слободни термини во наредните 3 месеци.</div></div>

I expect find_element_by_id return value off the element, in this case :

"Нема слободни термини во наредните 3 месеци"

I tried many different ways but every time i get empty string or unable to find element.

Note: translate for value: "There are no free schedules in the next 3 months"

undetected Selenium · Answer 1 · 2020-09-02T14:25:59.837

The element with text as Нема слободни термини во наредните 3 месеци. is within the parent <div> element having the style attribute set as display: none;.

Hence, as the element is not visible on the webpage Selenium won't be able to interact with the WebElement in a generic way.

However, there are a couple of approaches through which you can extract the text as follows:

Using CSS_SELECTOR:

print(WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.CSS_SELECTOR, "div#cdk-describedby-message-0"))).get_attribute("innerHTML"))

Using XPATH:

print(WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, "//div[@id='cdk-describedby-message-0']"))).get_attribute("innerHTML"))

Using document.getElementById and textContent:

print(driver.execute_script("return document.getElementById('cdk-describedby-message-0').textContent;"))

Console Output:

Нема слободни термини во наредните 3 месеци.

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

I guess its not correct to say outright selenium wont be able to interact with such elements. You can always use Java script executioner to interact with such elements. your view ? — rahul rai, Sep 02 '20 at 13:51

score 0 · Answer 2 · answered Sep 02 '20 at 13:54

As <div> you are looking for is overlapped by another <div> with style attribute as display: none. You should use java script to get your text. Use below:

termins = d.execute_script("return   document.getElementById(\"cdk-describedby-message-0\").innerHTML;")

Output

score 0 · Accepted Answer · answered Sep 02 '20 at 13:56

Use get_attribute("textContent") instead of text.

Induce WebDriverWait() and wait for presence_of_element_located() and following css selector.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

d=webdriver.Chrome()
d.get("http://www.mojtermin.mk/map/spetsijalist?specialty=meditsinska_genetika&location=grad_skopje&clinic=jzu_uk_za_ginekologija_i_akusherstvo&resource=873745910")
d.maximize_window()
print(WebDriverWait(d,20).until(EC.presence_of_element_located((By.CSS_SELECTOR,"#cdk-describedby-message-container>#cdk-describedby-message-0"))).get_attribute("textContent"))

Output:

Нема слободни термини во наредните 3 месеци.

score 0 · Answer 4 · answered Sep 10 '20 at 06:23

Hi I found solution from @KunduK like a very useful, and I continued testing that solution, Here is my finally solution, plus I add function for send mail notification when its have free schedule.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import yagmail
import datetime
import time

local_time = time.ctime()

def send_mail():
    receiver = "somemail@hotmail.com"
    body = slot
    yag = yagmail.SMTP(user="somemail28@gmail.com", password="")
    yag.send(
    to=receiver,
    subject="Мој термин",
    contents=body
)

d=webdriver.Chrome()
d.get("http://www.mojtermin.mk/map/spetsijalist?specialty=interna_meditsina&location=grad_skopje&clinic=pzu_dr_aleksandar_manolev_skopje&resource=1221739979")
#d.maximize_window()
slot = (WebDriverWait(d,20).until(EC.presence_of_element_located((By.CSS_SELECTOR,"#cdk-describedby-message-container>#cdk-describedby-message-0"))).get_attribute("textContent"))
if slot == "Нема слободни термини во наредните 3 месеци.":
    print("Нема слободни термини во наредните 3 месеци.")
    print(local_time," - Nema slobodni termini", file=open("output.txt", "a"))
else:
    print(slot)
    send_mail()

d.quit()

Thanks to all for your effort

Webscraping an element with style="display: none;" using Python and Selenium Webdriver

4 Answers4