Scraping text with Python Selenium: unable to locate element which really exist

Question

I tried to scrape a text from following page source:

I used selenium and python to scrape "Diese Termine stehen zu ...".

What have I tried so far?

1. Using xpath to find elements and use the absolute location:

availability = driver.find_elements_by_xpath("//*[@id='booking-content']/div[2]/div[4]/div/div[2]/div/div/div/div[1]/div/div/span")

2. using class name:

elements = driver.find_elements_by_class_name("dl-text dl-text-body dl-text-regular dl-text-s dl-text-color-inherit")

3. using css selector:

use following keyword: .booking-message .dl-text

availability = driver.find_element_by_css_selector('.booking-message .dl-text')

All of above didn't work. With step 3, I am sure, it should work, because as can be seen in screenshot, I could find the element using the same keyword in Chrome. But still no luck.

The error message is:

Traceback (most recent call last):
  File "/Users/GunardiLin/Desktop/Codes/Tracker.py", line 18, in <module>
    availability = driver.find_element_by_css_selector('.booking-message .dl-text')
  File "/Users/GunardiLin/opt/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 598, in find_element_by_css_selector
    return self.find_element(by=By.CSS_SELECTOR, value=css_selector)
  File "/Users/GunardiLin/opt/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 978, in find_element
    'value': value})['value']
  File "/Users/GunardiLin/opt/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/Users/GunardiLin/opt/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".booking-message .dl-text"}
  (Session info: chrome=90.0.4430.212)

I am aware of another posting with the same problem: Python with selenium: unable to locate element which really exist

Which is why I checked if the site is using "iframe". I checked it by searching for "iframe-tags" just like in the screenshot. The search result is 0, which means nothing is found.

Could someone give a pointer how to scrape the text? I prefer to use the css selector (option 3) and dislike using option 1 (xpath + absolute location). But currently I would be thankful with any solution.

Thank you in advance:-)

Update:

My code so far:

import os
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import Select

url = r"https://www.doctolib.de/gemeinschaftspraxis/muenchen/fuchs-hierl?practitioner_id=any&speciality_id=5593&utm_campaign=website-button&utm_source=fuchs-hierl-website-button&utm_medium=referral&utm_content=custom&utm_term=fuchs-hierl"

chrome_options = Options()
chrome_options.add_argument('--headless')
driver = webdriver.Chrome(executable_path="/Applications/chromedriver", options=chrome_options)
driver.get(url)
print('*** Title:', driver.title)
# print(driver.page_source.encode("utf-8"))
dropdown_besuchgrund = driver.find_element_by_id("booking_motive")
select_besuchgrund = Select(dropdown_besuchgrund)
# print(dir(select_besuchgrund))
select_besuchgrund.select_by_visible_text("Erste Impfung Covid-19 (BioNTech-Pfizer)")
# availability = driver.find_elements_by_xpath("//*[@id='booking-content']/div[2]/div[4]/div/div[2]/div/div/div/div[1]/div/div/span")
#elements = driver.find_elements_by_class_name("dl-text dl-text-body dl-text-regular dl-text-s dl-text-color-inherit")
# availability = driver.find_element_by_css_selector('.booking-message .dl-text')
availability = driver.find_element_by_xpath(".//div[contains(@class,'booking-message')]/span")
print("***")
print(availability.text)
# for elem in elements:
#     print ("***", elem.text)
#     if elem.text == "Diese Termine stehen zu einem späteren Zeitpunkt wieder für eine Online-Buchung zur Verfügung. ":
#         print("*** Ausgebucht")
driver.quit()

@itronic1990 22.05.2021 07:45: I have checked your suggestion with:

driver.find_element_by_xpath(".//div[contains(@class,'booking-message')]/span").text

As you can see above, chrome can find the text with your filter. But if I run the code, it can't find it. My test code:

import os
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
url = r"https://www.doctolib.de/gemeinschaftspraxis/muenchen/fuchs-hierl"
chrome_options = Options()
chrome_options.add_argument('--headless')
driver = webdriver.Chrome(executable_path="/Applications/chromedriver", options=chrome_options)
driver.get(url)
element_text = driver.find_element_by_xpath(".//div[contains(@class,'booking-message')]/span").text
print(element_text)
driver.quit()

Error Message:

NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":".//div[contains(@class,'booking-message')]/span"}
  (Session info: headless chrome=90.0.4430.212)

I can't understand how could why? Thank you for any advice.

Answer 1

You have used find_elements in xpath and by classname. Is that right?

Try this

driver.find_element_by_xpath(".//div[contains(@class,'booking-message')]/span").text

Answer 2

Why bother with Selenium? Fetch the data straight from the source:

import requests

url = 'https://www.doctolib.de/availabilities.json'
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36'}
payload = {
'start_date': '2021-05-21',
'visit_motive_ids': '2820334',
'agenda_ids': '466608',
'insurance_sector': 'public',
'practice_ids': '25230',
'limit': '4'}

jsonData = requests.get(url, headers=headers, params=payload).json()

Output:

print(jsonData['message'])
Diese Termine stehen zu einem späteren Zeitpunkt wieder für eine Online-Buchung zur Verfügung. 

I'm not familiar with German, otherwise I might be able to make this more efficient. Put basically use the practice_id to feed into it and get the data from each practice.

import requests
from bs4 import BeautifulSoup
from datetime import datetime

# Get location practice_ids
url = 'https://www.doctolib.de/allgemeinmedizin/81667-muenchen'
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36'}

practice_ids_list = []
for page in range(1,100):
    payload = {'page':page}

    response = requests.get(url, headers=headers, params=payload)
    if response.status_code == 404:
        break
    
    else:
        print('Page: %s' %page)
        soup = BeautifulSoup(response.text, 'html.parser')
        divs = soup.find_all('div',{'class':'dl-search-result'})
        
        for div in divs:
            practice_id = div['id'].split('-')[-1]
            practice_ids_list.append(practice_id)

today = datetime.today().strftime('%Y-%m-%d')

url = 'https://www.doctolib.de/availabilities.json'
for practice_id in practice_ids_list:
    payload = {
    'start_date': today,
    'visit_motive_ids': '2820334',
    'agenda_ids': '466606',
    'insurance_sector': 'public',
    'practice_ids': '%s' %practice_id,
    'limit': '15'}
    
    jsonData = requests.get(url, headers=headers, params=payload).json()
    
   
    if jsonData['total'] == 0 and 'next_slot' not in jsonData.keys():
        #print('\t', jsonData['message'],'\n')
        print(practice_id)
    else:
        # Get Clinic Details
        clinic_url = 'https://www.doctolib.de/search_results/%s.json' %practice_id
        clinic_jsonData = requests.get(clinic_url, headers=headers).json()
        clinic_name = clinic_jsonData['search_result']['name_with_title']
        address = clinic_jsonData['search_result']['address']
        city = clinic_jsonData['search_result']['city']
        zipcode = clinic_jsonData['search_result']['zipcode']
        print('%s\n%s %s %s' %(clinic_name, address, city, zipcode))
        
        payload.update({'start_date':jsonData['next_slot']})
        jsonData = requests.get(url, headers=headers, params=payload).json()
        print('\n\t','*'*50,'\nThe follow dates are available:')
        for each_date in jsonData['availabilities']:
            if len(each_date['slots']) > 0:
                print('\t\t',each_date['date'])