0

I'm trying to automate a process with python to visit a website with my username and password and check whether some appointments options are available. The reprex code could be accessed here.

The original code is shown below, and the selenium 4.11.2 works pretty well when the headless option is commented:

from dotenv import load_dotenv
import os
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
import time

load_dotenv()

# Replace these with actual values
login_url = 'https://prenotami.esteri.it/Services/Booking/2391'
login = os.getenv('LOGIN')
password = os.getenv('PASSWORD')
search_text = "Stante l'elevata richiesta i posti disponibili per il servizio scelto sono esauriti."

# Set up Chrome options for headless mode
chrome_options = Options()
# chrome_options.add_argument("--headless")

# Initialize the Selenium webdriver with the configured options
driver = webdriver.Chrome(options=chrome_options)
driver.implicitly_wait(3)

# Open the login page
driver.get(login_url)
driver.get_screenshot_as_file("1_login_page.png")

# Find the login and password input fields and fill them
login_field = driver.find_element('name', 'Email')
password_field = driver.find_element('name', 'Password')
login_field.send_keys(login)
password_field.send_keys(password)

# Submit the form (you can replace this with actual form submission method if needed)
password_field.send_keys(Keys.RETURN)

# Allow some time for the page to load and the login to complete
time.sleep(3)
driver.get_screenshot_as_file("2_message_page.png")

# Check page message
if search_text in driver.page_source:
    print('Closed appointment')
else:
    print("Open appointment")

# Close the browser when done
driver.quit()

The successful after-login-page could be seen in this picture.

But, unfortunately, I must run the process with headless set to True. In this case, the source page shows only the text Unavailable, as we can see after this commit onwards, which shows the picture of the after-login-page.

This Stackoverflow question looks mostly mine case and the solutions suggests adding User-Agent, what I did but didn't work (actually I tried both Windows and Linux user agent options).

Then, I tried adding many chrome options like suggested here, used a wait process as suggested here and even just add it language as another chrome option, but nothing worked.

What I'm missing here? It looks like the headless process is being blocked by the system. Am I right? If yes, it's another possibility to test besides what I've already done?

Gabriel Braico Dornas
  • 439
  • 3
  • 11

2 Answers2

1

Seemed like your site has anti-scrapping mechanism that detects if browser is in headless mode.

Use UndetectedSelunium driver to hide that you're in headless mode

Reference

import undetected_chromedriver as uc
from undetected_chromedriver import ChromeOptions

options = ChromeOptions()
options.add_argument('--headless')
driver = uc.Chrome(options)


driver.get("https://prenotami.esteri.it/Services/Booking/2391")
# your code

It's usually used to avoid CloudFlare / Captcha / headless mode detection. In your case it also would work (I tested and was able to see page after success login)

Yaroslavm
  • 1,762
  • 2
  • 7
  • 15
  • Hello @Yaroslavm, thank you so much for your help. It worked when I started the driver as suggested by the documentation or `driver = uc.Chrome(headless=True,use_subprocess=False)`. It's still a little strange because if I run twice, the second one will fail. If I wait some minutes (15 for example) it works. Can you imagine why this is happening? – Gabriel Braico Dornas Sep 02 '23 at 12:28
  • @GabrielBraicoDornas I tried to execute your code with undetected selenium 5 times in a row and each time it passed. Maybe, you had network problems / site was down? – Yaroslavm Sep 02 '23 at 12:41
0

Since that site uses anti-selenium services, you can use https://github.com/seleniumbase/SeleniumBase to reach the site:

from seleniumbase import Driver

driver = Driver(uc=True, headless=True)
driver.get("https://prenotami.esteri.it/Services/Booking/2391")
# ...
driver.quit()

(After pip install seleniumbase and running with python)

Michael Mintz
  • 9,007
  • 6
  • 31
  • 48