Selenium - Getting error page when trying to load site?

Question

I try to load this site https://www.pferdewetten.de/ with the following code:

import time
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from sys import platform
import os, sys
import xlwings as xw
from selenium.webdriver.support.ui import WebDriverWait
from webdriver_manager.chrome import ChromeDriverManager
from fake_useragent import UserAgent

if __name__ == '__main__':
  SAVE_INTERVAL = 5
  WAIT = 3
   
  print(f"Checking chromedriver...")
  os.environ['WDM_LOG_LEVEL'] = '0' 
  ua = UserAgent()
  userAgent = ua.random
  options = Options()
  # options.add_argument('--headless')
  options.add_experimental_option ('excludeSwitches', ['enable-logging'])
  options.add_experimental_option("prefs", {"profile.default_content_setting_values.notifications": 1})    
  options.add_argument("--disable-infobars")
  options.add_argument("--disable-extensions")  
  options.add_argument("start-maximized")
  options.add_argument('window-size=1920x1080')                               
  options.add_argument('--no-sandbox')
  options.add_argument('--disable-gpu')  
  options.add_argument(f'user-agent={userAgent}')   
  srv=Service(ChromeDriverManager().install())
  driver = webdriver.Chrome (service=srv, options=options)    
  waitWebDriver = WebDriverWait (driver, 10)         
  
  link = f"https://www.pferdewetten.de/" 
  driver.get (link)

But i allways only get this information:

Is there any way to load this site using selenium?

score 1 · Accepted Answer · answered Apr 09 '22 at 20:57

Possibly elenium driven ChromeDriver initiated google-chrome Browsing Context is geting detected as a bot.

To evade the detection you can make a few tweaks as follows:

Remove the --no-sandbox argument and execute as non-root user.
Remove the --disable-infobars argument as it is no more effective.
Remove the --disable-extensions argument as it is no more effective.
Add an experimental option "excludeSwitches", ["enable-automation"] to evade detection.
Add an experimental option 'useAutomationExtension', False to evade detection.
Add the argument '--disable-blink-features=AutomationControlled' to evade detection.

Effectively your code block will be:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

options = Options()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('excludeSwitches', ['enable-logging'])
options.add_experimental_option('useAutomationExtension', False)
options.add_argument('--disable-blink-features=AutomationControlled')
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
driver.get("https://www.pferdewetten.de/")
driver.quit()

Thanks for the answer - it seems not to work. Only for my understanding - what do you mean with "execute as non-root user" - where are you doing this in your code? — Rapid1898, Apr 10 '22 at 15:48
_execute as non-root user_: I meant not as an admin user but as a regular user. — undetected Selenium, Apr 10 '22 at 16:03

Selenium - Getting error page when trying to load site?

1 Answers1