1

I want to scrap some data with selenium python. I have this type of screen sometimes :

enter image description here

Do you know to proceed in order to remove this type of verification ? Here my code :

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

options = Options()
options.add_argument("start-maximized")
options.add_argument("--disable-web-security")
options.add_argument("--disable-site-isolation-trials")
options.add_argument("--allow-running-insecure-content")


driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)

driver.get('THE_WEBSITE_COM')
Alan CUZON
  • 35
  • 4
  • 1
    Does this answer your question? [Selenium webdriver: Modifying navigator.webdriver flag to prevent selenium detection](https://stackoverflow.com/questions/53039551/selenium-webdriver-modifying-navigator-webdriver-flag-to-prevent-selenium-detec) – nordmanden Aug 29 '22 at 13:41
  • That's perfect ! It's avoid the security message, thanks a lot for the tip ! – Alan CUZON Aug 29 '22 at 14:28

2 Answers2

3

Selenium specifically and other automation tools have certain user agents and other identifiers which indicate that it's automated. So maybe have a play around with things like that. Some websites use anti bot tools to analyze browsing behaviours and patterns so try to slow it and randomize it eg. random time between page requests

Another trick is to look at the website and try to find if there are any alternative routes to get the information. For example: is there a public API you can use to bypass it? Is there a mobile version of the website? Sometimes mobile versions have less aggressive Captcha enforcement.

thenullptr
  • 381
  • 2
  • 7
0

What I do most of the time is to launch the browser separately and connect to it using Dev port which is beautifully explained in this article.

To enable Chrome to open a port for remote debugging, we need to launch it with a custom flag –

chrome.exe --remote-debugging-port=9222 --user-data-dir="C:\selenum\ChromeProfile"

Then connect to the browser using this

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

chrome_options = Options()
chrome_options.add_experimental_option("debuggerAddress", "127.0.0.1:9222")
#Change chrome driver path accordingly
chrome_driver = "C:\chromedriver.exe"
driver = webdriver.Chrome(chrome_driver, chrome_options=chrome_options)
print driver.title
Muhammad Hassan
  • 103
  • 2
  • 8