1

Im trying to scrape Amazon when running headless with a up to date UserAgent I am getting rate limited. When I # out the headless line I don't get detected or rate limited. Below is my code!

options = Options()
options.add_experimental_option("excludeSwitches", ["enable-logging"])
options.add_experimental_option("useAutomationExtension", False)
#options.add_argument("--headless")
options.add_argument("--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36")
service = ChromeService(executable_path=ChromeDriverManager().install())
browser = webdriver.Chrome(service=service, options=options)
browser.get("https://www.amazon.com.au/Oculus-Quest-2-Virtual-Reality-Headset/dp/B08FSZ8QWH")
print(browser.page_source)

When I remove the # at options.add_argument("--headless") and I run the code I get a server busy line. Does anyone know of a fix?

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
HK Gaming
  • 15
  • 1
  • 5

1 Answers1

1

To avoid detection using add the following argument through add_argument() as follows:

--disable-blink-features=AutomationControlled

Sample Code:

options = Options()
options.headless = True
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('excludeSwitches', ['enable-logging'])
options.add_experimental_option('useAutomationExtension', False)
options.add_argument('--disable-blink-features=AutomationControlled')
s = Service('C:\\BrowserDrivers\\chromedriver.exe')
driver = webdriver.Chrome(service=s, options=options)
driver.get("https://www.amazon.com.au/Oculus-Quest-2-Virtual-Reality-Headset/dp/B08FSZ8QWH")
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
  • 1
    If I put it in options.add_experimental_option() I get an error `TypeError: ChromiumOptions.add_experimental_option() missing 1 required positional argument: 'value'` Do I just need to keep it in `options.add_argument`? – HK Gaming Jan 14 '22 at 10:50
  • @HKGaming My bad, typo in the verbatim. Updated it, should have been `add_argument()` – undetected Selenium Jan 14 '22 at 11:17
  • 1
    Perfect seems to be working at the moment, thank you very much! – HK Gaming Jan 14 '22 at 11:30