Getting 403 when using Selenium to automate checkout process

Question

I am trying to create a script using python and selenium to automate the checkout process at bestbuy.ca.

I get all the way to the final stage where you click to review the final order, but get the following 403 forbidden message (as seen in the network response) when I try to click through to the final step.

Is there something server side that has detected that I am using selenium and preventing me to proceed?

How can I hide the fact that it is selenium being used?

These are the options I am using for selenium:

options = Options()
options.add_argument('--disable-blink-features=AutomationControlled')
options.add_argument("start-maximized")
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(options=options)

I currently have 10 second delays after each action(ie open page, wait, click add to cart, wait, click checkout, wait)

I have implemented a random useragent to be used on each run:

import fake_useragent
ua = UserAgent()
userAgent = ua.random
options.add_argument(f'user-agent={userAgent}')

I have also modified my chromedriver binary as per the comments in THIS THREAD

Error seen when proceeding to order review page:

Best guess is this actually due to timing issues - sending in selenium actions faster than the page can react to them. Try inserting some delays. — David K. Hess, Apr 07 '21 at 16:09
@DavidK.Hess I currently have 10 second delays after each action (ie open page, wait, click add to cart, wait, click checkout, wait) — Bajan, Apr 07 '21 at 17:18
@KunduK I have also tested with fake_useragent : ua = UserAgent() userAgent = ua.random options.add_argument(f'user-agent={userAgent}') — Bajan, Apr 07 '21 at 17:23
Are you logged-in to bestbuy.ca or processing the order as a guest? — Life is complex, Apr 12 '21 at 03:06
It seems strange to have an `http://` request on any major retailers' site, especially in payment flow. How is that URL getting generated, is it in your script or theirs? — Matt E., Apr 12 '21 at 21:24

score 10 · Accepted Answer · answered Apr 15 '21 at 16:49

After much testing the last few days, here are the options that have allowed me to bypass the restrictions I was facing.

Modified cdc_ string in my chromedriver

Chromedriver options:

 options.add_argument('--disable-blink-features=AutomationControlled')
 options.add_argument("--disable-extensions")
 options.add_experimental_option('useAutomationExtension', False)
 options.add_experimental_option("excludeSwitches", ["enable-automation"])
 chrome_driver = webdriver.Chrome(options=options)

Change the property value of the navigator for webdriver to undefined:

chrome_driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")

After all three of these were implemented I no longer faced any 403 error when navigating the site and the cart/checkout process.

Thanks for the question and answer. I had been facing a similar issue for some time. The "--disable-blink-features=AutomationControlled" did the trick for me. Thank you! — BigRed118, Aug 05 '22 at 16:33

score 1 · Answer 2 · answered Apr 13 '21 at 06:53

In my case, either using code to control the browser, or simply starting Chrome through python and manually using the browser always leads to the 403 error, even just adding a product to the cart.

As you said, I think that this site someway knows that the user is using Selenium or some sort of automation tool and the server is blocking API requests.

Searching in stackoverflow I found this https://stackoverflow.com/a/52108199/3228768 but editing the chromedriver results anyway in a failure.

The only way I completed the flow is settings this options:

u = 'https://www.bestbuy.ca/en-ca/category/appliances/26517'
# relevant part start here
options = webdriver.ChromeOptions()
options.add_argument("--disable-blink-features")
options.add_argument("--disable-blink-features=AutomationControlled")
# relevant part ends here 
driver = webdriver.Chrome(executable_path=r"chromedriver.exe", options=options)
driver.maximize_window()
driver.get(u)

In this way I managed to add a product to the cart. I think you could use it to proceed the flow until checkout. Let me know.

Hey thanks for the reply - Yes, that option was indeed what I had used in my original post to enable me to add it to the cart (as well as the modified chrome driver). My issue unfortunately lies when browsing to to the final checkout process page to order. — Bajan, Apr 13 '21 at 23:56
Really sorry, I was so happy that my tests were (at firts) successfull I forgot to check if you already had implemented my solution. Glad to see you resolved it anyway. — Mattia Galati, Apr 19 '21 at 08:11

score 0 · Answer 3 · answered Apr 13 '21 at 10:42

0

Try this one: https://github.com/ultrafunkamsterdam/undetected-chromedriver

It avoids the selenium detection quite well, I've been having good result with it so far. Headless is not guaranteed though.

import undetected_chromedriver as uc
driver = uc.Chrome()
driver.get('https://www.bestbuy.ca/')

answered Apr 13 '21 at 10:42

jackblk

1,076
8
19

Thanks for the reply, I might give this a shot if all else fails! – Bajan Apr 13 '21 at 23:57

Getting 403 when using Selenium to automate checkout process

3 Answers3

Linked