Selenium: access denied

Question

I am trying to scrape some data from LV website with Selenium and keep getting 'Access Denied' screen once 'sign in' button clicked. I feel like there is a protection against this because all seems to be working fine when I do the same manually. Oddly, I need to click 'sign in' button twice to be able to sign in manually.

My code:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'chromedriver.exe')
driver.get('https://secure.louisvuitton.com/eng-gb/mylv')
WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, "//span[@class='ucm-wrapper']")))
driver.find_element_by_xpath("//button[@class='ucm-button ucm-button--default ucm-choice__yes']").click()
driver.find_element_by_id('loginloginForm').send_keys('xxx@xxx.com')
driver.find_element_by_id ('passwordloginForm').send_keys('xxxxxx')
driver.find_element_by_id('loginSubmit_').click()

Error:

You don't have permission to access "http://secure.louisvuitton.com/eng-gb/mylv;jsessionid=xxxxxxx.front61-prd?" on this server.

Is there a way to login with Selenium and bypass this?

sorry, for not being clear. i get 'access denied' after single click with Selenium. When do it manually nothing happens, if i click once and logs in after second click only. — Mudyla, Sep 19 '20 at 19:21
it seems the site detects that it's a bot. I 'm afraid it might not be possible so simply — Vishesh Mangla, Sep 19 '20 at 19:30
Maybe you should try adding lots of time.sleep in between to clone human like behaviour. Nowadays site detect how fast you type, how fast you scroll etc. — Vishesh Mangla, Sep 19 '20 at 19:31
i have already tried adding 2s between adding user, then 2s for pass and then 2s for submit but it didn't help. I get similar 'access denied' message when Selenium does .refresh() — Mudyla, Sep 19 '20 at 19:56

score 6 · Answer 1 · answered Sep 19 '20 at 20:23

I took your code added a few tweaks and ran the test as follows:

Code Block:

from selenium import webdriver
driver.get('https://secure.louisvuitton.com/eng-gb/mylv')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//span[text()='Accept and Continue']"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@id='loginloginForm']"))).send_keys("Mudyla@stackoverflow.com")
driver.find_element_by_xpath("//input[@id='passwordloginForm']").send_keys('Mudyla')
driver.find_element_by_xpath("//input[@id='loginSubmit_']").click()

Observation

Similar to your observation, I have hit the same roadblock with no results as follows:

Deep Dive

It seems the click() on Sign In does happens. But while inspecting the DOM Tree of the webpage you will find that some of the <script> tag refers to JavaScripts having keyword akam. As an example:

akam-sw.js install script version 1.3.3 "serviceWorker"in navigator&&"find"in[]&&function()
<script type="text/javascript" src="https://secure.louisvuitton.com/akam/11/7f0e2ae6" defer=""></script>
<noscript><img src="https://secure.louisvuitton.com/akam/11/pixel_7f0e2ae6?a=dD0xOWNjNTRjMmMxYzdmNmMwZjI0NTUwOGZmZDM5ZTQzMWQ5NjI5ZmIwJmpzPW9mZg==" style="visibility: hidden; position: absolute; left: -999px; top: -999px;" /></noscript>

Which is a clear indication that the website is protected by Bot Manager an advanced bot detection service provided by Akamai and the response gets blocked.

Bot Manager

As per the article Bot Manager - Foundations:

akamai_detection

Conclusion

So it can be concluded that the request for the data is detected as being performed by Selenium driven WebDriver instance and the response is blocked.

References

A couple of documentations:

tl; dr

A couple of relevant discussions:

1

Thanks for looking into it! – Mudyla Sep 19 '20 at 20:50

score 3 · Accepted Answer · answered Jan 17 '21 at 23:05

It's been a while since I had posted this question but if anyone is interested below are the steps I've taken to solve the problem.

Open chromedriver.exe in hex editor, find the string $cdc and replace with something else of the same length. Then save and run modified binary. Read more in this answer and the replies to it.
Selenium python code:

options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_argument('--disable-blink-features=AutomationControlled')
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path='chromedriver.exe')
driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) '
                                                                     'AppleWebKit/537.36 (KHTML, like Gecko) '
                                                                     'Chrome/85.0.4183.102 Safari/537.36'})

Could you extend on what does the script exactly do?, works but I'd like to understand why — nck, Feb 06 '21 at 15:29
I needed to periodically check whether one rare bag is in stock but the website had some sort of anti-bot/anti-selenium script which prevented me from doing that. The steps above allowed to overcome the problem. — Mudyla, Feb 06 '21 at 15:37
@Mudyla Hi looks like you know your stuff. If you are interested in working with me on a good and small project, please email me: crew.gladiator@gmail.com — teller.py3, Jun 04 '21 at 18:16

score 1 · Answer 3 · edited May 08 '21 at 19:52

1

For me it worked when I added the following line just after launching a driver:

 driver.manage().deleteAllCookies();

edited May 08 '21 at 19:52

C. Peck

3,641
3
19
36

answered May 08 '21 at 08:48

Prajakta

11
1