105

I'm trying to automate a very basic task in a website using selenium and chrome but somehow the website detects when chrome is driven by selenium and blocks every request. I suspect that the website is relying on an exposed DOM variable like this one https://stackoverflow.com/a/41904453/648236 to detect selenium driven browser.

My question is, is there a way I can make the navigator.webdriver flag false? I am willing to go so far as to try and recompile the selenium source after making modifications, but I cannot seem to find the NavigatorAutomationInformation source anywhere in the repository https://github.com/SeleniumHQ/selenium

Any help is much appreciated

P.S: I also tried the following from https://w3c.github.io/webdriver/#interface

Object.defineProperty(navigator, 'webdriver', {
    get: () => false,
  });

But it only updates the property after the initial page load. I think the site detects the variable before my script is executed.

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
Ajanth
  • 2,435
  • 3
  • 20
  • 23

15 Answers15

151

First the update 1

execute_cdp_cmd(): With the availability of execute_cdp_cmd(cmd, cmd_args) command now you can easily execute commands using Selenium. Using this feature you can modify the navigator.webdriver easily to prevent Selenium from getting detected.


Preventing Detection 2

To prevent Selenium driven WebDriver getting detected a niche approach would include either / all of the below mentioned steps:

  • Adding the argument --disable-blink-features=AutomationControlled

    from selenium import webdriver
    
    options = webdriver.ChromeOptions() 
    options.add_argument('--disable-blink-features=AutomationControlled')
    driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
    driver.get("https://www.website.com")
    

You can find a relevant detailed discussion in Selenium can't open a second page

  • Rotating the through execute_cdp_cmd() command as follows:

    #Setting up Chrome/83.0.4103.53 as useragent
    driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
    
  • Change the property value of the navigator for webdriver to undefined

    driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
    
  • Exclude the collection of enable-automation switches

    options.add_experimental_option("excludeSwitches", ["enable-automation"])
    
  • Turn-off useAutomationExtension

    options.add_experimental_option('useAutomationExtension', False)
    

Sample Code 3

Clubbing up all the steps mentioned above and effective code block will be:

from selenium import webdriver

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
print(driver.execute_script("return navigator.userAgent;"))
driver.get('https://www.httpbin.org/headers')

History

As per the W3C Editor's Draft the current implementation strictly mentions:

The webdriver-active flag is set to true when the user agent is under remote control which is initially set to false.

Further,

Navigator includes NavigatorAutomationInformation;

It is to be noted that:

The NavigatorAutomationInformation interface should not be exposed on WorkerNavigator.

The NavigatorAutomationInformation interface is defined as:

interface mixin NavigatorAutomationInformation {
    readonly attribute boolean webdriver;
};

which returns true if webdriver-active flag is set, false otherwise.

Finally, the navigator.webdriver defines a standard way for co-operating user agents to inform the document that it is controlled by WebDriver, so that alternate code paths can be triggered during automation.

Caution: Altering/tweaking the above mentioned parameters may block the navigation and get the WebDriver instance detected.


Update (6-Nov-2019)

As of the current implementation an ideal way to access a web page without getting detected would be to use the ChromeOptions() class to add a couple of arguments to:

  • Exclude the collection of enable-automation switches
  • Turn-off useAutomationExtension

through an instance of ChromeOptions as follows:

  • Java Example:

    System.setProperty("webdriver.chrome.driver", "C:\\Utility\\BrowserDrivers\\chromedriver.exe");
    ChromeOptions options = new ChromeOptions();
    options.setExperimentalOption("excludeSwitches", Collections.singletonList("enable-automation"));
    options.setExperimentalOption("useAutomationExtension", false);
    WebDriver driver =  new ChromeDriver(options);
    driver.get("https://www.google.com/");
    
  • Python Example

    from selenium import webdriver
    
    options = webdriver.ChromeOptions()
    options.add_experimental_option("excludeSwitches", ["enable-automation"])
    options.add_experimental_option('useAutomationExtension', False)
    driver = webdriver.Chrome(options=options, executable_path=r'C:\path\to\chromedriver.exe')
    driver.get("https://www.google.com/")
    
  • Ruby Example

      options = Selenium::WebDriver::Chrome::Options.new
      options.add_argument("--disable-blink-features=AutomationControlled")
      driver = Selenium::WebDriver.for :chrome, options: options
    

Legends

1: Applies to Selenium's Python clients only.

2: Applies to Selenium's Python clients only.

3: Applies to Selenium's Python clients only.

John Smith
  • 7,243
  • 6
  • 49
  • 61
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
  • 5
    so how do we do this? Iam sorry i dint get the point. Iam using python client. – Ja8zyjits Apr 11 '19 at 10:39
  • 1
    @DebanjanB Thank you for a great answer! Can you clarify what is the best approach for today? Should I use the code from `Update (6-Nov-2019)` or I must add three lines of `execute_cdp_cmd` from `Sample Code 3`? – F. Vosnim May 19 '20 at 20:23
  • 1
    @F.Vosnim Go incremental. In normal circumstances _Update (6-Nov-2019)_ should suffice. If it doesn't implement the `execute_cdp_cmd()`. – undetected Selenium May 19 '20 at 20:26
  • 4
    @DebanjanB Should I also use `--disable-blink-features` and `--disable-blink-features=AutomationControlled`? – F. Vosnim May 19 '20 at 20:30
  • 1
    @F.Vosnim I still won't recommend `--disable-blink-features` as a lot of Chrome features are controlled through that flag. I am yet to conclude on my experiments with those flags. However, those flags may help you to avoid getting detected but possibly Chrome won't be behaving like normal Chrome anymore. Call is with you. – undetected Selenium May 19 '20 at 20:35
  • 3
    `navigator.webdriver` is still `true` when it comes to open a new page – jhd Jul 03 '20 at 08:03
  • Hello, It work only on Python client ? right ? can I prevent detection on Java client ? – ndhcoder Aug 10 '20 at 07:08
  • 4
    This is not working anymore. They patched the switches "enable-automation" and "useAutomationExtension" and they are now behaving correctly, showing navigator.wedriver as true. Furthermore Im failry sure that this part never worked: driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})") – aydunno Aug 24 '20 at 19:41
  • still getting the reCaptcha prompt every other time, using `chrome=84.0.4147.135`, `selenium==3.141.0` – RichieV Aug 27 '20 at 02:12
  • It seems useAutomationExtension can not be set false anymore. I am getting a warning: "[WARNING]: Deprecated chrome option is ignored: useAutomationExtension" – Spero Sep 25 '20 at 13:39
  • @Spero Yeap, that would give rise to a warning and we can live with it for the time being. But definitely we need to explore some other loopholes now :) – undetected Selenium Sep 25 '20 at 13:51
  • 4
    On January 21, none of these techniques work anymore. I have the feeling everything stopped working after a chrome update. – Thomas AMET Jan 22 '21 at 22:22
  • 2
    Does anyone have an equivalent for Firefox >= 88? profile.set_preferences("dom.webdriver.enabled", False) no longer works. – heyyouyayou Apr 29 '21 at 16:02
  • @DebanjanB could you help me do the sameon webdriverio. I tried adding args: ['enable-automation', '--disable-blink-features=AutomationControlled', '--disable-blink-features'] and didn't help. – ChanChow Oct 01 '21 at 04:26
  • Note that `navigator.webdriver` is either `true` of `false` by the webdriver spec, it should never be `undefined` as suggested since antibots can use it to detect spoofed browser. Also the property overriding is vulnerable to `toString()` check. – hldev Oct 11 '21 at 21:54
  • 1
    Still working (for me at least) – farhan jatt Dec 27 '21 at 06:10
  • @farhanjatt What version of Chrome/Selenium are you using? – Mecanik Jan 12 '22 at 07:35
  • I dont know the exact version of selenium but I am using latest as I updated it and then started using it. Chromes version is 97.0.4692.71 – farhan jatt Jan 12 '22 at 07:45
77

ChromeDriver:

Finally discovered the simple solution for this with a simple flag! :)

--disable-blink-features=AutomationControlled

navigator.webdriver=true will no longer show up with that flag set.

For a list of things you can disable, check them out here

010011100101
  • 1,630
  • 1
  • 10
  • 20
  • 5
    OMG!! It worked for me in chromedriver 83. I tried a lot of solutions but this was the only one that worked perfectly. I added with python using chromeOptions.add_argument. Thanks – rodrigorf Jun 21 '20 at 22:58
  • Can I apply on Java Client ? I read above comment, it only work for python client ? – ndhcoder Aug 10 '20 at 07:10
  • 2
    Absolutely brilliant! ... for anyone using symfony/panther with chrome you can achieve this by setting the PANTHER_CHROME_ARGUMENTS="--disable-dev-shm-usage --disable-blink-features=AutomationControlled --window-size=1920x1080" – Gabriel Spiteri Aug 12 '20 at 08:30
  • 12
    Thanks so much!! The code in python: `options.add_argument("--disable-blink-features=AutomationControlled")` – Pablo Vilas Sep 12 '20 at 10:47
  • This is not working for me in python selenium, but the chosen answer does. – Stoopkid Sep 19 '20 at 21:07
  • 1
    Hi can anyone send me exact line of code to avoid the detection? Thanks – Abdul Haseeb Oct 01 '20 at 11:14
  • 3
    worked very well. but now headless doesn't work. Any help ? – Himanshu Mange Oct 29 '20 at 00:32
  • 1
    I spent about a day trying to get the accepted answer to work, even went to the lengths of setting up Java/eclipse rather than C# I'm familiar with - still didn't work then tried this and worked instantly. – wlf Dec 25 '20 at 15:08
  • It does not work if you make browser headless. – ibrahim Apr 24 '21 at 13:45
  • @ibrahim Did you check if navigator.webdriver was being set to true in the headless browser? Whatever site your using, it could possibly be detecting automation through an alternative method that doesn't have to do with navigator.webdriver. Just a thought... Goodluck – 010011100101 Apr 26 '21 at 06:19
  • @010011100101 I did, but I found that I need to add driver_.execute_cdp_cmd('Network.setUserAgentOverride', { "userAgent": user_agent }) where use_agent need to be selected randomly – ibrahim Apr 29 '21 at 12:21
31

Do not use cdp command to change webdriver value as it will lead to inconsistency which later can be used to detect webdriver. Use the below code, this will remove any traces of webdriver.

options.add_argument("--disable-blink-features")
options.add_argument("--disable-blink-features=AutomationControlled")
David Buck
  • 3,752
  • 35
  • 31
  • 35
Baki Billah
  • 370
  • 4
  • 6
  • 1
    I use Chrome 88.0.4324.150 , is work fine. thanks. – huagang Feb 08 '21 at 08:29
  • @Baki Billah could you help me do the sameon webdriverio. I tried adding args: ['enable-automation', '--disable-blink-features=AutomationControlled', '--disable-blink-features'] and didn't help. – ChanChow Oct 01 '21 at 04:27
  • Chrome 116.0.5845.9600, work fine – Jim Aug 24 '23 at 11:20
24

Before (in browser console window):

> navigator.webdriver
true

Change (in selenium):

// C#
var options = new ChromeOptions();
options.AddExcludedArguments(new List<string>() { "enable-automation" });

// Python
options.add_experimental_option("excludeSwitches", ['enable-automation'])

After (in browser console window):

> navigator.webdriver
undefined

This will not work for version ChromeDriver 79.0.3945.16 and above. See the release notes here

CodeIt
  • 3,492
  • 3
  • 26
  • 37
petke
  • 1,345
  • 10
  • 25
  • 4
    How do to the same in Firefox? https://stackoverflow.com/questions/57122151/exclude-switches-in-firefox-webdriver-options – yegor256 Aug 23 '19 at 05:25
  • 1
    I can also confirm this--to my eye--no longer works. Doesn't successfully change `navigator.webdriver`. Would love alternatives. – Rishi Oct 31 '19 at 18:48
  • Worked for me, the variable has been erased. – XavierAM Nov 01 '19 at 11:11
  • Worked for me too. Chrome 78.0.3904.97, with chromedriver 78.0.3904.10500. – deerchao Nov 21 '19 at 03:34
  • 1
    The C# version of this worked for me. driver 85.0.4183.87 – Roro Sep 25 '20 at 15:48
  • @petke: working for Chrome 97.0.4692.71. with sel -- console code -- "navigator.webdriver" "*false*". w sel (chromedriver instantiated with experimental options you've suggested -- console code -- "navigator.webdriver" "*false*" – JB-007 Jan 14 '22 at 23:24
18

To exclude the collection of enable-automation switches as mentioned in the 6-Nov-2019 update of the top voted answer doesn't work anymore as of April 2020. Instead I was getting the following error:

ERROR:broker_win.cc(55)] Error reading broker pipe: The pipe has been ended. (0x6D)

Here's what's working as of 6th April 2020 with Chrome 80.

Before (in the Chrome console window):

> navigator.webdriver
true

Python example:

options = webdriver.ChromeOptions()
options.add_argument("--disable-blink-features")
options.add_argument("--disable-blink-features=AutomationControlled")

After (in the Chrome console window):

> navigator.webdriver
undefined
Rick
  • 305
  • 2
  • 5
16

Nowadays you can accomplish this with cdp command:

driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
  "source": """
    Object.defineProperty(navigator, 'webdriver', {
      get: () => undefined
    })
  """
})

driver.get(some_url)

by the way, you want to return undefined, false is a dead giveaway.

pguardiario
  • 53,827
  • 19
  • 119
  • 159
  • 3
    nowadays (i.e. Chrome 97.0.4692.71) it's actually *false*. Can't comment for whether false/undefined for Chrome back in '19 though surprised if it's changed since (?) – JB-007 Jan 14 '22 at 23:28
9

Since this question is related to selenium a cross-browser solution to overriding navigator.webdriver is useful. This could be done by patching browser environment before any JS of target page runs, but unfortunately no other browsers except chromium allows one to evaluate arbitrary JavaScript code after document load and before any other JS runs (firefox is close with Remote Protocol).

Before patching we needed to check how the default browser environment looks like. Before changing a property we can see it's default definition with Object.getOwnPropertyDescriptor()

Object.getOwnPropertyDescriptor(navigator, 'webdriver');
// undefined

So with this quick test we can see webdriver property is not defined in navigator. It's actually defined in Navigator.prototype:

Object.getOwnPropertyDescriptor(Navigator.prototype, 'webdriver');
// {set: undefined, enumerable: true, configurable: true, get: ƒ}

It's highly important to change the property on the object that owns it, otherwise the following can happen:

navigator.webdriver; // true if webdriver controlled, false otherwise
// this lazy patch is commonly found on the internet, it does not even set the right value
Object.defineProperty(navigator, 'webdriver', {
  get: () => undefined
});
navigator.webdriver; // undefined
Object.getOwnPropertyDescriptor(Navigator.prototype, 'webdriver').get.apply(navigator);
// true

A less naive patch would first target the right object and use right property definition, but digging deeper we can find more inconsistences:

const defaultGetter = Object.getOwnPropertyDescriptor(Navigator.prototype, 'webdriver').get;
defaultGetter.toString();
// "function get webdriver() { [native code] }"
Object.defineProperty(Navigator.prototype, 'webdriver', {
  set: undefined,
  enumerable: true,
  configurable: true,
  get: () => false
});
const patchedGetter = Object.getOwnPropertyDescriptor(Navigator.prototype, 'webdriver').get;
patchedGetter.toString();
// "() => false"

A perfect patch leaves no traces, instead of replacing getter function it would be good if we could just intercept the call to it and change the returned value. JavaScript has native support for that throught Proxy apply handler:

const defaultGetter = Object.getOwnPropertyDescriptor(Navigator.prototype, 'webdriver').get;
defaultGetter.apply(navigator); // true
defaultGetter.toString();
// "function get webdriver() { [native code] }"
Object.defineProperty(Navigator.prototype, 'webdriver', {
  set: undefined,
  enumerable: true,
  configurable: true,
  get: new Proxy(defaultGetter, { apply: (target, thisArg, args) => {
    // emulate getter call validation
    Reflect.apply(target, thisArg, args);
    return false;
  }})
});
const patchedGetter = Object.getOwnPropertyDescriptor(Navigator.prototype, 'webdriver').get;
patchedGetter.apply(navigator); // false
patchedGetter.toString();
// "function () { [native code] }"

The only inconsistence now is in the function name, unfortunately there is no way to override the function name shown in native toString() representation. But even so it can pass generic regular expressions that searches for spoofed browser native functions by looking for { [native code] } at the end of its string representation. To remove this inconsistence you can patch Function.prototype.toString and make it return valid native string representations for all native functions you patched.

To sum up, in selenium it could be applied with:

chrome.execute_cdp_cmd('Page.addScriptToEvaluateOnNewDocument', {'source': """
    Object.defineProperty(Navigator.prototype, 'webdriver', {
        set: undefined,
        enumerable: true,
        configurable: true,
        get: new Proxy(
            Object.getOwnPropertyDescriptor(Navigator.prototype, 'webdriver').get,
            { apply: (target, thisArg, args) => {
                // emulate getter call validation
                Reflect.apply(target, thisArg, args);
                return false;
            }}
        )
    });
"""})

The playwright project maintains a fork of Firefox and WebKit to add features for browser automation, one of them is equivalent to Page.addScriptToEvaluateOnNewDocument, but there is no implementation for Python of the communication protocol but it could be implemented from scratch.

hldev
  • 914
  • 8
  • 18
  • Do you know where can I get this information for Python, ChromeRemotedriver, Selenium4? – The Dan Oct 27 '21 at 18:30
  • @TheDan the code above is for python, in selenium 4 you need to get the CDP session and call `execute_cdp_cmd`. You should ask a new question for that. – hldev Oct 28 '21 at 21:50
  • Appreciate the in depth answer. [Page.add_init_script](https://playwright.dev/python/docs/api/class-page#page-add-init-script) seems well suited for playwright-python – Mattwmaster58 Dec 09 '22 at 03:43
8

Finally this solved the problem for ChromeDriver, Chrome greater than v79.

ChromeOptions options = new ChromeOptions();
options.addArguments("--disable-blink-features");
options.addArguments("--disable-blink-features=AutomationControlled");
ChromeDriver driver = new ChromeDriver(options);
Map<String, Object> params = new HashMap<String, Object>();
params.put("source", "Object.defineProperty(navigator, 'webdriver', { get: () => undefined })");
driver.executeCdpCommand("Page.addScriptToEvaluateOnNewDocument", params);
Elias Vargas
  • 1,126
  • 1
  • 18
  • 32
6

Simple hack for python:

options = webdriver.ChromeOptions()    
options.add_argument("--disable-blink-features=AutomationControlled")
Gil Baggio
  • 13,019
  • 3
  • 48
  • 37
3

As mentioned in the above comment - https://stackoverflow.com/a/60403652/2923098 the following option totally worked for me (in Java)-

ChromeOptions options = new ChromeOptions();
options.addArguments("--incognito", "--disable-blink-features=AutomationControlled");
Xtraterrestrial
  • 651
  • 1
  • 6
  • 12
3

Python

I tried most of the stuff mentioned in this post and i was still facing issues. What saved me for now is https://pypi.org/project/undetected-chromedriver

pip install undetected-chromedriver


import undetected_chromedriver.v2 as uc
from time import sleep
from random import randint


driver = uc.Chrome()
driver.get('www.your_url.here')
driver.maximize_window() 

sleep(randint(3,9))

A bit slow but i will take slow over non working.

I guess if every interested could go over the source code and see what provides the win there.

Yev Guyduy
  • 1,371
  • 12
  • 13
  • 1
    Thanks. I also tried much of the solutions up there and didn't work. Now with undetected-chromedriver, it works. – kitokid May 19 '22 at 05:08
2

I would like to add a Java alternative to the cdp command method mentioned by pguardiario

Map<String, Object> params = new HashMap<String, Object>();
params.put("source", "Object.defineProperty(navigator, 'webdriver', { get: () => undefined })");
driver.executeCdpCommand("Page.addScriptToEvaluateOnNewDocument", params);

In order for this to work you need to use the ChromiumDriver from the org.openqa.selenium.chromium.ChromiumDriver package. From what I can tell that package is not included in Selenium 3.141.59 so I used the Selenium 4 alpha.

Also, the excludeSwitches/useAutomationExtension experimental options do not seem to work for me anymore with ChromeDriver 79 and Chrome 79.

DUDSS
  • 110
  • 1
  • 9
2

For those of you who've tried these tricks, please make sure to also check that the user-agent that you are using is the user agent that corresponds to the platform (mobile / desktop / tablet) your crawler is meant to emulate. It took me a while to realize that was my Achilles heel ;)

Albion
  • 135
  • 5
0

If you use a Remote Webdriver , the code below will set navigator.webdriver to undefined.

work for ChromeDriver 81.0.4044.122

Python example:

    options = webdriver.ChromeOptions()
    # options.add_argument("--headless")
    options.add_argument('--disable-gpu')
    options.add_argument('--no-sandbox')
    driver = webdriver.Remote(
       'localhost:9515', desired_capabilities=options.to_capabilities())
    script = '''
    Object.defineProperty(navigator, 'webdriver', {
        get: () => undefined
    })
    '''
    driver.execute_script(script)
fcce
  • 1,034
  • 1
  • 12
  • 24
0

Use --disable-blink-features=AutomationControlled to disable navigator.webdriver

Wesley
  • 855
  • 1
  • 9
  • 23