How to use authenticated proxy in selenium chromedriver?

Question

After searching for many hours I am starting to think this is impossible.

I need to run Chrome through selenium using different authenticated (not public) proxy's for each run.

PROXY_IP = "<some IP address>"
UID = "<the user id>"
PWD = "<the password">

options = webdriver.ChromeOptions()
options.add_argument("--proxy-server=%s:%s@%s" % (UID,PWD,PROXY_IP))

driver = webdriver.Chrome(executable_path=".\\driver\\chromedriver.exe",
                          chrome_options=options)
driver.get("<site URL>")

Chrome will fire-up and display the error:

This webpage is not available
ERR_NO_SUPPORTED_PROXIES

If I use a public proxy requiring no authentication like this...

PROXY_IP = "<public proxy IP address>"

options = webdriver.ChromeOptions()
options.add_argument("--proxy-server=%s" % PROXY_IP)

driver = webdriver.Chrome(executable_path=".\\driver\\chromedriver.exe",
                          chrome_options=options)
driver.get("<site URL>")

...it runs just fine and displays the site while using the proxy.

I also tried a variant with http:// in front of the user ID:

options.add_argument("--proxy-server=http://%s:%s@%s" % (UID,PWD,PROXY_IP))

The fact that I have searched far and wide and haven't found a solution leads me to believe none might exist.

I did find this but I can't make sense out of it:

selenium chromedriver authentication proxy

Not sure what browswermob-proxy is or is supposed to do or how to implement and test in Python. I hate piling up band-aid solutions unless they are absolutely necessary.

EDIT (08NOV21):

I have been away from using Selenium for many years. Because of this I now lack the context (and time, sorry) to go through the newer answers being provided and mark one as the solution to this problem. Does SO have a mechanism one could use to effectively delegate this function to someone who might be a current practitioner with expertise in this domain?

I've just submitted a bug report. https://bugs.chromium.org/p/chromedriver/issues/detail?id=1605 — jcomeau_ictx, Nov 19 '16 at 01:16

Blackster · Answer 1 · 2019-05-23T13:59:16.717

24

To use proxies with auth in python selenium you can use seleniumwire.

Fistly, install it with pip install selenium-wire

Then import webdriver from seleniumwire instead selenium

from seleniumwire import webdriver
options = {
    'proxy': {
        'http': 'http://username:password@host:port', 
        'https': 'https://username:password@host:port',
        'no_proxy': 'localhost,127.0.0.1' # excludes
    }
}
browser = webdriver.Chrome(path_to_driver, seleniumwire_options=options)

Now you can use your browser instance exact the same way as selenium: browser.get('https://api.ipify.org') and so on...

edited May 23 '19 at 13:59

answered May 23 '19 at 13:52

Blackster

343
2
5

Can i update proxy option later for same webdriver instance? – Lorem Ipsum Nov 14 '19 at 13:39
@TaimurSaeed, sorry but actually I do not know. Need to test it – Blackster Nov 17 '19 at 01:14
3

@Blackster I have tested it, and you can't change the configuration once an instance is created. The work around is to create a new driver instance with different configuration – Lorem Ipsum Nov 18 '19 at 07:30

score 4 · Answer 2 · answered Feb 21 '17 at 09:15

4

I have checked for most of the solutions on the web and for none of them authentication via chrome/firefox desired capabilities is working. Check this link: https://github.com/webdriverio/webdriverio/issues/324. Finally the temporary solution is to whitelist your IP address with the proxy provider.

answered Feb 21 '17 at 09:15

Anurag Narra

95
1
9

2

whitelisting it not a solution if username contains parameters – Toolkit Mar 04 '17 at 18:44

score 3 · Answer 3 · answered Feb 23 '19 at 08:32

I could not find any solution for chrome. We can not add extensions with headless option. I am using Heroku with chrome-buildpack. There are following options

Use xvfb instead of headless options and install extension
Use local proxy forwarder that forwards traffic to authenticated proxy; we can use Squid, mitProxy, or something like proxy-login-automator

Instead of these workaround I switched to Firefox where i was able to fill Username and Password on Proxy authentication Pop-up. Like given below. Following code is for Ruby using Capybara. You should be able to do something like this on your platform


page.driver.browser.switch_to.alert.send_keys('proxy-username' + Selenium::WebDriver::Keys::KEYS[:tab] + 'my-password')
page.driver.browser.switch_to.alert.accept

score 3 · Answer 4 · answered Jul 31 '19 at 05:00

3

This is the best solution I found and is the ONLY one that worked - all other answers on this question are outdated. It basically generates an auth extension for Chrome on the fly. Simply use the function as defined in the script as follows:

driver = proxy_chrome(host, port, username, password)
driver.get("http://www.icanhazip.com/")
driver.get("https://www.reddit.com/")
print('Terminated without issues.')

Note that this doesn't work with the --headless option. However, on Linux, you can simply use the x virtual frame buffer to simulate that. It's as easy as this in python:

import xvfbwrapper
x = xvfbwrapper.Xvfb()
x.start()

answered Jul 31 '19 at 05:00

Muppet

5,767
6
29
39

Simple example on how to use xvfbwrapper: https://gist.github.com/cgoldberg/4151516 – Spherical Cowboy Sep 21 '21 at 03:19
Also make sure to create an empty directory named 'extension' in case you use the proxy.py file linked to above, this is where the plugin file proxy_auth_plugin.zip will be stored. Moreover, host, username and password are strings, but port needs to be passed as an int. – Spherical Cowboy Sep 21 '21 at 03:22

OMERINB · Answer 5 · 2018-08-30T08:38:53.203

after trying many solutions that didn't actually work properly, i finally managed to set the authenticated proxy using the suggested extension from previous answers. what you need to do is to enter this link:

http://crxextractor.com/ and paste this url: https://www.crx4chrome.com/crx/1446/

It will let you download the extention as a .crx file without installing it. than i used this code:

proxy = {'address': 'pr.oxylabs.io:7777',
     'username': 'USERNAME',
     'password': 'PASSWORD'}

capabilities = dict(DesiredCapabilities.CHROME)
capabilities['proxy'] = {'proxyType': 'MANUAL',
                         'httpProxy': proxy['address'],
                         'ftpProxy': proxy['address'],
                         'sslProxy': proxy['address'],
                         'noProxy': '',
                         'class': "org.openqa.selenium.Proxy",
                         'autodetect': False,
                         'socksUsername': proxy['username'],
                         'socksPassword': proxy['password']}

options = webdriver.ChromeOptions()
options.add_extension("./extension_2_0_0_0.crx")
driver = webdriver.Chrome(executable_path=CHROME_PATH, desired_capabilities=capabilities, chrome_options=options)

This solution requires manual entry of the credentials the first time it's run. — Nikolay Shindarov, Oct 03 '19 at 20:29

score 2 · Answer 6 · edited Jul 08 '23 at 16:45

2

If you have permanent IP address (e.g. by leasing a machine in the cloud) you can contact your proxy provider to access the proxy server with your IP instead of the user and password. In my experience that's easier.

If that's not possible, the other solution is:

pip install undetected-chromedriver
pip install selenium-wire

Then set up proxy credentials using the 'proxy' key of a dictionary passed seleniumwire_options argument:

from seleniumwire.undetected_chromedriver.v2 import Chrome

options = {
    'proxy': {
        'http': 'http://user:pass@ip:port',
        'https': 'https://user:pass@ip:port',
        'no_proxy': 'localhost,127.0.0.1'
    }
}

driver = Chrome(seleniumwire_options=options)

edited Jul 08 '23 at 16:45

mirekphd

4,799
3
38
59

answered Nov 29 '21 at 22:20

V-cash

330
3
14

Thanks for the reply. Please read my edit with regards to no-longer being in a position to evaluate answers. Still, I hope your answer and those of others might help those with similar issues. It might very well help me in the future if I ever come back to this domain. – martin's Dec 01 '21 at 02:40
That's why I did it. – V-cash Dec 02 '21 at 20:47
1

how can I pass username and password in second method you told without seleniumwire – Inzamam Malik May 11 '23 at 21:05

score 0 · Answer 7 · answered May 26 '15 at 18:41

0

I tryed lot of time to do the same.

Chrome is using only proxy of the OS where it is installed. You can check it by going to options-> find: proxy -> change proxy settings

So without additional addons and configuring this addons you cannot do this.

Or you can change your OS proxy settings -- this is much easier.

Also you can use phantomjs -- it have the same engine(WebKit) as chrome. using something like this:

String PROXY = proxyIP + ":" + proxyPort;
String proxyAuth= proxyUser + ":" + proxyPass;
        OpenQA.Selenium.Proxy proxy = new OpenQA.Selenium.Proxy();
                    proxy.HttpProxy = PROXY;
                    proxy.FtpProxy = PROXY;
                    proxy.SslProxy = PROXY;
                    proxy.SocksProxy = PROXY;

    var serviceJS = PhantomJSDriverService.CreateDefaultService(phantomPath);
    serviceJS.AddArguments("--proxy=" + PROXY, "--proxy-type=http", "--proxy-auth=" + proxyAuth);

answered May 26 '15 at 18:41

Andrew_STOP_RU_WAR_IN_UA

9,318
5
65
101

Chrome is NOT using OS proxy, you can easily set it by `options.AddArguments("--proxy-server=...")` – Toolkit Mar 04 '17 at 14:07
@Toolkit did you try this code and it has work? I tried to work with chrome proxy with such code and it's didn't work for me. The same is for FF browser. – Andrew_STOP_RU_WAR_IN_UA Mar 04 '17 at 15:23
yes adding proxy to ChromeDriver is trivial, `options.AddArguments("--proxy-server=http://proxy.com:111");` – Toolkit Mar 04 '17 at 15:49
3

I saw this and tried 2 years ago. More than 8 hrs tried different ways to work with proxy and no one was successful. – Andrew_STOP_RU_WAR_IN_UA Mar 04 '17 at 16:48
i am using it with C# – Toolkit Mar 04 '17 at 16:52
@Toolkit have you used socks5 proxy with authorization? I am having the same issue as the topic starter. If you have a working configuration, could you please post it somewhere? – Impworks Aug 12 '17 at 20:07
1

At this point it is clear that Chrome with an un-authenticating proxy is trivial. However, an authenticated proxy is not. The question is specific on using an authenticated proxy. I have been unable to locate a solution barring using an outside plugin so far. – Liquidgenius Jun 28 '18 at 17:17
> you can easily set it by options.AddArgument It is not easy as you say. Already tried @Toolkit – Volatil3 Aug 07 '18 at 08:15

score 0 · Answer 8 · answered Dec 23 '15 at 07:59

THis is a temporary solution might work in initial state: Code is in Python: Download the plugin first from chrome plugin store : Proxy-Auto-Auth_v2.0.crx

        options = webdriver.ChromeOptions()
        options.add_extension("./Proxy-Auto-Auth_v2.0.crx")) #this will provide you a window to enter user name and proxy 
        driver = webdriver.Remote(command_executor=selenium_server,desired_capabilities=options.to_capabilities())

        or 

        driver = webdriver.Chrome(chrome_options=options)

Hairy Ass · Answer 9 · 2021-11-07T14:21:00.820

I have been searching extensively. This is what it works for me



    PROXY_HOST = 'FILL IN'  # rotating proxy or host
    PROXY_PORT = 8080 # port
    PROXY_USER = 'FILL IN' # username
    PROXY_PASS = 'FILL IN' # password
    http_proxies = { 'https' : 'http://' + PROXY_USER + ':' + PROXY_PASS + '@' + PROXY_HOST + ':' + str(PROXY_PORT) }

    manifest_json = """
    {
        "version": "1.0.0",
        "manifest_version": 2,
        "name": "Chrome Proxy",
        "permissions": [
            "proxy",
            "tabs",
            "unlimitedStorage",
            "storage",
            "<all_urls>",
            "webRequest",
            "webRequestBlocking"
        ],
        "background": {
            "scripts": ["background.js"]
        },
        "minimum_chrome_version":"22.0.0"
    }
    """

    background_js = """
    var config = {
            mode: "fixed_servers",
            rules: {
            singleProxy: {
                scheme: "http",
                host: "%s",
                port: parseInt(%s)
            },
            bypassList: ["localhost"]
            }
        };

    chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

    function callbackFn(details) {
        return {
            authCredentials: {
                username: "%s",
                password: "%s"
            }
        };
    }

    chrome.webRequest.onAuthRequired.addListener(
                callbackFn,
                {urls: ["<all_urls>"]},
                ['blocking']
    );
    """ % (PROXY_HOST, PROXY_PORT, PROXY_USER, PROXY_PASS)




    def get_chromedriver(self, use_proxy=False):

        driver_path = 'FILL IN'
        path = os.path.dirname(os.path.abspath(__file__))
        chrome_options = webdriver.ChromeOptions()
        if use_proxy:
            pluginfile = 'proxy_auth_plugin.zip'

            with zipfile.ZipFile(pluginfile, 'w') as zp:
                zp.writestr("manifest.json", libshared.manifest_json)
                zp.writestr("background.js", libshared.background_js)
            chrome_options.add_extension(pluginfile)

        driver = webdriver.Chrome(
            executable_path=driver_path,
            chrome_options=chrome_options)
        return driver

How to use authenticated proxy in selenium chromedriver?

9 Answers9

Linked