Python proxy authentication through Selenium chromedriver

Question

We tried for a few days to setup Proxy Authentication with selenium chromedriver in Python. We couldn't set ip up because Chrome throws a pop-up for authentication. Problem is that selenium can't switch to that window and so, can't type. The only solution that worked for us was using pyautogui which is a bad solution for us because we want to use the headless function.

Here are all the methods we tried:

driver.switch_to_window()

driver.switch_to_active_element()

driver.switch_to_alert()

ActionChains(driver).send_keys("test").perform()

driver.switch_to_alert()

options = webdriver.ChromeOptions()
options.add_argument('--proxy-server=http://user:pass@3.223.68.195:31112')

driver.get("https://user:pass@google.com")

proxy = {'address': '3.209.253.119:31112',
         'username': 'user',
         'password': 'pass'}


capabilities = dict(DesiredCapabilities.CHROME)
capabilities['proxy'] = {'proxyType': 'MANUAL',
                         'httpProxy': proxy['address'],
                         'ftpProxy': proxy['address'],
                         'sslProxy': proxy['address'],
                         'noProxy': '',
                         'class': "org.openqa.selenium.Proxy",
                         'autodetect': False}

capabilities['proxy']['socksUsername'] = proxy['username']
capabilities['proxy']['socksPassword'] = proxy['password']

driver = webdriver.Chrome(executable_path="chromedriver.exe", desired_capabilities=capabilities)
driver.get("http://google.com")

Any help would be really appreciated :)

score 2 · Answer 1 · edited Jan 20 '21 at 02:46

If you need to use a proxy with no authentication (no username or password) with python and Selenium library with chromedriver you usually use the following code:

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--proxy-server=%s' % hostname + ":" + port)
driver = webdriver.Chrome(chrome_options=chrome_options)

It works fine unless proxy requires authentication. if the proxy requires you to log in with a username and password it will not work. In this case, you have to use more tricky solution that is explained below.

To set up proxy authentication we will generate a file and upload it to chromedriver dynamically using the following code below. This effectively create a chrome extension. This code configures selenium with chromedriver to use HTTP proxy that requires authentication with user/password pair.

import os
import zipfile

from selenium import webdriver


def create_chromedriver(PROXY_HOST, PROXY_PORT, PROXY_USER, PROXY_PASS, USER_AGENT):
    manifest_json = """
    {
    "version": "1.0.0",
    "manifest_version": 2,
    "name": "Chrome Proxy",
    "permissions": [
        "proxy",
        "tabs",
        "unlimitedStorage",
        "storage",
        "<all_urls>",
        "webRequest",
        "webRequestBlocking"
    ],
    "background": {
        "scripts": ["background.js"]
    },
    "minimum_chrome_version":"22.0.0"
    }
    """

    background_js = """
    var config = {
        mode: "fixed_servers",
        rules: {
        singleProxy: {
            scheme: "http",
            host: "%s",
            port: parseInt(%s)
        },
        bypassList: ["localhost"]
        }
    };

    chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

    function callbackFn(details) {
        return {
            authCredentials: {
            username: "%s",
            password: "%s"
            }
        };
    }

    chrome.webRequest.onAuthRequired.addListener(
            callbackFn,
            {urls: ["<all_urls>"]},
            ['blocking']
    );
    """ % (PROXY_HOST, PROXY_PORT, PROXY_USER, PROXY_PASS)


    def get_chromedriver(use_proxy=True, user_agent=USER_AGENT):
        path = os.path.dirname(os.path.abspath(__file__))
        chrome_options = webdriver.ChromeOptions()
        if use_proxy:
            pluginfile = 'proxy_auth_plugin.zip'
            with zipfile.ZipFile(pluginfile, 'w') as zp:
                zp.writestr("manifest.json", manifest_json)
                zp.writestr("background.js", background_js)
            chrome_options.add_extension(pluginfile)
        if user_agent:
            chrome_options.add_argument('--user-agent=%s' % USER_AGENT)
        driver = webdriver.Chrome(
            os.path.join(path, 'chromedriver'),
            chrome_options=chrome_options)
        return driver

    driver = get_chromedriver(use_proxy=True)
    # driver.get('https://www.google.com/search?q=my+ip+address')
    driver.get('https://httpbin.org/ip')


user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36'
create_chromedriver('192.168.3.2', 8080, 'user', 'pass', user_agent)

Function get_chromedriver returns configured selenium web driver that you can use in your application. This code is tested and works just fine.

To implement this into your own code just call the function create_chromedriver with args of host, port, user, pass and user-agent. If you do not want to use proxy or user-agent just edit get_chromedriver accordingly so the args are false.

Yes, we came across this solution become it seems like this is the only one that works. Unfortunately it's still not a perfect solution since you can't use extensions in headless mode. — DavidKoskasa, Nov 16 '20 at 00:38
Also, the above solution currently uses Manifest V2 extension syntax, which will stop working in January 2023 https://stackoverflow.com/questions/55582136/how-to-set-proxy-with-authentication-in-selenium-chromedriver-python — lam vu Nguyen, Mar 14 '23 at 08:20

user20986396 · Answer 2 · 2023-01-11T21:22:41.653

1

@luke-hamilton You can configure an authenticated proxy via an extension like this and run headlessly if you include this option

chrome_options.add_argument("--headless=chrome")

edited Jan 11 '23 at 21:22

answered Jan 11 '23 at 21:20

user20986396

11
3

score 0 · Answer 3 · answered Jun 28 '23 at 16:12

Using Selenium-Profiles, this is allready implemented into the driver.

Example script:

from selenium_profiles.webdriver import Chrome
from selenium_profiles.profiles import profiles

profile = profiles.Windows() # or .Android()
profile["proxy"] = {
  "proxy":"http://user1:pass1@example_host.com:41149"
  }

driver = Chrome(profile=profile)

driver.quit()  # Execute on the End!

Note: Setting the proxy dynamically is supported as well

Disclaimer: I am the author of Selenium-Profiles

Python proxy authentication through Selenium chromedriver

3 Answers3

Linked