how to set proxy with authentication in selenium chromedriver python?

Question

I am creating a script that crawls one website to gather some data but the problem is that they blocked me after too many requests but using a proxy I can send more request then currently I do. I have integrated proxy with chrome option --proxy-server

options.add_argument('--proxy-server={}'.format('http://ip:port'))

but I am using a paid proxy so it requires authentication and as below screenshot it gives the alert box for username and password

Then I tried to use it with username and password

options.add_argument('--proxy-server={}'.format('http://username:password@ip:port'))

But it also does not seems to work. I was looking for a solution and found below solution and I used it with the chrome extension proxy auto auth and without the chrome extension

proxy = {'address': settings.PROXY,
             'username': settings.PROXY_USER,
             'password': settings.PROXY_PASSWORD}

capabilities = dict(DesiredCapabilities.CHROME)
capabilities['proxy'] = {'proxyType': 'MANUAL',
                             'httpProxy': proxy['address'],
                             'ftpProxy': proxy['address'],
                             'sslProxy': proxy['address'],
                             'noProxy': '',
                             'class': "org.openqa.selenium.Proxy",
                             'autodetect': False,
                             'socksUsername': proxy['username'],
                             'socksPassword': proxy['password']}
options.add_extension(os.path.join(settings.DIR, "extension_2_0.crx")) # proxy auth extension

but neither of above worked properly it seems working because after above code the proxy authentication alert disappeared and when I checked my IP by googling what is my IP and confirmed that is not working.

please anyone who can help me to authenticate the proxy server on chromedriver.

Answer 1

Selenium Chrome Proxy Authentication

Setting chromedriver proxy with Selenium using Python

If you need to use a proxy with python and Selenium library with chromedriver you usually use the following code (Without any username and password:

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--proxy-server=%s' % hostname + ":" + port)
driver = webdriver.Chrome(chrome_options=chrome_options)

It works fine unless proxy requires authentication. if the proxy requires you to log in with a username and password it will not work. In this case, you have to use more tricky solution that is explained below. By the way, if you whitelist your server IP address from the proxy provider or server it should not ask proxy credentials.

HTTP Proxy Authentication with Chromedriver in Selenium

To set up proxy authentication we will generate a special file and upload it to chromedriver dynamically using the following code below. This code configures selenium with chromedriver to use HTTP proxy that requires authentication with user/password pair.

import os
import zipfile

from selenium import webdriver

PROXY_HOST = '192.168.3.2'  # rotating proxy or host
PROXY_PORT = 8080 # port
PROXY_USER = 'proxy-user' # username
PROXY_PASS = 'proxy-password' # password


manifest_json = """
{
    "version": "1.0.0",
    "manifest_version": 2,
    "name": "Chrome Proxy",
    "permissions": [
        "proxy",
        "tabs",
        "unlimitedStorage",
        "storage",
        "<all_urls>",
        "webRequest",
        "webRequestBlocking"
    ],
    "background": {
        "scripts": ["background.js"]
    },
    "minimum_chrome_version":"22.0.0"
}
"""

background_js = """
var config = {
        mode: "fixed_servers",
        rules: {
        singleProxy: {
            scheme: "http",
            host: "%s",
            port: parseInt(%s)
        },
        bypassList: ["localhost"]
        }
    };

chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

function callbackFn(details) {
    return {
        authCredentials: {
            username: "%s",
            password: "%s"
        }
    };
}

chrome.webRequest.onAuthRequired.addListener(
            callbackFn,
            {urls: ["<all_urls>"]},
            ['blocking']
);
""" % (PROXY_HOST, PROXY_PORT, PROXY_USER, PROXY_PASS)


def get_chromedriver(use_proxy=False, user_agent=None):
    path = os.path.dirname(os.path.abspath(__file__))
    chrome_options = webdriver.ChromeOptions()
    if use_proxy:
        pluginfile = 'proxy_auth_plugin.zip'

        with zipfile.ZipFile(pluginfile, 'w') as zp:
            zp.writestr("manifest.json", manifest_json)
            zp.writestr("background.js", background_js)
        chrome_options.add_extension(pluginfile)
    if user_agent:
        chrome_options.add_argument('--user-agent=%s' % user_agent)
    driver = webdriver.Chrome(
        os.path.join(path, 'chromedriver'),
        chrome_options=chrome_options)
    return driver

def main():
    driver = get_chromedriver(use_proxy=True)
    #driver.get('https://www.google.com/search?q=my+ip+address')
    driver.get('https://httpbin.org/ip')

if __name__ == '__main__':
    main()

Function get_chromedriver returns configured selenium webdriver that you can use in your application. This code is tested and works just fine.

Read more about onAuthRequired event in Chrome.

Answer 2

Use selenium-wire.

Example code from the documentation:

HTTP proxies

from seleniumwire import webdriver

options = {
    'proxy': {
        'http': 'http://user:pass@192.168.10.100:8888',
        'https': 'https://user:pass@192.168.10.100:8888',
        'no_proxy': 'localhost,127.0.0.1'
    }
}

driver = webdriver.Chrome(seleniumwire_options=options)

SOCKS proxies

from seleniumwire import webdriver

options = {
   'proxy': {
        'http': 'socks5://user:pass@192.168.10.100:8888',
        'https': 'socks5://user:pass@192.168.10.100:8888',
        'no_proxy': 'localhost,127.0.0.1'
    }
}
driver = webdriver.Chrome(seleniumwire_options=options)

Install with:

pip install selenium-wire

Answer 3

Here is a quick, creative solution that doesn't require modification of selenium's Options or uploading a file to chromedriver. It makes use of pyautogui (can use any python package that simulates key presses) to enter proxy auth details. It also uses threading to account for chrome authentication popup window that would otherwise pause the script.

import time
from threading import Thread
import pyautogui
from selenium.webdriver.chrome.options import Options
from selenium import webdriver

hostname = "HOST_NAME"
port = "PORT"
proxy_username = "USERNAME"
proxy_password = "PASSWORD"

chrome_options = Options()
chrome_options.add_argument('--proxy-server={}'.format(hostname + ":" + port))
driver = webdriver.Chrome(options=chrome_options)


def enter_proxy_auth(proxy_username, proxy_password):
    time.sleep(1)
    pyautogui.typewrite(proxy_username)
    pyautogui.press('tab')
    pyautogui.typewrite(proxy_password)
    pyautogui.press('enter')


def open_a_page(driver, url):
    driver.get(url)


Thread(target=open_a_page, args=(driver, "http://www.example.com/")).start()
Thread(target=enter_proxy_auth, args=(proxy_username, proxy_password)).start()

NOTE: For any serious project or test suite I would recommend opting for a more robust solution. However, if you are just experimenting and require a quick and effective solution, this is an option.

Answer 4

I was looking for the same answer, but only for Java code, so here is my variant of @itsmnthn Python code.

Don't forget to change String fields of MainTest class to your ip, port, login, password and a chromeDriver path.

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;

import java.io.*;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;

public class MainTest {
    private static final String PROXY_HOST = "127.0.0.1";
    private static final String PROXY_PORT = "8080";
    private static final String PROXY_USER = "login";
    private static final String PROXY_PASS = "password";
    private static final String CHROMEDRIVER_PATH = "chromeDriverPath";
    private static final String PROXY_OPTION_TEMPLATE = "--proxy-server=http://%s";

    public static void main(String[] args) throws IOException {
        System.setProperty("webdriver.chrome.driver", CHROMEDRIVER_PATH);
        ChromeOptions options = new ChromeOptions();
        String manifest_json = "{\n" +
                "  \"version\": \"1.0.0\",\n" +
                "  \"manifest_version\": 2,\n" +
                "  \"name\": \"Chrome Proxy\",\n" +
                "  \"permissions\": [\n" +
                "    \"proxy\",\n" +
                "    \"tabs\",\n" +
                "    \"unlimitedStorage\",\n" +
                "    \"storage\",\n" +
                "    \"<all_urls>\",\n" +
                "    \"webRequest\",\n" +
                "    \"webRequestBlocking\"\n" +
                "  ],\n" +
                "  \"background\": {\n" +
                "    \"scripts\": [\"background.js\"]\n" +
                "  },\n" +
                "  \"minimum_chrome_version\":\"22.0.0\"\n" +
                "}";

        String background_js = String.format("var config = {\n" +
                "  mode: \"fixed_servers\",\n" +
                "  rules: {\n" +
                "    singleProxy: {\n" +
                "      scheme: \"http\",\n" +
                "      host: \"%s\",\n" +
                "      port: parseInt(%s)\n" +
                "    },\n" +
                "    bypassList: [\"localhost\"]\n" +
                "  }\n" +
                "};\n" +
                "\n" +
                "chrome.proxy.settings.set({value: config, scope: \"regular\"}, function() {});\n" +
                "\n" +
                "function callbackFn(details) {\n" +
                "return {\n" +
                "authCredentials: {\n" +
                "username: \"%s\",\n" +
                "password: \"%s\"\n" +
                "}\n" +
                "};\n" +
                "}\n" +
                "\n" +
                "chrome.webRequest.onAuthRequired.addListener(\n" +
                "callbackFn,\n" +
                "{urls: [\"<all_urls>\"]},\n" +
                "['blocking']\n" +
                ");", PROXY_HOST, PROXY_PORT, PROXY_USER, PROXY_PASS);

        FileOutputStream fos = new FileOutputStream("proxy_auth_plugin.zip");
        ZipOutputStream zipOS = new ZipOutputStream(fos);

        createFile("manifest.json", manifest_json);
        createFile("background.js", background_js);

        File file = new File("proxy_auth_plugin.zip");
        writeToZipFile("manifest.json", zipOS);
        writeToZipFile("background.js", zipOS);
        zipOS.close();
        fos.close();
        options.addExtensions(file);

        WebDriver driver = new ChromeDriver(options);
        try {
            driver.get("https://2ip.ru");
        } finally {
            driver.close();
        }

    }

    public static void writeToZipFile(String path, ZipOutputStream zipStream) throws FileNotFoundException, IOException {
        System.out.println("Writing file : '" + path + "' to zip file");
        File aFile = new File(path);
        FileInputStream fis = new FileInputStream(aFile);
        ZipEntry zipEntry = new ZipEntry(path);
        zipStream.putNextEntry(zipEntry);
        byte[] bytes = new byte[1024];
        int length;
        while ((length = fis.read(bytes)) >= 0) {
            zipStream.write(bytes, 0, length);
        }
        zipStream.closeEntry();
        fis.close();
    }

    public static void createFile(String filename, String text) throws FileNotFoundException {
        try (PrintWriter out = new PrintWriter(filename)) {
            out.println(text);
        }
    }


}

Answer 5

I had the same problem - and isn´t it possible to combine the selenium-wire function with the headless function from the Options - for me this code was working for example - is there anything wrong with that?

from seleniumwire import webdriver
from selenium.webdriver.chrome.options import Options
import os, sys, time
from dotenv import load_dotenv, find_dotenv

path = os.path.abspath (os.path.dirname (sys.argv[0]))
cd = '/chromedriver.exe'
load_dotenv(find_dotenv()) 
PROXY_CHEAP_USER = os.environ.get("PROXY_CHEAP_USER")
PROXY_CHEAP_PW= os.environ.get("PROXY_CHEAP_PW")
PROXY_HOST = 'proxyhost.com'  # rotating proxy or host
PROXY_PORT = port # port
PROXY_USER = PROXY_CHEAP_USER # username
PROXY_PASS = PROXY_CHEAP_PW # password

options = Options()
options.add_argument('--headless')
options.add_argument("--window-size=1920x1080")
options.add_argument('--no-sandbox')
options.add_argument('--disable-gpu')

options_seleniumWire = {
    'proxy': {
        'https': f'https://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}',
    }
}
 
driver = webdriver.Chrome (path + cd, options=options, seleniumwire_options=options_seleniumWire)
driver.get("https://ifconfig.co/")

I think this solution also works in headless mode.

Answer 6

Because it does not seem to be possible to directly configure the Chromedriver to use a proxy that requires authentication, you can use a local downstream proxy that does not require any authentication. This local proxy then sends all requests to your "real" proxy that you wanted to use in the first place with the required authentication.

I have used tinyproxy to do this. You can add the following line to the tinyproxy-configuration (tinyproxy.conf):

upstream http user:pass@host:port

Make sure to replace user, pass, host and port with the values of the proxy that you want to use.

Then you can configure your Chromedriver to use the tinyproxy as already described in earlier answers. Tinyprox runs on port 8888 by default, so you can reach it on 127.0.0.1:8888. As already mentioned in this answer, it is pretty easy to use a proxy without authentication:

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--proxy-server=127.0.0.1:8888')
driver = webdriver.Chrome(chrome_options=chrome_options)

Answer 7

Here is manifest version 3 to accompany latest versions of Chrome and @itsmnthn solution

{
    "version": "1.0.0",
    "manifest_version": 3,
    "name": "Chrome Proxy",
    "permissions": [
        "proxy",
        "tabs",
        "unlimitedStorage",
        "storage",
        "webRequest",
        "webRequestAuthProvider"
        ],
    "host_permissions": [
        "<all_urls>"
    ],
    "background": {
        "service_worker": "background.js"
    },
    "minimum_chrome_version":"22.0.0"
}

And C# class that handles this issue (again, based on @itsmnthn solution)

public class ProxiedChromeClient : IDisposable
{
    private const string MANIFEST_JSON = @"
    {
        ""version"": ""1.0.0"",
        ""manifest_version"": 3,
        ""name"": ""Chrome Proxy"",
        ""permissions"": [
            ""proxy"",
            ""tabs"",
            ""unlimitedStorage"",
            ""storage"",
            ""webRequest"",
            ""webRequestAuthProvider""
            ],
        ""host_permissions"": [
            ""<all_urls>""
        ],
        ""background"": {
            ""service_worker"": ""background.js""
        },
        ""minimum_chrome_version"":""22.0.0""
    }";

    private const string BACKGROUND_JS = @"
    var config = {{
        mode: ""fixed_servers"",
        rules: {{
            singleProxy: {{
                scheme: ""{0}"",
                host: ""{1}"",
                port: parseInt({2})
            }},
            bypassList: [""localhost""]
        }}
    }};

    chrome.proxy.settings.set({{value: config, scope: ""regular""}}, function() {{}});

    function callbackFn(details) {{
        return {{
            authCredentials: {{
                username: ""{3}"",
                password: ""{4}""
            }}
        }};
    }}

    chrome.webRequest.onAuthRequired.addListener(
        callbackFn,
        {{urls: [""<all_urls>""]}},
        ['blocking']
    );";

    protected ProxiedChromeClient(ProxyInfo proxy = null)
    {
        var options = new ChromeOptions();
        if (proxy != null)
        {
            extensionPath = CreateProxyExtension(proxy);
            options.AddExtension(extensionPath);
        }

        chromeDriverInstance = new ChromeDriver(options);
    }


    protected readonly ChromeDriver chromeDriverInstance;
    private readonly object @lock = new();
    private readonly string extensionPath;

    private static string CreateProxyExtension(ProxyInfo proxy)
    {
        // per https://stackoverflow.com/a/55582859/307584
        var tempFile = Path.GetTempFileName();
        using var z = new ZipArchive(new FileStream(tempFile, FileMode.Create), ZipArchiveMode.Create);
        var entry = z.CreateEntry("manifest.json");
        using (var writer = new StreamWriter(entry.Open()))
        {
            writer.Write(MANIFEST_JSON);
        }

        entry = z.CreateEntry("background.js");
        var url = new Uri(proxy.Url);
        using (var writer = new StreamWriter(entry.Open()))
        {
            writer.Write(BACKGROUND_JS, url.Scheme, url.Host, url.Port, proxy.User, proxy.Password);
        }

        return tempFile;
    }

    public void Dispose()
    {
        lock (@lock)
        {
            chromeDriverInstance.Quit();
            if (extensionPath != null)
            {
                File.Delete(extensionPath);
            }
        }
    }
}

Answer 8

Along the way, in the updates the solution using the extension doesnt work (windows at least), while mac and linux does. I think it was chromedriver v2.44 the last working version with extensions

Answer 9

There are several workarounds for this issue, but it is currently impossible to resolve the authentication dialogue box in Selenium. See this issue:

There is currently no way to handle HTTP authentication prompts when navigating to a page, only pre-authentication with username/password in the URL works (and, apparently, not without workarounds in some browsers like IE).

Answer 10

There is a more simple way, how you should do authentication to proxy with selenium! I don't know from which version it works, but in selenium-api-4.5.3 for java there is a interface HasAuthentication. You can use register method after initializing driver to set user+password for whole browser.

This worked for me with chromium:

((ChromiumDriver) driver).register(() -> new UsernameAndPassword("USER", "PASS"));

and set Proxy in options

ChromeOptions options = new ChromeOptions();
options.setCapability("proxy", initProxy());
new ChromeDriver(options);

works for me in headless mode too.