Proxy server for Selenium and Python

Question

I'm trying to use a proxy server to launch the Selenium Chrome driver. The only solution so far that I found was to use a sort of plugin for Chrome to authenticate, but it's not very reliable, so I was wondering if there is any other option.

Here is what I use now

    manifest_json = """
    {
    "version": "1.0.0",
    "manifest_version": 2,
    "name": "Chrome Proxy",
    "permissions": [
    "proxy",
    "tabs",
    "unlimitedStorage",
    "storage",
    "<all_urls>",
    "webRequest",
    "webRequestBlocking"
    ],
    "background": {
    "scripts": ["background.js"]
    },
    "minimum_chrome_version":"22.0.0"
    }
    """

background_js = """
    var config = {
    mode: "fixed_servers",
    rules: {
    singleProxy: {
    scheme: "http",
    host: "",
    port: parseInt(6060)
    },
    bypassList: ["foobar.com"]
    }
    };

    chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

    function callbackFn(details) {
    return {
    authCredentials: {
    username: "",
    password: ""
    }
    };
    }

    chrome.webRequest.onAuthRequired.addListener(
    callbackFn,
    {urls: ["<all_urls>"]},
    ['blocking']
    );
    """


pluginfile = 'proxy_auth_plugin.zip'

with zipfile.ZipFile(pluginfile, 'w') as zp:
    zp.writestr("manifest.json", manifest_json)
    zp.writestr("background.js", background_js)

co = Options()
co.add_argument("--start-maximized")
co.add_extension(pluginfile)


driver = webdriver.Chrome(executable_path='/usr/local/bin/chromedriver')

Answer 1

I don't think plugins are a very good solution, certainly not when there are pure HTTP solutions that work the same across all browsers, versions, and operating systems.

You could use this Python wrapper for Browsermob, which is a very well-known standalone or embedded proxy server written in Java.

The code shows how the headers() Python method can be used to add headers, without the need to POST to the REST API. The tests have some examples of it in use, e.g.:

self.client = Client("localhost:9090")
self.client.headers({'User-Agent': 'rubber ducks floating in a row'})
self.client.headers({'Authorization': 'Basic dXNlcjpwYXNz'})  # User 'user', password 'pass'

Update:

Just to clarify how the WebDriver and the proxy fit together:

• Start your proxy server first, and wait until it's ready. You can do that externally and pass host:port to WebDriver, or start it embedded in your application then pass WebDriver a proxy object.
• This example demonstrates the second approach, using Firefox profiles and Chrome options.
• Alternatively, start the proxy embedded, use it to obtain a Proxy object, which represents a Selenium proxy server, then add it to your DesiredCapabilities object, and create your driver that way, as per this example.

From that point on, the proxy-listening is automatic, and you can start to create your HAR files.

---

Alternatively, you could see this answer for a custom Python proxy server using Twisted.

I have a longer answer about Selenium proxies in Python here.