Using a proxy with selenium and ruby

Question

I'm trying to get selenium make request through a proxy server but I can't seem it get it working.

This bit of code gets a response but isn't using the proxy

Selenium::WebDriver.logger.level = :info

proxy = Selenium::WebDriver::Proxy.new(
  :http => "http://username:password@host:port", 
  :ssl => "http://username:password@host:port"
)

caps = Selenium::WebDriver::Remote::Capabilities.chrome(
  proxy: proxy,
  'goog:chromeOptions' => {
    'args' => ['headless', 'no-sandbox', 'disable-gpu', 'window-size=1920,1080']
  }
)

driver = Selenium::WebDriver.for(:chrome, :capabilities => caps)

driver.navigate.to "https://www.showmyip.com/"

2021-06-21 12:51:16 INFO Selenium -> POST session
2021-06-21 12:51:16 INFO Selenium    >>> http://127.0.0.1:9515/session | {"capabilities":{"alwaysMatch":{"proxy":{"proxyType":"manual","httpProxy":"http://username:password@host:port","sslProxy":"http://username:password@host:port"},"browserName":"chrome","goog:chromeOptions":{"args":["headless","no-sandbox","disable-gpu","window-size=1920,1080"]}}}}
2021-06-21 12:51:16 INFO Selenium <- {"value":{"capabilities":{"acceptInsecureCerts":false,"browserName":"chrome","browserVersion":"91.0.4472.114","chrome":{"chromedriverVersion":"91.0.4472.101 (af52a90bf87030dd1523486a1cd3ae25c5d76c9b-refs/branch-heads/4472@{#1462})","userDataDir":"/tmp/.com.google.Chrome.8Qfzwu"},"goog:chromeOptions":{"debuggerAddress":"localhost:46057"},"networkConnectionEnabled":false,"pageLoadStrategy":"normal","platformName":"linux","proxy":{"httpProxy":"http://username:password@host:port","proxyType":"manual","sslProxy":"http://username:password@host:port"},"setWindowRect":true,"strictFileInteractability":false,"timeouts":{"implicit":0,"pageLoad":300000,"script":30000},"unhandledPromptBehavior":"dismiss and notify","webauthn:extension:largeBlob":true,"webauthn:virtualAuthenticators":true},"sessionId":"b5a74fa8a44d298fb1fbffcc0516df83"}}
2021-06-21 12:51:16 INFO Selenium -> POST session/b5a74fa8a44d298fb1fbffcc0516df83/url
2021-06-21 12:51:16 INFO Selenium    >>> http://127.0.0.1:9515/session/b5a74fa8a44d298fb1fbffcc0516df83/url | {"url":"https://www.showmyip.com/"}
2021-06-21 12:51:18 INFO Selenium <- {"value":null}

Removing the scheme from the proxy string results in the same issue

2021-06-21 14:47:24 INFO Selenium    >>> http://127.0.0.1:9515/session | {"capabilities":{"alwaysMatch":{"proxy":{"proxyType":"manual","httpProxy":"username:password@host:port","sslProxy":"username:password@host:port"},"browserName":"chrome","goog:chromeOptions":{"args":["headless","no-sandbox","disable-gpu","window-size=1920,1080"]}}}}
2021-06-21 14:47:24 INFO Selenium <- {"value":{"capabilities":{"acceptInsecureCerts":false,"browserName":"chrome","browserVersion":"91.0.4472.114","chrome":{"chromedriverVersion":"91.0.4472.101 (af52a90bf87030dd1523486a1cd3ae25c5d76c9b-refs/branch-heads/4472@{#1462})","userDataDir":"/tmp/.com.google.Chrome.aVNdyH"},"goog:chromeOptions":{"debuggerAddress":"localhost:33363"},"networkConnectionEnabled":false,"pageLoadStrategy":"normal","platformName":"linux","proxy":{"httpProxy":"username:password@host:port","proxyType":"manual","sslProxy":"username:password@host:port"},"setWindowRect":true,"strictFileInteractability":false,"timeouts":{"implicit":0,"pageLoad":300000,"script":30000},"unhandledPromptBehavior":"dismiss and notify","webauthn:extension:largeBlob":true,"webauthn:virtualAuthenticators":true},"sessionId":"3d95da424ededc7f0a9f59dd8e20386d"}}
2021-06-21 14:47:24 INFO Selenium -> POST session/3d95da424ededc7f0a9f59dd8e20386d/url
2021-06-21 14:47:24 INFO Selenium    >>> http://127.0.0.1:9515/session/3d95da424ededc7f0a9f59dd8e20386d/url | {"url":"https://www.showmyip.com/"}
2021-06-21 14:47:25 INFO Selenium <- {"value":null}

I've also tried this approach

client = Selenium::WebDriver::Remote::Http::Default.new
client.proxy = Selenium::WebDriver::Proxy.new(
  :http => "http://username:password@host:port",
  :ssl => "http://username:password@host:port"
)

caps = Selenium::WebDriver::Remote::Capabilities.chrome(
  'goog:chromeOptions' => {
    'args' => ['headless', 'no-sandbox', 'disable-gpu', 'window-size=1920,1080']
  }
)

driver = Selenium::WebDriver.for(:remote, :http_client => client, :capabilities => caps)

driver.navigate.to "https://www.showmyip.com/"

Which gives me the following 502 error. The proxies I'm using work with Net/Http so I don't believe they are the problem

/home/will/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/selenium-webdriver-4.0.0.beta4/lib/selenium/webdriver/remote/http/common.rb:93:in `create_response': unexpected response, code=502, content-type="text/html" (Selenium::WebDriver::Error::WebDriverError)
<html><head><title>502 Bad Gateway</title></head>\r
<body><h2>502 Bad Gateway</h2><h3>Host Not Found or connection failed</h3></body></html>

If I enter the wrong username or password intentionally using the same code I get this error

/home/will/.rbenv/versions/3.0.0/lib/ruby/gems/3.0.0/gems/selenium-webdriver-4.0.0.beta4/lib/selenium/webdriver/remote/http/common.rb:93:in `create_response': unexpected response, code=407, content-type="text/html" (Selenium::WebDriver::Error::WebDriverError)
<html><head><title>407 Proxy Authentication Required</title></head>\r
<body><h2>407 Proxy Authentication Required</h2><h3>Access to requested resource disallowed by administrator or you need valid username/password to use this resource</h3></body></html>

Update

After further digging it appears that selenium is ignoring proxies with auth.

Using proxy_chain_rb to setup a proxy on localhost without authentication that redirects to your actual proxy seems to work.

This is not an ideal solution, I plan on using a different proxy for each request and as far as I can tell I will need to start the proxy server on each request which takes a few seconds.

real_proxy = "http://username:password@host:port"
server = ProxyChainRb::Server.new
generated_proxy = server.start(real_proxy)

proxy = {
  http: generated_proxy,
  ssl:  generated_proxy
}

caps = Selenium::WebDriver::Remote::Capabilities.chrome(
  proxy: proxy,
  'goog:chromeOptions' => {
    'args' => ['headless', 'no-sandbox', 'disable-gpu', 'window-size=1920,1080']
  }
)

driver = Selenium::WebDriver.for(:chrome, :capabilities => caps)

driver.navigate.to "https://www.showmyip.com/"

People who are maintaining the Ruby Selenium Binding are maintaining the WATIR binding, If you don't add, WATIR tag, you wouldn't get the help.So let me add the WATIR tag here. — Rajagopalan, Jun 21 '21 at 11:44
@Rajagopalan please do not add the Watir tag to these questions. I added a watch on ruby+selenium-webdriver questions. — titusfortner, Jun 21 '21 at 14:05
The second example you gave is if you want your Selenium commands to be proxied through something before getting sent to the driver. You want the first example, which routes the traffic being loaded by your browser to go through a proxy. It looks correct syntax wise... Are you... sure the username & password is correct? Can you turn on logging (`Selenium::WebDriver.logger.level = :info`) and provide the results? We can double check it correctly matches the spec https://w3c.github.io/webdriver/#proxy but the syntax should be good. — titusfortner, Jun 21 '21 at 14:15
@titusfortner Thanks for the help. The username and password are correct, it looks like it is being ignored because the proxy contains authentication. I've updated my question with the logs and further troubleshooting. — user2034595, Jun 21 '21 at 20:22
@titusfortner Hi add it for ruby+selenium not for selenium-webdriver, very few would add selenium-webdriver. — Rajagopalan, Jun 23 '21 at 02:46
Thanks for the solution in your update, helped me out too. Small comment: This library is not maintained any uses linux commands not available on windows, so you won't actually be able to test this on windows. but on a unix system works great. — NemyaNation, Sep 17 '21 at 02:55

titusfortner · Answer 1 · 2022-01-19T02:48:36.513

Update: This is getting downvotes, but it is the intended solution according to the WebDriver specification (additional reference: https://github.com/w3c/webdriver/issues/1150). If this does not work for you in Chrome, please file a bug with chromedriver, and/or use Firefox.

According to the spec, valid values for httpProxy and sslProxy are:

A host and optional port for [the] scheme

Further:

A host and optional port for a scheme is defined as being a valid host, optionally followed by a colon and a valid port. The host may include credentials. If the port is omitted and scheme has a default port, this is the implied port. Otherwise, the port is left undefined.

So... try removing the scheme from the String you are passing in:

proxy = Selenium::WebDriver::Proxy.new(
            http: "username:password@host:port", 
            ssl: "username:password@host:port"
)

Removing the scheme from the string still results in the proxy being ignored unfortunately. I've added the logs from this attempt to the original question — user2034595, Jun 21 '21 at 21:56
Does it make a difference if it isn't in headless mode? And does it by Chance work in Firefox? — titusfortner, Jun 22 '21 at 02:18

Using a proxy with selenium and ruby

Update

1 Answers1

Linked