Selenium webdriver.Remote driver does not work with tor proxy(webdriver.Chrome does)

Question

I'm trying to use socks5 proxy on my remote driver which is hosted as a docker container on port 4444.

here is code-sample:

from selenium import webdriver

opts = webdriver.ChromeOptions()
opts.add_argument("--no-sandbox")
opts.add_argument("--disable-dev-shm-usage")
opts.add_argument("--proxy-server=socks5://127.0.0.1:9050")
driver = webdriver.Remote(command_executor="http://localhost:4444/wd/hub", desired_capabilities=opts.to_capabilities())

Then, when I try to open any page, I get error stating Check your proxy settings or contact your network administrator.

While using same code sample on regular proxy – it works just fine. When I do bold request through 9050 port – it works just fine.

And finally, when I use same code sample with webdriver.Chrome instead of webdriver.Remote it works fine!

I would appreciate any suggestions to make things work through Remote webdriver.

UPDATE: I'm using selenium==3.14.0 and RemoteDriver is getting docker image selenium/node-chrome-debug:3.141.59-radium.

Could you please provide the chromedriver version and selenium package version you are using? Thanks. — alecxe, Dec 09 '19 at 04:21
@alecxe, thank you for reaching out. I've updated the question body with versions. — An Se, Dec 09 '19 at 17:21
@pguardiario tor proxy is running on localhost, but it's accessible(and working, for example using `requests` lib) both from inside container and localhost. — An Se, Dec 12 '19 at 15:21
Ok but the ip would be different from inside the container, yes? — pguardiario, Dec 13 '19 at 00:26

Sers · Answer 1 · 2021-01-28T07:23:10.143

For macOS and Windows you can use host.docker.internal to access local host from container:

from selenium import webdriver

opts = webdriver.ChromeOptions()
opts.add_argument("--no-sandbox")
opts.add_argument("--disable-dev-shm-usage")
opts.add_argument("--proxy-server=socks5://host.docker.internal:9050")
driver = webdriver.Remote(command_executor="http://127.0.0.1:4444/wd/hub", desired_capabilities=opts.to_capabilities())

driver.get("http://jsonip.com/")
print(driver.find_element_by_css_selector("html").text)
driver.quit()

Here is how selenium hub works with tor proxy. You can create a network in docker, attach containers to it and then use container name as a proxy host:

docker network create mynetwork
docker run -it -p 8118:8118 -p 9050:9050 --name tor-proxy -d dperson/torproxy
docker run -d -p 4444:4444 --name selenium-hub -v /dev/shm:/dev/shm selenium/standalone-chrome:3.141.59-yttrium
docker network connect mynetwork hub
docker network connect mynetwork tor-proxy

Here is the same example but with docker-compose:

version: '3.5'

services:

  tor-proxy:
    image: dperson/torproxy
    container_name: tor-proxy
    ports:
      - "8118:8118"
      - "9050:9050"
    networks:
      - mynetwork
  
  selenium-hub:
    image: selenium/standalone-chrome:3.141.59-yttrium
    container_name: selenium-hub
    ports:
      - "4444:4444"
    networks:
      - mynetwork

networks:
  mynetwork:
    name: mynetwork
    driver: bridge

Python Code:

from selenium import webdriver

opts = webdriver.ChromeOptions()
opts.add_argument("--no-sandbox")
opts.add_argument("--disable-dev-shm-usage")
opts.add_argument("--proxy-server=socks5://tor-proxy:9050")
driver = webdriver.Remote(command_executor="http://127.0.0.1:4444/wd/hub", desired_capabilities=opts.to_capabilities())

driver.get("http://jsonip.com/")
print(driver.find_element_by_css_selector("html").text)
driver.quit()

Result:

{"ip":"18.27.197.252","about":"https://jsonip.com/about","Pro!":"http://getjsonip.com","Get Notifications": "https://jsonip.com/notify"}

Process finished with exit code 0

Run again with changed IP:

{"ip":"178.165.72.177","about":"https://jsonip.com/about","Pro!":"http://getjsonip.com","Get Notifications": "https://jsonip.com/notify"}

Process finished with exit code 0

thank you very much for your answer, it does work the way you describe. By any chance, do you have suggestions why doesn't it work with tor proxy that is started on localhost? — An Se, Jan 14 '20 at 09:25
@AnSe What is your OS? Did you try with `host.docker.internal`? — Sers, Jan 14 '20 at 14:40

score 1 · Answer 2 · answered Jan 07 '20 at 11:31

1

I suggest you to add the configuration of your proxy in docker config.json file:

{
 "proxies":
 {
   "default":
   {
     "httpProxy": "socks5://127.0.0.1:9050",
     "httpsProxy": "socks5://127.0.0.1:9050"
   }
 }
}

and remove it from your python script

answered Jan 07 '20 at 11:31

LinPy

16,987
4
43
57

Tried this, but unfortunately I got same error, `Check your proxy settings or contact your network administrator.` – An Se Jan 08 '20 at 09:08
did you restart docker service after adding the configs ? – LinPy Jan 08 '20 at 09:10
Yep, done that twice. Did you tried to build webdriver with such config.json and it worked for you? – An Se Jan 08 '20 at 09:14
I use some proxies in my docker for internet conections , but not for webdriver – LinPy Jan 08 '20 at 09:25

Selenium webdriver.Remote driver does not work with tor proxy(webdriver.Chrome does)

2 Answers2

Linked