Running selenium behind a proxy server

Question

I have been using selenium for automatic browser simulations and web scraping in python and it has worked well for me. But now, I have to run it behind a proxy server. So now selenium open up the window but could not open the requested page because of proxy settings not set on the opened browser. Current code is as follows (sample):

from selenium import webdriver

sel = webdriver.Firefox()
sel.get('http://www.google.com')
sel.title
sel.quit()

How do I change the above code to work with proxy server now as well?

you can try this http://stackoverflow.com/a/38168865/5409601 — Rafayet Ullah, Jul 03 '16 at 10:24

score 23 · Accepted Answer · edited May 23 '17 at 11:54

23

You need to set desired capabilities or browser profile, like this:

profile = webdriver.FirefoxProfile()
profile.set_preference("network.proxy.type", 1)
profile.set_preference("network.proxy.http", "proxy.server.address")
profile.set_preference("network.proxy.http_port", "port_number")
profile.update_preferences()
driver = webdriver.Firefox(firefox_profile=profile)

Also see related threads:

edited May 23 '17 at 11:54

Community

1
1

answered Aug 01 '13 at 08:35

alecxe

462,703
120
1,088
1,195

I tried what you suggested but still not able to get past the proxy server – Aryabhatt Aug 01 '13 at 13:00
I checked the browser settings opened by selenium after doing the preference update. Actually issue is that, it is not setting http_port correctly (and leaving it out to be 0), because of which it is not connecting. Are there any issues in port setting ? – Aryabhatt Aug 01 '13 at 13:10
Hm, can you try setting it as a number (not string)? – alecxe Aug 01 '13 at 13:12
I was doing a stupid error. Basically, you dont need to put the port number under braces. It works well now. – Aryabhatt Aug 01 '13 at 13:21
how to do the same thing for phantomjs. Basically,instead of ` profile = webdriver.FirefoxProfile() profile.set_proxy(proxy.selenium_proxy())`,i want to use phantomjs for same proxy server. – proprius Jan 11 '16 at 08:31

score 8 · Answer 2 · answered Nov 04 '13 at 21:16

The official Selenium documentation (http://docs.seleniumhq.org/docs/04_webdriver_advanced.jsp#using-a-proxy) provides clear and helpful guidelines about using a proxy. For Firefox (which is the browser of choice in your sample code) you should do the following:

from selenium import webdriver
from selenium.webdriver.common.proxy import *

myProxy = "host:8080"

proxy = Proxy({
    'proxyType': ProxyType.MANUAL,
    'httpProxy': myProxy,
    'ftpProxy': myProxy,
    'sslProxy': myProxy,
    'noProxy': '' # set this value as desired
    })

driver = webdriver.Firefox(proxy=proxy)

score 3 · Answer 3 · answered Apr 01 '15 at 07:41

3

This will do the job:

import selenium
from selenium.webdriver.common.proxy import *

proxyHost = "my.proxy.host or IP"
proxyPort = "55555"

fp = webdriver.FirefoxProfile()
fp.set_preference("network.proxy.type", 1)
#fp.set_preference("network.proxy.http", proxyHost) #HTTP PROXY
#fp.set_preference("network.proxy.http_port", int(proxyPort))
#fp.set_preference("network.proxy.ssl", proxyHost) #SSL  PROXY
#fp.set_preference("network.proxy.ssl_port", int(proxyPort))
fp.set_preference('network.proxy.socks', proxyHost) #SOCKS PROXY
fp.set_preference('network.proxy.socks_port', int(proxyPort))
fp.update_preferences()

driver = webdriver.Firefox(firefox_profile=fp)

driver.get("http://www.whatismyip.com/")

answered Apr 01 '15 at 07:41

Pedro Lobito

94,083
31
258
268

2

how to add user:pass – Umair Ayub Sep 02 '16 at 18:48
1

Important to note: if `proxyHost` is a hostname should not contain the _"http://"_ prefix – salvob Feb 16 '17 at 15:48
@UmairAyub have you found a solution for adding user:pass? – Mpizos Dimitris Feb 25 '21 at 15:45

score 0 · Answer 4 · answered Mar 06 '18 at 13:26

def install_proxy(PROXY_HOST,PROXY_PORT):
    fp = webdriver.FirefoxProfile()
    print PROXY_PORT
    print PROXY_HOST
    fp.set_preference("network.proxy.type", 1)
    fp.set_preference("network.proxy.http",PROXY_HOST)
    fp.set_preference("network.proxy.http_port",int(PROXY_PORT))
    fp.set_preference("network.proxy.https",PROXY_HOST)
    fp.set_preference("network.proxy.https_port",int(PROXY_PORT))
    fp.set_preference("network.proxy.ssl",PROXY_HOST)
    fp.set_preference("network.proxy.ssl_port",int(PROXY_PORT))  
    fp.set_preference("network.proxy.ftp",PROXY_HOST)
    fp.set_preference("network.proxy.ftp_port",int(PROXY_PORT))   
    fp.set_preference("network.proxy.socks",PROXY_HOST)
    fp.set_preference("network.proxy.socks_port",int(PROXY_PORT))   
    fp.set_preference("general.useragent.override","Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/7046A194A")
    fp.update_preferences()
    return webdriver.Firefox(firefox_profile=fp)

Running selenium behind a proxy server

4 Answers4

Linked