Use selenium chrome with non local proxy

Question

import scrapy
from scrapy.crawler import CrawlerProcess
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.proxy import Proxy, ProxyType
class FooSpider(scrapy.Spider):
    name = 'foo'

    start_urls = ["https://www.whatismybrowser.com/"]
    index=1

    def __init__(self, *args, **kwargs):
        super(FooSpider, self).__init__(*args, **kwargs)
        self.download_delay = 0.25
        chrome_options = Options()  # Initializing Chrome
        #chrome_options.add_argument("--headless")
        chrome_options.add_argument('--ignore-certificate-errors')
        chrome_options.add_argument('--ignore-ssl-errors')
        IP = '176.31.69.183' # random free proxy from net
        PORT = 8080
        prox = Proxy()
        prox.proxy_type = ProxyType.MANUAL
        prox.http_proxy = f'{IP}:{PORT}'
        prox.socks_proxy = f'{IP}:{PORT}'
        prox.ssl_proxy = f'{IP}:{PORT}'
        capabilities = webdriver.DesiredCapabilities.CHROME
        prox.add_to_capabilities(capabilities)
        self.browser = webdriver.Chrome(executable_path="/home/timmy/Downloads/chromedriver",options=chrome_options, desired_capabilities=capabilities)
        #self.browser.implicitly_wait(60) # 

    def parse(self,response):
        self.browser.get(response.url)
        data= self.random_data()
        print(data)

process = CrawlerProcess({'LOG_LEVEL':'INFO',})
process.crawl(FooSpider)
spider = next(iter(process.crawlers)).spider
process.start()

this is the error I am getting

raise exception_class(message, screen, stacktrace) selenium.common.exceptions.InvalidArgumentException: Message: invalid argument: cannot parse capability: proxy from invalid argument: Specifying 'socksProxy' requires an integer for 'socksVersion' I got the proxy from SSlproxies and I am trying to use it, I am using the answer from this question running-selenium-webdriver-with-a-proxy-in-python but I got the error above

How can I fix?

Please look at my updated answer and try it out. Please accept the answer it solves your problem of using proxies with Chrome. — ThePyGuy, Nov 13 '20 at 10:39
it does not solve the problem, as after creating driver when you visit https://www.whatsmyip.org/ it shows you your original IP — Nauman Sharif, Jan 04 '21 at 07:39

score 6 · Accepted Answer · edited Jul 30 '22 at 09:46

6

You are using old syntax for chrome proxies. This is the method of setting a proxy server with no authentication.

from selenium.webdriver.chrome.options import Options

chrome_options = Options()
chrome_options.add_argument('--proxy-server=http://ipaddress:port')
driver = webdriver.Chrome(chrome_options=chrome_options)

edited Jul 30 '22 at 09:46

P i

29,020
36
159
267

answered Jun 21 '19 at 01:17

ThePyGuy

1,025
1
6
15

Its worth nothing sometimes you can connect to a proxy with http and still handle https requests I could be wrong on this. You're connection may be http but it can handle ssl on the backend I believe. So if your proxy is http but handles ssl then it should be fine. I just successfully used one of those proxies with this code...some of the proxies may be bad on a freelist and say bad internet but try ones in your area. They're free so they get hammered on hard and go down. Try both http and https but I was able to get the proxy to work. Just realize these proxies are probably real bad. – ThePyGuy Jun 22 '19 at 02:21
if you have problems with this if switching to firefox is an option then you can set http and https proxy explicitly. – ThePyGuy Jun 22 '19 at 02:22

Use selenium chrome with non local proxy

1 Answers1