0

I have a code that makes me repeatedly search and fetch data from a site. I am using Selenium in Python, and the code here seems to work even now and fetches a list of free IP addresses and Ports.

These are a few IP address, that I just fetched using that code.

{'IP Address': '35.213.91.45', 'Port': '80', 'Code': 'JP', 'Country': 'Japan', 'Anonymity': 'anonymous', 'Google': 'yes', 'Https': 'yes', 'Last Checked': '12 secs ago'}
{'IP Address': '140.238.245.116', 'Port': '8100', 'Code': 'IN', 'Country': 'India', 'Anonymity': 'anonymous', 'Google': '', 'Https': 'yes', 'Last Checked': '12 secs ago'}
{'IP Address': '182.23.113.179', 'Port': '8080', 'Code': 'ID', 'Country': 'Indonesia', 'Anonymity': 'anonymous', 'Google': '', 'Https': 'yes', 'Last Checked': '12 secs ago'}
{'IP Address': '198.27.74.6', 'Port': '9300', 'Code': 'CA', 'Country': 'Canada', 'Anonymity': 'anonymous', 'Google': '', 'Https': 'yes', 'Last Checked': '12 secs ago'}
{'IP Address': '195.158.30.89', 'Port': '3128', 'Code': 'UZ', 'Country': 'Uzbekistan', 'Anonymity': 'elite proxy', 'Google': 'no', 'Https': 'yes', 'Last Checked': '12 secs ago'}
{'IP Address': '77.247.108.17', 'Port': '33080', 'Code': 'BZ', 'Country': 'Belize', 'Anonymity': 'elite proxy', 'Google': 'no', 'Https': 'yes', 'Last Checked': '12 secs ago'}
{'IP Address': '190.202.3.22', 'Port': '32650', 'Code': 'VE', 'Country': 'Venezuela', 'Anonymity': 'elite proxy', 'Google': '', 'Https': 'yes', 'Last Checked': '12 secs ago'}
{'IP Address': '103.134.44.176', 'Port': '8080', 'Code': 'IN', 'Country': 'India', 'Anonymity': 'elite proxy', 'Google': 'no', 'Https': 'yes', 'Last Checked': '12 secs ago'}
{'IP Address': '202.86.138.18', 'Port': '8080', 'Code': 'MO', 'Country': 'Macau', 'Anonymity': 'elite proxy', 'Google': 'no', 'Https': 'yes', 'Last Checked': '12 secs ago'}
{'IP Address': '8.219.97.248', 'Port': '80', 'Code': 'SG', 'Country': 'Singapore', 'Anonymity': 'anonymous', 'Google': 'no', 'Https': 'yes', 'Last Checked': '12 secs ago'}

Here's the code I was using initially without changing any IP address.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()

driver.get("https://tuludictionary.in/dictionary/cgi-bin/web/frame.html")
driver.maximize_window()

But since this is a long process, the program gets pretty slow after some time. I would want to rotate the IP address, say every 10/15 minutes, and restart the driver for that. (Also, What header options should I add which might make it faster?)

I saw this answer here, but I am not sure if I could utilise it in my code.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from fake_useragent import UserAgent
from selenium.webdriver.support.ui import WebDriverWait
from webdriver_manager.chrome import ChromeDriverManager

WAIT = 10
srv=Service(ChromeDriverManager().install())
ua = UserAgent()
userAgent = ua.random
options = Options()
options.add_argument('--headless')
options.add_experimental_option ('excludeSwitches', ['enable-logging'])
options.add_argument("start-maximized")
options.add_argument('window-size=1920x1080')                                 
options.add_argument('--no-sandbox')
options.add_argument('--disable-gpu')  
options.add_argument(f'user-agent={userAgent}')     
driver = webdriver.Chrome (service=srv, options=options)    
waitWebDriver = WebDriverWait (driver, 10)  

link = "https://whatismyipaddress.com/"
driver.get(link)

I found a code here that apparently would work if I used the Firefox web driver, how can I update this for Chrome?

proxy = "124.240.187.80:82"

webdriver.DesiredCapabilities.FIREFOX['proxy'] = {
   "httpProxy":proxy,
   "ftpProxy":proxy,
   "sslProxy":proxy,
   "noProxy":None,
   "proxyType":"MANUAL",
   "class":"org.openqa.selenium.Proxy",
   "autodetect":False
}

I want to randomly choose one of the Proxy IP Addresses and its port stored as a list of dictionaries fetched from the site every 15 minutes and then restart the driver with those settings.

Milind
  • 11
  • 8
  • Does this answer your question? [How do I set proxy for chrome in python webdriver?](https://stackoverflow.com/questions/11450158/how-do-i-set-proxy-for-chrome-in-python-webdriver) – Zero Jun 28 '23 at 22:56
  • @Zero chrome_options doesn't work anymore, does it? – Milind Jun 28 '23 at 23:11
  • Try using `options` instead of `chrome_options`, most of the underlying functionality still works. – Zero Jun 28 '23 at 23:13

0 Answers0