I have a code that makes me repeatedly search and fetch data from a site. I am using Selenium in Python, and the code here seems to work even now and fetches a list of free IP addresses and Ports.
These are a few IP address, that I just fetched using that code.
{'IP Address': '35.213.91.45', 'Port': '80', 'Code': 'JP', 'Country': 'Japan', 'Anonymity': 'anonymous', 'Google': 'yes', 'Https': 'yes', 'Last Checked': '12 secs ago'}
{'IP Address': '140.238.245.116', 'Port': '8100', 'Code': 'IN', 'Country': 'India', 'Anonymity': 'anonymous', 'Google': '', 'Https': 'yes', 'Last Checked': '12 secs ago'}
{'IP Address': '182.23.113.179', 'Port': '8080', 'Code': 'ID', 'Country': 'Indonesia', 'Anonymity': 'anonymous', 'Google': '', 'Https': 'yes', 'Last Checked': '12 secs ago'}
{'IP Address': '198.27.74.6', 'Port': '9300', 'Code': 'CA', 'Country': 'Canada', 'Anonymity': 'anonymous', 'Google': '', 'Https': 'yes', 'Last Checked': '12 secs ago'}
{'IP Address': '195.158.30.89', 'Port': '3128', 'Code': 'UZ', 'Country': 'Uzbekistan', 'Anonymity': 'elite proxy', 'Google': 'no', 'Https': 'yes', 'Last Checked': '12 secs ago'}
{'IP Address': '77.247.108.17', 'Port': '33080', 'Code': 'BZ', 'Country': 'Belize', 'Anonymity': 'elite proxy', 'Google': 'no', 'Https': 'yes', 'Last Checked': '12 secs ago'}
{'IP Address': '190.202.3.22', 'Port': '32650', 'Code': 'VE', 'Country': 'Venezuela', 'Anonymity': 'elite proxy', 'Google': '', 'Https': 'yes', 'Last Checked': '12 secs ago'}
{'IP Address': '103.134.44.176', 'Port': '8080', 'Code': 'IN', 'Country': 'India', 'Anonymity': 'elite proxy', 'Google': 'no', 'Https': 'yes', 'Last Checked': '12 secs ago'}
{'IP Address': '202.86.138.18', 'Port': '8080', 'Code': 'MO', 'Country': 'Macau', 'Anonymity': 'elite proxy', 'Google': 'no', 'Https': 'yes', 'Last Checked': '12 secs ago'}
{'IP Address': '8.219.97.248', 'Port': '80', 'Code': 'SG', 'Country': 'Singapore', 'Anonymity': 'anonymous', 'Google': 'no', 'Https': 'yes', 'Last Checked': '12 secs ago'}
Here's the code I was using initially without changing any IP address.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
driver.get("https://tuludictionary.in/dictionary/cgi-bin/web/frame.html")
driver.maximize_window()
But since this is a long process, the program gets pretty slow after some time. I would want to rotate the IP address, say every 10/15 minutes, and restart the driver for that. (Also, What header options should I add which might make it faster?)
I saw this answer here, but I am not sure if I could utilise it in my code.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from fake_useragent import UserAgent
from selenium.webdriver.support.ui import WebDriverWait
from webdriver_manager.chrome import ChromeDriverManager
WAIT = 10
srv=Service(ChromeDriverManager().install())
ua = UserAgent()
userAgent = ua.random
options = Options()
options.add_argument('--headless')
options.add_experimental_option ('excludeSwitches', ['enable-logging'])
options.add_argument("start-maximized")
options.add_argument('window-size=1920x1080')
options.add_argument('--no-sandbox')
options.add_argument('--disable-gpu')
options.add_argument(f'user-agent={userAgent}')
driver = webdriver.Chrome (service=srv, options=options)
waitWebDriver = WebDriverWait (driver, 10)
link = "https://whatismyipaddress.com/"
driver.get(link)
I found a code here that apparently would work if I used the Firefox web driver, how can I update this for Chrome?
proxy = "124.240.187.80:82"
webdriver.DesiredCapabilities.FIREFOX['proxy'] = {
"httpProxy":proxy,
"ftpProxy":proxy,
"sslProxy":proxy,
"noProxy":None,
"proxyType":"MANUAL",
"class":"org.openqa.selenium.Proxy",
"autodetect":False
}
I want to randomly choose one of the Proxy IP Addresses and its port stored as a list of dictionaries fetched from the site every 15 minutes and then restart the driver with those settings.