I am attempting to web scrape in python using selenium with a proxy. Here is the code I am using:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import chromedriver_autoinstaller
chromedriver_autoinstaller.install()
PROXY = "127.0.0.1:9009" # not the actual proxy I am using
chrome_options = Options()
chrome_options.add_argument('--proxy-server=%s' % PROXY)
driver = webdriver.Chrome(options=chrome_options)
driver.get("http://example.org")
driver.save_screenshot('/home/pathtocwd/screenshot.png')
print(driver.title)
driver.quit()
The problem is that .get simply doesn't do anything. It doesn't give any errors, but it doesn't seem to fetch the website information either. I took a screenshot in an attempt to debug, but the image is completely blank (all white). Furthermore, the output of print(driver.title) is simply an empty line.
The proxies I am using are paid for and I have tried several of them, all experiencing the same issue. I am able to get information without using a proxy, so this implies to me that I am configuring the proxy incorrectly. I also tried following the approach by user Zyy in this stack exchange post; however, I experienced the exact same issue.
Could the only answer be that I bought some junk proxy addresses? I used the service webshare.io as I heard good things about it. I am also inexperienced in the realm of proxy usage, so I am unsure if I am missing something critical here.
