1

I am trying to build a selenium scraper that rotate its IP address after every n request, for this i need to access a free proxy website in order to construct a list of IP:port addresses, the problem is that i am executing my code from a country that bans proxy websites, thus the code can't access the proxy website and is returning me this error: raise URLError(err) urllib.error.URLError: urlopen error [Errno 104] Connection reset by peer

Here is my part of code that try to access https://www.sslproxies.org/ website:

ua = UserAgent() # From here we generate a random user agent
proxies = [] # Will contain proxies [ip, port]
proxies_req = Request('https://www.sslproxies.org/')
proxies_req.add_header('User-Agent', ua.random)
proxies_doc = urlopen(proxies_req).read().decode('utf8')
soup = BeautifulSoup(proxies_doc, 'html.parser')
proxies_table = soup.find(id='proxylisttable')

The error is occuring from the fifth line :

proxies_doc = urlopen(proxies_req).read().decode('utf8')

any suggestions?

Youcef
  • 1,103
  • 2
  • 11
  • 26
  • ...if the site is blocked you're either going to have to find a way around the block (and accept the risks of doing that) or just accept that it won't work. Are you able to load that site using other tools, e.g. a regular web browser? – ChrisGPT was on strike Sep 15 '18 at 14:07
  • with regular browser (chrome), i can open it manually using Ultrasurf chrome extension – Youcef Sep 15 '18 at 14:08
  • what you mean by around the block? – Youcef Sep 15 '18 at 14:12
  • Ultrasurf also uses a proxy web server. Do you know why Ultrasurf's proxy works? – bunbun Sep 15 '18 at 14:44
  • @bunbun because it is an extension not a website( the country blocks the websites not extenstions), am thinking of using the chrome browser with the as it is with the ultrasurf extension, i have just to close the browser : browser.quit() and open it again to get the IP changed, what you think? – Youcef Sep 15 '18 at 15:15
  • yep you can write a bash script to open and close the browser i guess. Not sure if selenium allows loading of plug-ins in chrome. If you still want to go the selenium way I can help you find out if thats possible, or if you need help with bash scripting – bunbun Sep 15 '18 at 15:27
  • `https://stackoverflow.com/questions/34222412/load-chrome-extension-using-selenium` – bunbun Sep 15 '18 at 15:29
  • Why use the bash script since the selenium command : browser.quit() does the job? – Youcef Sep 15 '18 at 15:29
  • ah i thought you wanted to open chrome with the ultrasurf extension manually, mb. yeah seems like you already know about selenium extension loading so go ahead with that – bunbun Sep 15 '18 at 15:31

0 Answers0