0

I have a python script, which scrapes some information from some site. This site has a daily limitation for 20 connections So, I decided to use module requests with specified "proxies".

After a couple of hours testing different "proxy-list" sites I've found one and I've been parsing from site http://free-proxy-list.net/.

Seems, this site doesn't get the list updated often and after testing my script I've wasted all the proxies and I can't access the site anymore.

All these searches make me exhausted and I feel like my script completely sucks.

Is there any way I can avoid "detecting" me by site or I just need to find another list of proxies? If there are some sites with daily updated and all new list of proxies - please, let me know.

P.S. I often have stumbled upon sites like https://hide.me where I just enter the link and it gives me full access. Maybe I can just code this in Python? If it's possible - show me, please, how.

GriMel
  • 2,272
  • 5
  • 22
  • 40
  • How does the 'daily limit for 20 connections' work? Can't you request multiple pages over the same connection? – Colonel Thirty Two Oct 07 '15 at 19:54
  • This is a ranking site. List of teams, points. Etc. When I open any team's link I go to the team's page with results. So, every time, I open it by requests.get(team_page, proxies={"http":proxy}) I have -1 connection. – GriMel Oct 07 '15 at 20:06
  • You can buy rotating proxies or vpn solutions like hidemyass pro vpn. – Vikas Ojha Oct 08 '15 at 06:52
  • Found a solution, using TOR. http://stackoverflow.com/questions/20304301/how-to-control-tor-when-use-tor-proxy. – GriMel Oct 08 '15 at 22:10

0 Answers0