2

I do webscrapring with Scrapy, using Polipo as proxy, and Tor as network. I know my proxy makes rotating IP, but the IP location is most of the time out of my country. On the websites I scrape, it could have some blocking considering the location of the IP. Then, how can I keep the rotating IP rule and to limit the location of the IP used?

Scrapy version: 1.5.0, Python version: 2.7.9, Tor version: 0.3.4.8, Vidalia: 0.2.21

AvyWam
  • 890
  • 8
  • 28

1 Answers1

2

Most probably you know this but the final ip which the website you are scraping it will see the ip of the exit node. As such you can control the country of the exit node using configuration

You can run multiple tor set up and mix and match or rotate tor service across your request set.

Biswanath
  • 9,075
  • 12
  • 44
  • 58