0

I am doing some web scrapping.

I want to rotate IP's in order to get away with blocking. My scraper is ready. I just need pool/list of dedicated IP's to make my scraper work. I already bought pro subscription of WindScribe. Till now I cannot buy dedicated IP's as I only need for testing

I want a solution through which I can get pool/list of IPs from my VPN(as it has multiple countries servers)

Or a way through which my VPN automatically gets switch on and off only giving me IP without connecting with whole device(changing devices IP)

PS: I have already tried doing it with OS library. It changes IP through out the device causing code to stuck and create exceptions.

Or is there any way I can get IP's for free(for now to test my scraper)

NOTE:

  • I am using Beautiful Soup, Request and CfScrape to download HTML and parse.
  • I want to make around 1000+ requests in order for my task to be successful
    • rafay-mahmood
    • 21
    • 3

    1 Answers1

    0

    I have a few caveats: I am not entirely clear what you are asking, and I am not familiar with WindScribe, and I haven't tried this myself. However, I believe it is possible to control the routing for a connection so it comes from one in a set of IP addresses.

    I am most familiar with Windows, however, I understand it is easier to make these type of configuration changes on linux.

    It is possible to open multiple VPNs at once. In Windows I typically have several VPNs configured and regularly connect to multiple networks at the same time. When you connect to a VPN your computer will be assigned an interface to interact with that network. In Windows if you run ipconfig /all you will see something like:

    PPP adapter xxx:

   Connection-specific DNS Suffix  . : xxx.xxx
   Description . . . . . . . . . . . : xxx
   Physical Address. . . . . . . . . :
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 172.20.64.19(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.255
   Default Gateway . . . . . . . . . :
   DNS Servers . . . . . . . . . . . : 172.20.1.1
   NetBIOS over Tcpip. . . . . . . . : Enabled

    In this case the interface I used to connect to the remote networks is 172.20.64.19.

    Typically with the default Windows VPN, if you want your connections to go out through that VPN you will chose the Use default gateway on remote network in the VPN configuration settings. This makes a change to your routing table which will give that VPN the highest metric. You can see the routing table and metric by typing the command route print on a command line.

    It is possible to manually update your routing table using the route add command to define specific routing for connections from different interfaces. This article talks about how to do that. So, as I understand it, with multiple VPNs open, depending on which local interface you bind to, the outgoing connection will be routed differently.

    This article suggests that there's a method which will allow you to tell the requests library to bind to a specific local interface.

    If you combine these, you should be able to open multiple VPNs, update your routing table and then python should be able to go out through whichever VPN you wish, or indeed multiple VPNs at once from the same process (although through different socket connections).

    With all that said, I must say while I do fiddle with my routing table occasionally, I understand it is possible to make a mess of your routing table which is difficult to fix, and it's something lots of people recommend against unless you really know what you're doing. You also need to make sure the private address ranges of the different VPNs do not clash, but I assume WindScribe should give you some configuration options here.

    John M.
    • 775
    • 4
    • 16