0

I'm writing a python script that will send http requests concurrently to the urls mentioned in a file using python. The script works fine for a single IP address. The OS I'm using is linux. I've generated virtual IP addresses like eth0:1,eth0:2 etc. I want to send HTTP requests using these virtual IP addresses along with the eth0 IP address concurrently. I use the requests module for http requests and threading module for concurrent requests. Kindly help me. I'm trying to develop a web testing tool.

Naveen
  • 95
  • 3
  • 7
  • Could you clarify? You want to sent HTTP requests "from" the various IP addresses, in order to see if the responses differ based on IP? Requests seems unlikely to help in that regard. – selllikesybok Sep 26 '14 at 04:37
  • I want to send HTTP requests from a various IP addresses just to generate web traffic. There are many web testing tools (like spirent) which support this feature but I want to develop one on my own using python. Is there any way to send HTTP requests using multiple Ips on python ? – Naveen Sep 26 '14 at 04:52
  • can you help me with the code ? I'm a beginner in python – Naveen Sep 26 '14 at 04:56

1 Answers1

0

I think you wanted to avoid "Crawl Delay" and do faster crawl on one server!

In this case, Remote Web Server will recognize request from only one IP!!

I think using parallel + curl + python script is more simple and best way.

or use https://pypi.python.org/pypi/pyparallelcurl/0.0.4

or use a lot of servers.

and refer to https://code.google.com/p/httplib2/issues/detail?id=91

han058
  • 908
  • 8
  • 19
  • can you help me with the code ? I'm a beginner in python. – Naveen Sep 26 '14 at 04:55
  • I see... You wanted to make stress tester by python. then... http://www.pylot.org/ or other tool. There is many python tools for test web perfomance. – han058 Sep 26 '14 at 05:00
  • Does is support multiple IP addresses ? Actually, the next requirement is limiting the bandwidth for a particular IP Address whereas the other IP addresses should work normally. My first motive is to generate web traffic using a range of IP address (say 10.91.55.1-10.91.55.10).. – Naveen Sep 26 '14 at 05:28
  • Please refer to http://superuser.com/questions/540373/how-to-assign-multiple-ip-addresses-to-a-single-computer – han058 Sep 26 '14 at 06:19
  • I think I did not explain my question properly. – Naveen Sep 26 '14 at 07:06
  • The question is :I have two IPs on my linux system. one is 10.91.56.2 and the other one is 10.91.56.3. I have configured a website on my server. I now want to send HTTP requests to this server using both the IP addresses. Is there a way to do this ? HOpe I have explained my question properly – Naveen Sep 26 '14 at 07:08
  • Are there two network devices on your server? – han058 Sep 26 '14 at 07:17
  • Ok I see! There is one server. And you wanted to request from local to local. So you said "limiting the bandwidth for a particular IP Address whereas the other IP addresses should work normally" Auctually... It's meaningless. anyway, you can use `curl` with option like `--interface IP` – han058 Sep 26 '14 at 07:29
  • sorry. I guess I still didn't explain my question properly. I have two virtual machines. one with ip address 10.91.55.2 which acts as the server. Another VM as a client with physical address 10.91.56.2 and virtual address 10.91.56.3 and so on. I now want to write a script on the client that will involve the virtual ip addresses also to send the HTTP requests. Hope I'm clear this time. Sorry, for making it too complicated.. – Naveen Sep 26 '14 at 08:02
  • I'm sorry I could not understand well.please refer to this page. http://bugs.python.org/msg73840 and I really hope this comment is helpful. – han058 Sep 26 '14 at 08:09
  • Yes, I want to achieve the same thing mentioned in the link you sent. Is there any way on python to do the same ? – Naveen Sep 26 '14 at 08:17
  • I'm not sure but Try to use pycurl! ( or subprocess.Popen("curl...") ) `import pycurl; c = pycurl.Curl(); c.setopt(pycurl.URL, "http://www.python.org/"); c.setopt(pycurl.CURLOPT_INTERFACE, "eth0"); c.perform();` or see the link http://stackoverflow.com/questions/1150332/source-interface-with-python-and-urllib2 – han058 Sep 26 '14 at 08:28
  • Thank you :) I will try pycurl.. HOpe it'll work out :) – Naveen Sep 26 '14 at 08:30
  • And Thank you for select my pooooooor answer! – han058 Sep 26 '14 at 08:32
  • It shows an error :( c.setopt(pycurl.CURLOPT_INTERFACE, "eth0") Traceback (most recent call last): File "", line 1, in AttributeError: 'module' object has no attribute 'CURLOPT_INTERFACE' – Naveen Sep 26 '14 at 08:33
  • Sorry I have no time now.... Could you read the page http://stackoverflow.com/questions/1150332/source-interface-with-python-and-urllib2 ? I think it's a good solution! – han058 Sep 26 '14 at 09:03