0

This sends around 5 requests per second to the server. I need to send around 40 requests per second. The server does not limit my requests (I have run 10 instances of this Python script and it has worked) and my internet does not limit the requests.

It's my code which limits my requests per second.

Is it possible to make my Python script send more requests?

JamesRicky
  • 201
  • 1
  • 3
  • 17
  • If your code has no errors, you should post your code on Code Review instead of Stack Overflow. People over there are more than happy to help you figure out more efficient ways to write your code. – Erik Apr 18 '17 at 07:43
  • @erik This is really asking for a different technique, not for a code cleanup. You'll likely need something like async or threaded requests here; that's not really what CR is for. – deceze Apr 18 '17 at 07:45
  • have you looked into multithreading? check out the `threading` package in python2.7 – RandomDude Apr 18 '17 at 07:47

3 Answers3

2

You can/should use twisted. treq based on twisted makes it very easy.since its completely async no of requests/sec will be very high, like 1000 or even more.

from treq import get
from twisted.internet import reactor


# This method is called whenever a response is recieved
def done(response):
   if response.code == 200:
       # do something with response
   reactor.stop()

for i in range(100000):
    get("http://www.somesite.com").addCallback(done)

reactor.run()

learning curve of async programming using twisted is high but at the end you get very good results. for further references look http://treq.readthedocs.io/en/latest/api.html

anekix
  • 2,393
  • 2
  • 30
  • 57
  • Is it also possible to limit requests/sec based on my needs? How would I implement it with my current code? – JamesRicky Apr 18 '17 at 07:50
  • you can see why its useful here http://blog.mailgun.com/stress-testing-http-with-twisted-python-and-treq/ – anekix Apr 18 '17 at 07:54
  • @JamesRicky how do you limit using `requests` module? – anekix Apr 18 '17 at 07:59
  • well as of right now, my code has no limit for requests/sec, but it should be possible to limit requests/sec in a way (less threads ect...). – JamesRicky Apr 18 '17 at 08:02
  • how would you go about limiting in requests you follow the same logic with twisted. also i don t think fixing a particular number is easy either with `twisted` or `requests` rather you can have workarounds for setting max upper bound. – anekix Apr 18 '17 at 08:10
  • you are absolutely right. I read the article you sent. How would I use treq to implement it in my code where it shows response when status code == 200? – JamesRicky Apr 18 '17 at 08:13
  • in my original code I have an array which adds each each "Password" in the array, so it does not have duplicates/skip any password. How would I do that. – JamesRicky Apr 18 '17 at 08:56
  • Maybe I'm pushing to the limit - but I'm getting consistently - memory error. Exploring how to prevent that - while allowing it to go at max speed. – Hangster Nov 03 '18 at 23:26
0

I agree with @anekix, using twisted is the best way to do it.

You need to know one thing, which ever method you use to 10000 HTTP request, there is open file descriptor limit, which is basically set to 1000 in Linux. What this mean is that you can have only 1000 concurrent TCP connections. However you can increase this from configurations /etc/security/limits.conf in Linux

Sumeet P
  • 144
  • 1
  • 5
-1

Why don't you use Scapy package for traffic generation with sendp (Layer 2), it has a really good performance. It is also very easy to generate different packets

sendp(Ether()/IP(dst="1.2.3.4",ttl=(1,4)), iface="eth1")
Meir Tseitlin
  • 1,878
  • 2
  • 17
  • 28