1

I need to do stress test for my API. I modified that example to send POST instead of HEAD, also I put some data to my request:

count = int(sys.argv[1])
url = "http://api.example.com/"
test_id = datetime.now().strftime("%H%M%S%f")
data = []
for i in range(count):
    req_data = {
        'test_id': test_id,
        'param1': i,
        'param2': 'aaa',
        'param3': 'bbb',
    }
    data.append(req_data)

start_time = time.time()
res = grequests.map(grequests.post(url, data=data[i]) for i in range(count))
sending_time = time.time() - start_time
print(sending_time)

After that performance dropped to 5 request per second! Also I tried all answers from that question and received the same result. Is it real to reached 1000 req/sec?

Community
  • 1
  • 1
Sawl Stone
  • 11
  • 1
  • 3
  • Well of course, POST is going to require much more processing in your web server than HEAD. How well does the web server handle concurrent requests? Can you configure it to use more workers? Is it *using* workers? By switching to POST, the scope of your question must now include the processing in the server as well as in your client. – PaulMcG May 19 '17 at 09:39
  • I'am using Twisted on server-side – Sawl Stone May 19 '17 at 10:26
  • Shift your attention to getting the server to handle 1000 req/second. The client code is not your bottleneck. Just using Twisted is not a magic wand. You will need to break up your handler into multiple Deferred's so that you better overlap processing and I/O from concurrent requests. – PaulMcG May 19 '17 at 11:15
  • It sounds like you may have discovered a bottleneck in your server. Back track through the server code and see where the slow down occurs and make adjustments. Look for things like `deferToThread` or any kind of deferred that takes a while to complete. – notorious.no May 19 '17 at 11:16

1 Answers1

0

Yes, you can do this.

You can achieve this by creating multiple processes and threads.

Ex. Using subprocess

Here, we are making 10 subprocesses and each will make 100 requests. So in a second, you will able to execute 1000 requests.

import subprocess

for i in range(10):
    subprocess.Popen(["python","pathToPythonScript/stressTesting.py","100"])
Ujjaval Moradiya
  • 222
  • 1
  • 12
  • Can anyone explain me what is the issue with this answer? Write down the comment here so that I can increase my knowledge. – Ujjaval Moradiya May 20 '17 at 04:49
  • It's not because 10 subprocesses each make 100 requests that they will get done in under a second. Also, the initial question is about grequests and possibly asyncio, which are going to be much faster than this. – Quentin Pradet May 20 '17 at 07:36
  • I don't know much about asyncio, but with this you can reach near to 100. And if you don't have memory or processing power issue you can increase the counter from 10 to 15 and you will get 1000 requests. I know this is work around of that problem and not the proper solution. But yes thanks for introducing me asyncio. – Ujjaval Moradiya May 20 '17 at 10:27