3

Consider the following case: There is a slow server which use about 200ms to handle a request (no including the network transfer time). And now, we need to send a bunch of requests every second.

After read this post, I have tried multi-thread, multi-process, twisted (agent.request) and eventlet. But the biggest speedup is only 6x, which is achieved via twisted and eventlet, both are using epoll.

The following code shows the test version with eventlet,

import eventlet
eventlet.monkey_patch(all=False, socket=True)

import requests

def send():
    pile = eventlet.GreenPile(30)
    for i in range(1000):
        pile.spawn(requests.get, 'https://api.???.com/', timeout=1)
    for response in pile:
        if response:
            print response.elapsed, response.text

Anyone could help me to make it clear why the speedup is so low? And is there any other mechanism could make it much faster?

Community
  • 1
  • 1
Jacky1205
  • 3,273
  • 3
  • 22
  • 44
  • How long does it currently take? – Martin Thoma Feb 04 '15 at 14:21
  • 1
    see if this helps you: http://stackoverflow.com/questions/2632520/what-is-the-fastest-way-to-send-100-000-http-requests-in-python – Urban48 Feb 04 '15 at 14:25
  • 1
    1000 requests at 200ms per requests = 3,3 minutes. I don't understand why you want to overload your server. Why 1000 ? why not 1M ? – Jérôme Radix Feb 04 '15 at 14:29
  • @JérômeRadix, In fact, my app is both a server and client, which will send some kinds of requests to the back-end server. Currently, 1000 is okay, 1M is also possible in future. – Jacky1205 Feb 04 '15 at 14:36
  • @Jacky You should post a job if it is challenge bro. SO is for clearing the doubts – Md. Mohsin Jul 29 '15 at 19:48

1 Answers1

7

I know this is an old post but someone might still need this.

If you want to do load testing but want to use python then you should use a tool like locust: http://locust.io/

Here is my solution which resulted in 10,000 requests in 10 seconds:

Needed Package: sudo pip install grequests

Code:

import grequests
import time

start_time = time.time()
# Create a 10000 requests
urls = ['http://www.google.co.il']*10000
rs = (grequests.head(u) for u in urls)

# Send them.
grequests.map(rs)

print time.time() - start_time # Result was: 9.66666889191
Eran Yogev
  • 891
  • 10
  • 20
  • I run the code on my laptop and got the result after 69.997563 seconds, and for 1k after 23.694893 (CPU: i7 10gen 3GHz, RAM: 16Gb, 256Gb SSD, GPU Nvidia MX230 2Gb) – Walid Bousseta Sep 12 '21 at 00:43