sending more than1000 requests per second in python

Question

I am trying to send requests concurrently to a server and then record the average latency using this code:

import Queue
import time
import threading
import urllib2

data = "{"image_1":"abc/xyz.jpg"}"
headers = {.....}
def get_url(q, url):
    num = 1
    sum = 0
    while num <= 200:
        start = time.time()
        req = urllib2.Request(url, data, headers)
        response = urllib2.urlopen(req)
        end = time.time()
        print end - start
        num = num + 1
        q.put(response.read())
        sum = sum + (end - start)
    print sum


theurls = ["http://example.com/example"]
q = Queue.Queue()

for u in theurls:
    t = threading.Thread(target = get_url, args = (q, u))
    t.daemon = True
    t.start()

while True:
    s = q.get()
    print s

This code is working just fine, but now I intend to send more than 1000 requests per second. I came across this answer but I am not sure how do I use grequests for my case. Some insights will be very helpful.

Thanks

May be you can take a look to an asynchronous framework like [aiohttp](http://aiohttp.readthedocs.io/en/stable/) on the top of python built-in asyncio ? There is some good posts here or on the web (like https://compiletoi.net/fast-scraping-in-python-with-asyncio/ or http://aosabook.org/en/500L/a-web-crawler-with-asyncio-coroutines.html or https://magic.io/blog/uvloop-blazing-fast-python-networking/) about using aiohttp to make several (almost concurrent) requests. — mgc, Aug 14 '16 at 11:39
Check out the repository for some examples. https://github.com/kennethreitz/grequests — 0xcaff, Nov 28 '16 at 22:22
it may be irrelevant to you after 4 years but this code does not send 200 connections per second. i opened 200 threads to get the job done — Hesham Yassin, Mar 29 '17 at 06:35

score 3 · Answer 1 · answered Nov 28 '16 at 22:37

The documentation isn't great but the source is. Read the Source! Check out the first few lines of grequests.py on github:

"""
grequests
~~~~~~~~~
This module contains an asynchronous replica of ``requests.api``, powered
by gevent. All API methods return a ``Request`` instance (as opposed to
``Response``). A list of requests can be sent with ``map()``.
"""

The package exports the following:

__all__ = (
    'map', 'imap',
    'get', 'options', 'head', 'post', 'put', 'patch', 'delete', 'request'
)

Those symbols are defined further down the file:

# Shortcuts for creating AsyncRequest with appropriate HTTP method
get = partial(AsyncRequest, 'GET')
options = partial(AsyncRequest, 'OPTIONS')
head = partial(AsyncRequest, 'HEAD')
post = partial(AsyncRequest, 'POST')
put = partial(AsyncRequest, 'PUT')
patch = partial(AsyncRequest, 'PATCH')
delete = partial(AsyncRequest, 'DELETE')

partial was imported from functools at the top of the file.

from functools import partial

The documentation for functool.partial says:

Return a new partial object which when called will behave like func called with the positional arguments args and keyword arguments keywords. If more arguments are supplied to the call, they are appended to args. If additional keyword arguments are supplied, they extend and override keywords.

Basically calling grequests.get calls AsyncRequest("GET", **other args go here**). AsyncRequest is a function which creates a new AsyncRequest. Its docs say:

""" Asynchronous request.
Accept same parameters as ``Session.request`` and some additional:
:param session: Session which will do request
:param callback: Callback called on response.
                 Same as passing ``hooks={'response': callback}``
"""

Session is defined earlier:

from requests import Session

Here's a guide on using requests sessions.

sending more than1000 requests per second in python

1 Answers1