Why does a request via Python `requests` takes almost seven times longer than in Postman?

Question

I am currently doing a HTTP Post request to a server in Python 2.7 with requests.post() which takes around 700ms. There is also absolutely no proxy server which could cause delays but still I am bypassing any proxies because it seems to be an issue of that library.

Nevertheless I was curious about that time because in my opinion it takes very long for an answer with about 230 characters. That is why I tried the same request in Postman. The result was that the request in Postman took less than 100ms! Which is much more appropriate for the task I have to do. It is all about time.

I want to know if there is any specific parameter in requests.post() which I have to set or is this function just that slow?

The request looks like this currently (very basic stuff):

req = requests.post(url, json={"Username": username, "Password": password, "TerminalNo": terminalno)}) 
json = req.json()

Header from the server if needed:

cache-control →private
content-length →228
content-type →application/json; charset=utf-8
date →Mon, 30 Jul 2018 17:58:05 GMT
server →Microsoft-IIS/7.5
x-aspnet-version →2.0.50727
x-powered-by →ASP.NET

In Python I got that time from `req.elapsed.total_seconds()`. In Postman it is shown with the result. — anton68, Jul 30 '18 at 19:13

Shariful Islam Mubin · Answer 1 · 2021-01-08T14:22:19.587

Don't know why this important question didn't answer yet.

Anyway, maybe the reason is: By default requests.get(...) or, requests.post(...) don't use sessions.
Because of this reason, every time when we call these methods, every time it reconfigures the connection and all other networking stuff.
That's why every time it took much and same time as the previous requests took.

To overcome this problem, we can use session.

import requests

url_call = "https://en.wikipedia.org/wiki/Jamal_Nazrul_Islam"
session = requests.Session()
session.get(url_call)

By using, session, it will take much time for the first call. From the second call, on the same domain, it will take much less time.

My experiment:

First call: 2154 ms
Second call: 318 ms
Third call: 124 ms
Fourth call: 125 ms

score 1 · Answer 2 · answered Jul 25 '21 at 15:28

Try using http.client

import http.client
import json

conn = http.client.HTTPSConnection("example.com")
payload = {'Username': 'username', 'Password': 'password', 'TerminalNo': 'terminalno'}
body = json.dumps(payload)
headers = {
  'authorization': 'XXXXXX-XXXX-XXXX-XXXX-XXXXXXXXX',
  'x-api-key': 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
}

for i in range(10):
    start = time.time()
    conn.request("GET", "/", body, headers)
    res = conn.getresponse()
    data = res.read()
    response_time = time.time() - start
    print('{} {} ms'.format(i+1, round(response_time * 1000)))

Result

1 906 ms
2 215 ms
3 193 ms
4 219 ms
5 203 ms
6 223 ms
7 210 ms
8 282 ms
9 204 ms
10 204 ms

score 0 · Answer 3 · answered Jun 26 '20 at 22:46

This could be due to the server's response being incorrectly formatted, leading to parsing problems.

You can check this by not reading the response you receive from the server. If the code is still slow, this is not your problem, but if this fixed it, the problem might lie with parsing the response.

In case some headers are set incorrectly, this can lead to parsing errors which prevents chunked transfer (source).
In other cases, setting the encoding manually might resolve parsing problems (source).

To fix those issues, try:

req = requests.post(url, json={"Username": username, "Password": password, "TerminalNo": terminalno)}) 
r.raw.chunked = True # Fix issue 1
r.encoding = 'utf-8' # Fix issue 2
json = req.json()

Didn't solve your issue?

If that did not solve your issue, I have collected some other possible solutions here.

Why does a request via Python `requests` takes almost seven times longer than in Postman?

3 Answers3

Didn't solve your issue?

Linked