First HTTPS request takes much more time than the rest

Question

I'm making a python app that needs to make fast HTTP API calls. I use the python requests package to make API calls on HTTPS. I noticed that first request always takes much more time than the rest. This is not due to establishing the HTTP connection because requests.get() establishes a new connection on every call (checked requests with logs).

I investigated my network with Wireshark and although I found that the first request takes more time than the others, the difference is not big.

I also noticed that even if I make the first call to host1 and all the rest to host2, the request to host1 takes much more time than all subsequent requests. That assures me that the issue is not connected to establishing some kind of connection with the host.

I checked with the ping command and didn't find such a difference between the first and subsequent ping requests.

Is it something to do with the OS, requests initial set up on the first request or something else?

My python code:

import requests
import time
import logging


logging.basicConfig(level=logging.DEBUG)

url = "https://bittrex.com/api/v1.1/public/getorderbook?market=BTC-LTC&type=both"

for i in range(10):
    start = time.time()

    requests.get(url)

    end = time.time()

    print('Time: {}. index: {}'.format(end - start, i))

Output:

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): bittrex.com
DEBUG:urllib3.connectionpool:https://bittrex.com:443 "GET /api/v1.1/public/getorderbook?market=BTC-LTC&type=both HTTP/1.1" 200 46568
Time: 0.8292889595031738. index: 0
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): bittrex.com
DEBUG:urllib3.connectionpool:https://bittrex.com:443 "GET /api/v1.1/public/getorderbook?market=BTC-LTC&type=both HTTP/1.1" 200 46568
Time: 0.14321112632751465. index: 1
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): bittrex.com
DEBUG:urllib3.connectionpool:https://bittrex.com:443 "GET /api/v1.1/public/getorderbook?market=BTC-LTC&type=both HTTP/1.1" 200 46568
Time: 0.10214948654174805. index: 2
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): bittrex.com
DEBUG:urllib3.connectionpool:https://bittrex.com:443 "GET /api/v1.1/public/getorderbook?market=BTC-LTC&type=both HTTP/1.1" 200 46568
Time: 0.10616683959960938. index: 3
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): bittrex.com
DEBUG:urllib3.connectionpool:https://bittrex.com:443 "GET /api/v1.1/public/getorderbook?market=BTC-LTC&type=both HTTP/1.1" 200 46568
Time: 0.1061558723449707. index: 4
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): bittrex.com
DEBUG:urllib3.connectionpool:https://bittrex.com:443 "GET /api/v1.1/public/getorderbook?market=BTC-LTC&type=both HTTP/1.1" 200 46568
Time: 0.09714269638061523. index: 5
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): bittrex.com
DEBUG:urllib3.connectionpool:https://bittrex.com:443 "GET /api/v1.1/public/getorderbook?market=BTC-LTC&type=both HTTP/1.1" 200 46568
Time: 0.0861358642578125. index: 6
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): bittrex.com
DEBUG:urllib3.connectionpool:https://bittrex.com:443 "GET /api/v1.1/public/getorderbook?market=BTC-LTC&type=both HTTP/1.1" 200 46568
Time: 0.08713865280151367. index: 7
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): bittrex.com
DEBUG:urllib3.connectionpool:https://bittrex.com:443 "GET /api/v1.1/public/getorderbook?market=BTC-LTC&type=both HTTP/1.1" 200 46568
Time: 0.09714365005493164. index: 8
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): bittrex.com
DEBUG:urllib3.connectionpool:https://bittrex.com:443 "GET /api/v1.1/public/getorderbook?market=BTC-LTC&type=both HTTP/1.1" 200 46568
Time: 0.09714889526367188. index: 9

How can we know without your specific code or something to test? — roganjosh, Aug 20 '17 at 15:31

score 1 · Answer 1 · answered Aug 20 '17 at 16:18

See:
Python requests module is very slow on specific machine
Why is Python 3 http.client so much faster than python-requests?
requests: how to disable / bypass proxy

For some the issue seems to be around proxies.
This change to your code speeded things up dramatically for me.

import requests
import time
import logging
logging.basicConfig(level=logging.DEBUG)
url = "https://bittrex.com/api/v1.1/public/getorderbook?market=BTC-LTC&type=both"
session = requests.Session()
session.trust_env = False

for i in range(10):
    start = time.time()

    session.get(url)

    end = time.time()

    print('Time: {}. index: {}'.format(end - start, i))

From requests: how to disable / bypass proxy
Written by @Lukas Graf
"The only way I'm currently aware of for disabling proxies entirely is the following:

Create a session
Set session.trust_env to False
Create your request using that session"

Your mileage may differ

score 1 · Accepted Answer · edited May 31 '19 at 20:33

I have the same problem too. The first request takes 5s, and others is fast. I tried to update requests and python, problem still exist. Then I check server's load and network connections, nothing wrong. I tried this code:

logging.basicConfig()   
logging.getLogger().setLevel(logging.DEBUG)  
requests_log = logging.getLogger("requests.packages.urllib3")  
requests_log.setLevel(logging.DEBUG)  
requests_log.propagate = True

It shows that the first DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1) last for 5s.

I also tried use session instead of get, no improvement.

At last I add the domain to /etc/hosts, the problem was solved.

So I think the key is DNS.

First HTTPS request takes much more time than the rest

2 Answers2