MacOS High Sierra 10.13.6 -> Extremely Slow Requests Library

Question

In python, when I try to use the requests library or any other library that requires connecting to the internet, the response comes extremely slow.

Code as simple as the following can take 2 to 5 minutes.

import requests
requests.get('https://www.google.com')

The same code finishes within 1 second on my Linux machine.

If I use non-python method to achieve the same thing, i.e. curl

curl 'https://www.google.com'

The response comes in less than 1 second.

What may be causing this? Where should I be looking? I don't know much about networking or Macs in general, so can someone help me debug with detailed instructions?

Answer 1

Surprisingly, by default requests uses infinite timeout.

Try setting different connect and read timeouts and see what is the exception.

r = requests.get('https://github.com', timeout=(10, 30))

You might want to print verbose logs to see what's going on:

logging.basicConfig() 
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

See for the reference and context:

• http://docs.python-requests.org/en/master/user/advanced/#timeouts
• Why doesn't requests.get() return? What is the default timeout that requests.get() uses?
• https://github.com/requests/requests/issues/3070