2

I have been using the python requests framework for pretty much all my projects for many years now and haven't come across a problem like this.

This happens only for the website www.pagedna.com as far as I know. The code sample to reproduce is ridiculously simple

import requests
requests.get("https://www.pagedna.com")

Chrome/Firefox browsers open this site up, no problem, but the snippet above gives me the generic SSL failure exception

Traceback (most recent call last):
  File "inline.py", line 2, in <module>
    requests.get("https://www.pagedna.com")
  File "/home/rishi/workspace/misc/venvs/mc3.4/lib/python3.4/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/home/rishi/workspace/misc/venvs/mc3.4/lib/python3.4/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/rishi/workspace/misc/venvs/mc3.4/lib/python3.4/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/rishi/workspace/misc/venvs/mc3.4/lib/python3.4/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/home/rishi/workspace/misc/venvs/mc3.4/lib/python3.4/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.pagedna.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))

OpenSSL gives me the right information if I use the servername argument

openssl s_client -showcerts -servername www.pagedna.com -connect www.pagedna.com:443

CONNECTED(00000003)
---
Certificate chain
 0 s:/OU=Domain Control Validated/CN=*.pagedna.com
   i:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
-----BEGIN CERTIFICATE-----
MIIGrjCCBZagAwIBAgIJAJdLy96ULpdIMA0GCSqGSIb3DQEBCwUAMIG0MQswCQYD
VQQGEwJVUzEQMA4GA1UECBMHQXJpem9uYTETMBEGA1UEBxMKU2NvdHRzZGFsZTEa
MBgGA1UEChMRR29EYWRkeS5jb20sIEluYy4xLTArBgNVBAsTJGh0dHA6Ly9jZXJ0
cy5nb2RhZGR5LmNvbS9yZXBvc2l0b3J5LzEzMDEGA1UEAxMqR28gRGFkZHkgU2Vj
dXJlIENlcnRpZmljYXRlIEF1dGhvcml0eSAtIEcyMB4XDTE4MDcyNzE4MzUyMloX
DTIwMDkyNTE3MTExMVowOzEhMB8GA1UECxMYRG9tYWluIENvbnRyb2wgVmFsaWRh
dGVkMRYwFAYDVQQDDA0qLnBhZ2VkbmEuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEAt6B/jE5FjTKWdEtNj8oRxzqlk5fGjBfYbsKjdPuSi0t6i0a8
lN9yVYLhLM988s3XXvcU8E9d30aQ51JDjxtkID9VP4z4x8E/fWjGpJZ/Nh1XMrSf
rmIZEXr+SZzUBp2dISIJKgmNNr3vVWS4BmViL7YXxvgZzGiZwQRACkoFdfGkkGcT
h1WOPIYf7nzxyasLvddSN5j6BU4XgiTpdlJSWofXFdMJBFfHoTyLUiH+stMzE4wU
gA+oxKzRqhDHJ+L393i2mv4y2FXiQ0InTqyKRGGuX8zmoPirHCRAmMItk9gmUoi+
/3nDVnUlIG6LOgcXylVbn22W5WXN4iwtZ8TCVwIDAQABo4IDOTCCAzUwDAYDVR0T
AQH/BAIwADAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDgYDVR0PAQH/
BAQDAgWgMDcGA1UdHwQwMC4wLKAqoCiGJmh0dHA6Ly9jcmwuZ29kYWRkeS5jb20v
Z2RpZzJzMS04NTIuY3JsMF0GA1UdIARWMFQwSAYLYIZIAYb9bQEHFwEwOTA3Bggr

etc...

However, it gives me a different certificate if the -servername option is missing, which I suspect is the issue here.

The relevant pip installations are as follows

pyOpenSSL==19.0.0
requests==2.21.0
requests-toolbelt==0.8.0

Using python3.4

Is this a requests framework bug? Should I open a bug report or am I missing something?

Rishi Diwan
  • 338
  • 1
  • 10
  • Have you tried using the [`verify`](https://2.python-requests.org/en/master/user/advanced/#ssl-cert-verification) kwarg? Furthermore, I'd double-check the value of the `CA_BUNDLE` environment variable, which can point requests by default to a specific file/folder where your certificates are – C.Nivs Jun 17 '19 at 13:58
  • So I have ```certifi==2019.6.16``` installed so that should cover the latest certs. It also works with `verify=false`, but that is not the goal I need to achieve in my script. I need the valid SSL based request to work. – Rishi Diwan Jun 17 '19 at 14:01
  • Adding `import os; print(os.environ.get('CA_BUNDLE'))` gives me `None`. Other sites (millions of them) work though. – Rishi Diwan Jun 17 '19 at 14:04
  • True, `certifi` does provide certs from the Mozilla truststore, but if you know what your certificate is already, you can use `verify='/path/to/cert'` to use a local one on your machine. Otherwise, you can `export CA_BUNDLE=/path/to/cert`, which `requests` will leverage by default – C.Nivs Jun 17 '19 at 14:25
  • Yes that's right, but this is unfortunately not a special case script, it needs to work for millions of sites so that solution just won't work. Should I make a bug report on github? – Rishi Diwan Jun 17 '19 at 14:28
  • I'm not sure that it would be a bug on the `requests` side, seems like something maybe a `try/except` might cover until you hit another case – C.Nivs Jun 17 '19 at 14:31
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/195070/discussion-between-rishi-diwan-and-c-nivs). – Rishi Diwan Jun 17 '19 at 14:36

1 Answers1

2

... but the snippet above gives me the generic SSL failure exception ...

 ...'tls_process_server_certificate', 'certificate verify failed')],)",),))

Actually, this is not a generic SSL failure. It clearly fails because it could not verify the certificate. And the reason is not SNI vs. non-SNI since SNI is used by default in requests for a long time.

The reason is instead a misconfiguration of the server. From the SSLLabs report:

This server's certificate chain is incomplete. Grade capped to B.

 

Chrome/Firefox browsers open this site up, no problem ...

Desktop browsers usually work around successfully this type of common misconfiguration, for example by caching intermediate certificates from other connections and using these to fill in missing parts of the certificate chain. But if you try for example a Firefox with a fresh profile you'll get the same kind of SSL problems.

Steffen Ullrich
  • 114,247
  • 10
  • 131
  • 172
  • We found 3 other sites giving a similar issue and this seems to be the problem. Is there a workaround in python-requests (apart from verify=false) – Rishi Diwan Jun 18 '19 at 05:21
  • 1
    @RishiDiwan: There is no automatic workaround. Instead you have to figure out which certificates are missing (SSLLabs report helps with this) and to add the missing chain certificates to your trust store so that OpenSSL knows about these even if the site fails to send them. See also https://stackoverflow.com/a/28667850/3081018. – Steffen Ullrich Jun 18 '19 at 05:24