4

I use python-requests to talk to HTTPS web services, some of which present incomplete certificate X509 chains. I'm having trouble figuring out how to access the invalid/incomplete certificates in order to explain the error to the user.

Here's an example illustrated by https://ssllabs.com/ssltest, where the server sends only the leaf certificate, and not the intermediate certificate which is necessary for validation, but missing from certifi's root CA store:

screenshot

When I try to connect with python-requests, I get an exception that isn't very useful:

request.get('https://web.service.com/path')

SSLError: HTTPSConnectionPool(host='web.service.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))

Obviously, I can use separate tools to figure out what's wrong in any particular case (e.g. gnutls-cli, openssl s_client, SSLLabs, etc.).

However, what I really want to be able to do is to be able to catch and diagnose the problem with the certificate chain in my Python code, so that I can present a more specific error message to the user. This answer suggests a monkey-patch to the response object; it's not particularly elegant, but it works—though only when the response object is returned successfully, and not in the case of an exception.

What are the cleanest ways to instrument requests to save the peer's certificate chain in the exception object returned when requests fails to validate the certificate chain itself?

Dan Lenski
  • 76,929
  • 13
  • 76
  • 124
  • 1
    Could you give us a website to test? There isn't many websites with invalid/incomplete certificates... – Sraw Jul 17 '18 at 04:03
  • I cannot share either of my specific candidates, unfortunately, but _any_ URL with a self-signed cert or mismatched cert will throw a similar exception, or any site addressed by its IPv4 address instead of domain name. (e.g. [https://8.8.8.8](https://8.8.8.8/) for Google). – Dan Lenski Jul 17 '18 at 04:27

1 Answers1

4

Take requests.get("https://151.101.1.69") # stackoverflow's ip as an example:

try:
    requests.get("https://151.101.1.69")
except requests.exceptions.SSLError as e:
    cert = e.args[0].reason.args[0]._peer_cert

Then cert is a dict contains the peer's certificate. As I'm not very familiar with SSL, I don't know if it is enough for your case.

BTW, in this case the error is "hostname '151.101.1.69' doesn't match either of '*.stackexchange.com', ...omitted. I'm not sure about the structure of exception in your real case, so you may need to find it on your own. I think it should have the same name _peer_cert.

update

The above method doesn't work when handshake fails... But it still can be done:

try:
    requests.get("https://fpslinux1.finalphasesystems.com/")
except requests.exceptions.SSLError:
    import ssl
    import OpenSSL
    cert = ssl.get_server_certificate(('fpslinux1.finalphasesystems.com', 443))
    cert = OpenSSL.crypto.load_certificate(OpenSSL.crypto.FILETYPE_PEM, cert)
    print(cert.get_issuer())
    print(cert.get_subject().get_components())

Yes it is a little dirty but I don't have a better method as a ssl socket doesn't even return invalid certs from C level :/

To use OpenSSL, you need to install pyopenssl.

Sraw
  • 18,892
  • 11
  • 54
  • 87
  • 1
    Aha, thanks, this is very helpful. Unfortunately, in the case of an incomplete cert chain (or a self-signed cert, e.g. `fpslinux1.finalphasesystems.com`), the exception thrown is a [`SSLError`](https://docs.python.org/2/library/ssl.html#ssl.SSLError) rather than a [`CertificateError`](https://docs.python.org/2/library/ssl.html#ssl.CertificateError), and it lacks the requisite `_peer_cert` attribute. This inconsistency is quite frustrating: they're **both** errors in validating the certificate, right? – Dan Lenski Jul 17 '18 at 05:12
  • I verified the same (frustrating) behavior with Python 2.7 and Python 3.5… – Dan Lenski Jul 17 '18 at 05:44
  • 1
    @DanLenski Updated... Although not a pythonical way. – Sraw Jul 17 '18 at 05:59
  • Thanks, @sraw. This is very useful. It's somewhat frustrating that the second version requires initiating a second TLS connection, which raises race condition/reproducibility issues. Do you understand why the **handshake fails** when a cert chain can't be verified… but the failure occurs **post-handshake** when a cert has a mismatched hostname? – Dan Lenski Jul 17 '18 at 18:28
  • 1
    @DanLenski I am not familiar with this part. I just guess these are two different situations. One is we don't have public key to validate, another is validate succeeds but certificate mismatches. Anyway, the critical point isn't related to this. It is the implementation of `ssl` module in C. `ssl` module relies on C extension to decrypt certificate, but its C implementation doesn't return a decoded certificate if it isn't valid. Instead, an empty dict. – Sraw Jul 18 '18 at 01:24