Python Requests: verify using self-signed cert

Question

I've been trying to test Requests SSL verification using a self-signed cert and setting that as the verify param CA bundle. Instead of the test passing, I get [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590).

Here is my test code:

class TestSubjectVerification(HttpTestSupport):
    @classmethod
    def setUpClass(cls):
        cls.https_port = get_random_port()
        cls.httpsd = server.HTTPServer(('', cls.https_port), Handler)
        cls.httpsd.socket = ssl.wrap_socket(cls.httpsd.socket,
                                            certfile=os.path.join(TEST_RESOURCES_DIR, "certificate.pem"),
                                            keyfile=os.path.join(TEST_RESOURCES_DIR, "private.key"),
                                            server_side=True)
        Thread(target=cls.httpsd.serve_forever).start()

    def test_hello(self):
        port = self.__class__.https_port
        ca = os.path.join(TEST_RESOURCES_DIR, "certificate.pem")
        r = requests.get("https://127.0.0.1:%s" % port, verify=ca)
        self.assertEqual("hello", r.text)

I created my key and cert with the following openssl commands (referencing this stackoverflow post):

openssl genrsa -out private.key 3072

openssl req -new -x509 -key private.key -sha256 -out certificate.pem -days 730

Customizing openssl.cnf, I specified 127.0.0.1 as an IP subjectAltName. Examining the certificate:

$ openssl x509 -in certificate.pem -text -noout

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 15231302753528708005 (0xd3607527ca81fba5)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, ST=North Carolina, L=Durham, O=Foo Company, OU=Foo Org, CN=Craig McDaniel/emailAddress=foobie@example.com
        Validity
            Not Before: Apr 18 19:40:03 2017 GMT
            Not After : Apr 18 19:40:03 2019 GMT
        Subject: C=US, ST=North Carolina, L=Apex, O=DataStax, OU=OpsCenter, CN=Craig McDaniel/emailAddress=craig.mcdaniel@datastax.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (3072 bit)
                Modulus:
                    00:e7:3a:b0:b9:28:bd:e1:d0:74:9f:4b:a7:9b:f6:
                    fe:ba:7e:58:da:05:fc:b4:b7:9c:fc:7a:1f:c4:d5:
                    02:09:77:94:14:07:d6:dd:e3:59:ad:09:fc:e1:79:
                    c5:5d:7f:6e:56:ca:08:21:f3:af:c1:0d:31:c5:aa:
                    2d:78:e9:2f:26:79:0e:76:f9:64:77:9e:94:32:4b:
                    09:c0:3b:15:d0:64:b0:58:85:53:96:e4:f0:ac:da:
                    b1:f4:e3:17:dc:6d:ba:94:e7:49:88:dd:fe:f7:a7:
                    ac:36:4b:3e:07:4b:d0:dc:c1:ae:54:56:a5:4e:8e:
                    89:0b:63:a1:0f:10:12:25:3b:11:0c:8a:a4:03:3d:
                    ef:67:4f:0c:2b:3c:d9:f3:a6:b4:eb:e1:d4:e5:d0:
                    12:45:c6:6a:42:d3:a0:c7:c0:2e:e5:c7:95:17:98:
                    8b:9d:d7:32:f5:a4:87:5a:b5:77:86:91:44:0c:82:
                    f7:94:31:dc:f5:5c:5e:bc:82:73:de:e5:31:84:e3:
                    99:af:bd:7f:80:15:61:cd:f9:5a:85:ae:3a:96:fb:
                    36:27:51:c2:ee:37:8b:65:61:47:f6:9f:0c:8f:1d:
                    d2:a2:99:de:a9:a9:d5:30:04:55:8e:27:d2:c1:cf:
                    ec:b3:3e:47:7b:5a:b6:ac:92:1b:56:0f:65:81:f3:
                    22:97:96:15:9a:0c:a7:8f:ca:72:6b:90:c4:0e:dc:
                    d8:e2:d4:36:32:8e:fd:94:35:a4:b6:44:75:32:3d:
                    8a:7a:96:bc:bc:00:6a:1f:ef:a2:93:29:7f:6a:95:
                    87:47:5b:af:99:88:3d:57:ea:c7:9a:8b:57:b1:a3:
                    5f:26:f6:4e:f8:68:9d:ed:a7:ae:f3:97:20:46:c4:
                    e8:a2:c6:e4:f1:d2:92:1d:9a:08:fa:24:f0:cf:42:
                    74:d6:ed:e2:68:36:2e:dc:5a:6e:e0:ae:33:5e:c1:
                    c1:79:37:a8:49:b7:a3:79:46:40:3c:3b:e5:1b:a2:
                    93:23:04:f9:4b:1e:9d:4f:04:f7
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                08:80:0D:CF:99:29:C4:25:4A:1E:BE:E8:B1:86:F7:83:1E:1C:98:BD
            X509v3 Authority Key Identifier:
                keyid:08:80:0D:CF:99:29:C4:25:4A:1E:BE:E8:B1:86:F7:83:1E:1C:98:BD

            X509v3 Basic Constraints:
                CA:TRUE
            X509v3 Key Usage:
                Digital Signature, Key Encipherment
            X509v3 Subject Alternative Name:
                IP Address:127.0.0.1
    Signature Algorithm: sha256WithRSAEncryption
         54:53:a9:f8:ba:1d:7d:ed:f8:b6:be:c3:46:38:cf:3a:f8:b1:
         25:25:c5:66:c6:d7:b1:eb:7a:c4:4d:0a:55:a2:89:82:31:13:
         37:14:d2:5e:5c:e8:a5:2b:f9:ba:0f:02:21:cd:b8:f7:57:36:
         31:48:8a:8c:a8:04:cd:26:90:e9:7a:99:ab:ec:6d:bf:40:1a:
         d7:6f:37:6d:6c:11:73:5e:44:c3:cd:97:0d:cc:9d:78:91:1e:
         26:8f:48:40:c8:7e:5d:10:f1:35:1b:1e:04:ea:c0:be:54:14:
         39:01:33:86:e9:e3:2a:cc:55:9d:cd:95:eb:74:ab:3b:e2:f1:
         87:ce:1e:dd:50:ac:00:28:8e:71:89:87:78:e9:e0:ba:52:6f:
         05:1c:13:7e:9c:32:77:da:f0:b2:34:05:02:df:3f:95:2a:57:
         96:d5:22:59:c1:e3:c5:45:80:da:01:15:e2:fa:5d:e6:f2:88:
         97:09:a1:3d:5c:81:85:ce:34:f9:d2:21:59:f5:40:50:b5:59:
         09:78:41:8f:99:03:aa:5c:70:e9:ee:95:4d:48:3c:09:e8:bf:
         55:dd:9a:21:4e:f1:2b:4e:bb:7e:5a:c0:9b:5d:dc:57:0b:0b:
         f6:55:18:bc:53:4a:d1:15:41:02:4a:1c:33:ae:c6:ec:eb:66:
         ca:aa:17:05:f9:2d:99:b7:c6:73:5e:2b:9d:0b:30:16:ab:b3:
         06:5b:f5:79:91:a4:f8:2a:4e:19:87:33:50:7d:c4:23:0d:39:
         6d:4e:2b:88:59:55:77:d4:b5:bd:a3:4f:1c:b8:18:e4:56:07:
         13:ef:57:ce:9e:67:60:55:1d:0d:42:ea:27:50:07:fa:f9:30:
         fd:69:5c:39:ba:cf:36:a1:f7:e0:ff:69:8b:38:41:f1:4c:7d:
         39:ac:25:a9:41:ed:2b:c7:b9:88:0f:1e:d0:12:50:c8:ba:4a:
         5e:07:d7:12:77:c7:68:09:8f:37:be:b8:a3:d4:e2:ea:9d:88:
         36:25:df:f6:3c:f8

Also, curl seems to validate the cert just fine and prints the response (though it gives some other error):

curl --cacert /path/to/my/certificate.pem  https://127.0.0.1:8443/
curl: (56) GnuTLS recv error (-110): The TLS connection was non-properly terminated.
hello

curl https://127.0.0.1:8443/
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none

Answer 1

My guess is that you are using Python 2.7. According to the source code of ssl.py only DNS entries in subject alternative names are checked which means your IP address entry is simply ignored. This seems to be valid even in the latest version of Python 2.7, i.e. 2.7.13.

With Python 3 this is fixed. According to ssl.py in Python 3.5 it will specifically check the IP address type in subject alternative names.

This means you either have to use a DNS entry instead of the IP address entry which although wrong works with Python 2.7 but stops working with Python 3. Or you can use Python 3 instead. Or use a hostname.