0

I'm trying to scrape an endpoint that requires an SSL certificate for authentication. When I try accessing the site in the browser, a window comes up where I select the certificate to be sent with the request. When downloading the webpage, I've tried doing:

import urllib
urllib.request.urlopen("https://mywebpage.com/some/endpoint", cafile = "C:/Users/Me/path/to/my/cert.p12")

but I got the following error:

ssl.SSLError: [X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found (_ssl.c:4149)

I thought this might be because I was sending a PKCS12 instead of a PEM file, so I did the following:

openssl pkcs12 -in "C:/Users/Me/path/to/my/cert.p12" -out "C:/Users/Me/path/to/my/cert.pem"

I provided the same value for the PKCS12 passphrase and the PEM password. I then tried doing this:

import urllib
urllib.request.urlopen("https://mywebpage.com/some/endpoint", cafile = "C:/Users/Me/path/to/my/cert.pem")

However, this returns the following SSL error:

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)

When I do ssl.SSLContext().load_verify_locations(cafile = "C:/Users/Me/path/to/my/cert.pem") I get no errors and the certificate information seems to load.

I then decided to try getting around the issue and tried searching for the specific SSL error. I came across this SO question. Since I'm using Windows, my version of OpenSSL is likely configured to be different from what Python is expecting, so I tried doing the following:

import ssl
import certifi
from urllib import request
request.urlopen("https://mywebpage.com/some/endpoint", 
    context = ssl.create_default_context(cafile = certifi.where()))

but this returned the following SSL error:

ssl.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1007)

So, what am I doing wrong here? It appears that the server was unable to verify the SSL certificate I sent with the request but, when I check it, it appears to match. How can I make this request work?

Woody1193
  • 7,252
  • 5
  • 40
  • 90

1 Answers1

2

CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate

This error message is unrelated to the site requiring a client certificate. Instead it is about the client (your program) not being able to validate the server certificate.

While nothing about the server certificate is known it might be caused by using the expected client certificate as cafile argument. But cafile is for specifying the trusted root CA for server validation, not the client certificate. To specify this you have to create a SSL context:

from urllib.request import urlopen
import ssl
ctx = ssl.create_default_context()
ctx.load_verify_locations('ca-which-signed-the-server-certificate.pem')
ctx.load_cert_chain('my-client-cert.pem','my-client-key.pem')
urlopen('https://needs-mtls.example.com', context=ctx)
Steffen Ullrich
  • 114,247
  • 10
  • 131
  • 172
  • I'm not entirely sure how to implement your answer as I only have the one PEM file, and that was generated from a PKCS12 file that I received so I'm not sure I should be using it that way at all. – Woody1193 Jun 13 '23 at 07:17
  • Okay, I still don't understand how that worked but it worked, so thank you. – Woody1193 Jun 13 '23 at 07:31
  • 1
    @Woody1193: the PEM file you've created from pkcs12 contains both certificate and key. – Steffen Ullrich Jun 13 '23 at 07:32
  • Hmm. Except the issue I'm having when I do this is that I get a login page back instead of the actual data page I'm expecting. – Woody1193 Jun 13 '23 at 09:06
  • 1
    @Woody1193: Try to export the key from the PEM file w/o password, i.e. `openssl pkcs12 -in cert.p12 -out client-cert.pem` followed by `openssl pkey -in client-cert.pem -out client-key.pem` – Steffen Ullrich Jun 13 '23 at 09:32
  • So, the reason I was being directed to a login page was that it was a .faces file and I wasn't posting the correct cookie, apparently. – Woody1193 Jun 13 '23 at 23:57