obtain the urllib.requests certificate bundle used with urlopen

Question

I am on windows, and I am seeing a difference between urllib.requests.urlopen and requests package when making calls to the same site.

When I perform the following:

import urllib.request
f = urllib.request.urlopen('https://<domain certificated server> ')

I can reach my server no problems.

When I do:

import requests
f = requests.get('https://<domain certificated server> ').text

I am getting an SSL Certificate error. I know this is caused by certifi. So my question is this, how can I leverage whatever Python code is doing and use that over certifi in requests?

Have you updated certifi? Read https://2.python-requests.org/en/master/user/advanced/#ca-certificates - have you confirmed that urllib is actually checking for a valid certificate - if it isn’t then the problem is with urllib rather than requests. If you don’t care about certificate checking ((e.g. intranet use) you can always disable it in requests too. — DisappointedByUnaccountableMod, Oct 09 '20 at 17:21
Read: https://stackoverflow.com/questions/24374400/verifying-https-certificates-with-urllib-request. — TrollGamingDude-__-, Oct 09 '20 at 17:22
@TrollGamingDude-__- what I'm trying to do is make requests library work with the same cert store as urllib.request.urlopen. I love the requests interface, but need the standard lib's cert verification. — JabberJabber, Oct 09 '20 at 17:28
Did you check the certificate is in fact valid - you can view it in your browser. — DisappointedByUnaccountableMod, Oct 09 '20 at 17:28
certificate is valid. It's a domain certificate, no errors in chrome, firefox and Edge (gasp I know) — JabberJabber, Oct 09 '20 at 17:29
What’s the URL you are accessing? What version of Python? Requests? — DisappointedByUnaccountableMod, Oct 09 '20 at 17:36
It's an internal server only, so I can't provide it (sorry about that), but I'm using Python 3.6.10 and 3.7.6. For requests, I'm using the latest release and the latest release of certifi. ( updated that manually) — JabberJabber, Oct 09 '20 at 18:54
Certifi is definitely more up to date than urllib in Python 3.6.10 and 3.7.6. So this internal server has a hostname that (on the network it is on) resolves to your server and that hostname matches the hostname in the certificate? — DisappointedByUnaccountableMod, Oct 09 '20 at 20:04

score 1 · Answer 1 · answered Oct 21 '20 at 12:52

1

Do you have any option to obtain a .pem file with the server certificates?.

urlopen is using ssl.enum_certificates on windows to load the certificates. So the certs are obtained from a windows service.

Once you get the certificate file, use it on your call requests.get('<server.internal>', verify='/path/to/certfile')

answered Oct 21 '20 at 12:52

pbacterio

1,094
6
12

I can enumerate the certificates, but I couldn't get requests to use them. – JabberJabber Nov 18 '20 at 12:20

score 1 · Answer 2 · answered Oct 22 '20 at 07:01

The simple way is to tell requests to use the system bundle, if you know where it is. On Linux it's usually in /etc/ssl/certs/ca-certificates.crt.

requests.get('<server.internal>', verify='/etc/ssl/certs/ca-certificates.crt')

You can also define your own adapter if you are on Windows, don't know where the system bundle is located, or just want to support all cases. The default adapter tells urllib3 to load certificates from DEFAULT_CA_BUNDLE_PATH which comes from calling certifi.where(). If you don't tell urllib3 where to get certificates, it will use the system default.

import requests
import requests.adapters

class SysCertsHTTPAdapter(requests.adapters.HTTPAdapter):
  def cert_verify(self, conn, url, verify, cert):
    # require verification, but use system defaults
    # default requests implementation sets conn.ca_certs and/or conn.ca_cert_dir
    conn.cert_reqs = 'CERT_REQUIRED'


session = requests.Session()
session.mount('https://', SysCertsHTTPAdapter())
session.get("https://google.com")

I find this did not work. The problem is the urllib3 defaults to certifi.where() location — JabberJabber, Oct 22 '20 at 16:34
What makes you say that? It [calls](https://github.com/urllib3/urllib3/blob/4fb7f3252b92bcc50cc8fa753f9dc6c147a6b2c0/src/urllib3/connection.py#L391-L398) [`load_default_certs()`](https://docs.python.org/3/library/ssl.html#ssl.SSLContext.load_default_certs) which uses system locations. The [documentation](https://github.com/urllib3/urllib3/blob/ccf907904d042625291472fe7a85971b62037948/docs/user-guide.rst#certificate-verification) states you have to manually use `ceritfi` and this code doesn't. — kichik, Oct 22 '20 at 17:03

obtain the urllib.requests certificate bundle used with urlopen

2 Answers2