Why python requests not use the system ssl cert by default?

Question

Background:

I'm working on Ubuntu 18.04.1 LTS, using next to install the self-signed certs:

$ cp -rf my.crt /usr/local/share/ca-certificates/
$ update-ca-certificates

Everything works fine, because now I could use next to visit my web successfully:

$ curl https://example.com

Problem:

However, when I use python3 requests to visit, it reports next error:

>>> requests.get("https://example.com")
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 421, in connect
    tls_in_tls=tls_in_tls,
  File "/usr/local/lib/python3.6/dist-packages/urllib3/util/ssl_.py", line 429, in ssl_wrap_socket
    sock, context, tls_in_tls, server_hostname=server_hostname
  File "/usr/local/lib/python3.6/dist-packages/urllib3/util/ssl_.py", line 472, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/usr/lib/python3.6/ssl.py", line 817, in __init__
    self.do_handshake()
  File "/usr/lib/python3.6/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:847)

Additional:

But, if I explicitly specify the crt, it works successfully:

>>> requests.get("https://example.com", verify='/etc/ssl/certs/ca-certificates.crt')
<Response [200]>

Question:

I'm using a third-party code, which means I can't change the python code.

So, my question is: why python requests not default to find verify in /etc/ssl/certs/ca-certificates.crt? It's the default certification on ubuntu I think, as curl successfully find this by default...

And, if possible I could set any variable with export which outside of python code, then python requests will default to find the certification?

Not only requests does not use the system certificates on Ubuntu by default, but neither do Firefox, Chrome, Java, ... . And it is not much different on other systems. Part of the reason are different formats of the CA store. Part is that it is seen by the developers as important to be consistent over different OS (which all have different ideas of location and format of trust store) than to be consistent with other applications on the same system. It's unfortunately a mess one has to live with. — Steffen Ullrich, Mar 29 '21 at 04:57
@SteffenUllrich So, what should I do in my case as I can't change the third-party code... — atline, Mar 29 '21 at 05:00
*"So, what should I do in my case?"* - if you want to use the system CA store than explicitly make your application use it, like you did. If you just want to get a specific CA trusted by a specific app don't use the system CA store but keep it local to the app. If you cannot change the app then change the certificate store actually used by the app. It might be certifi (common with requests), it be something else. See also https://incognitjoe.github.io/adding-certs-to-requests.html — Steffen Ullrich, Mar 29 '21 at 05:01
You either have to edit the third party code or you have to add the CA to the trust store used by the third party code - whatever this trust store is. You might use strace to figure out where it is looking for certificates. But it might also rely on fixed certificates set in the app, in which case you need to change the app. — Steffen Ullrich, Mar 29 '21 at 05:05
@SteffenUllrich, thanks for your hints :) Finally, I found ubuntu official already did similar things like you suggested, change the CA path of apps using a variant of certifi, then I can change nothing to make it work. Thanks! — atline, Mar 29 '21 at 06:47

atline · Accepted Answer · 2021-03-29T07:30:44.463

I find the root cause, the requests in fact will call where using from certifi import where to find the proper CA on this pc.

But, there are 2 kinds of ways to install certifi module in ubuntu:

Option 1:

apt-get install -y python3-certifi

Option 2:

pip3 install certifi

NOTE: if directly using pip3 install requests, it will implicitly install certifi using pip3 if no debian package installed for python3-certifi.

However, looks Canonical (the backers of Ubuntu) made some changes for certifi, so if using python3-certifi from apt, the code of def where is next or similar varies from different versions:

root@4e1aab76e082:/# cat /usr/lib/python3/dist-packages/certifi/core.py
def where():
    return "/etc/ssl/certs/ca-certificates.crt"

But if using pip3 to install, it will be:

root@4e1aab76e082:/# cat /usr/local/lib/python3.6/dist-packages/certifi/core.py
def where():
    _CACERT_CTX = get_path("certifi", "cacert.pem")

That's /usr/local/lib/python3.6/dist-packages/certifi/cacert.pem in this package.

So the solution on ubuntu is: use apt-get install python3-certifi to install the ubuntu variant of certifi, uninstall the pip one. Then, we can no change anything of app code.

UPDATE:

I find another way which works with official certifi, using next variable, I could also let python requests module go to fetch the CA from where I specified without change app code:

export REQUESTS_CA_BUNDLE='/etc/ssl/certs/ca-certificates.crt'

Why python requests not use the system ssl cert by default?

1 Answers1

Linked