cURL queries work, but python-request fails

Question

Summary

We are using a custom CA in our enviroment. I have imported the CA certificates in the trust store by downloading the Base64 certs and updating the trust store using update-ca-certificates.

I am able to run cURL queries to my REST API, however the request library fails with an SSL error when running the same.

What I have tried

I have tried specifying the root ca certificate file path to the library, but got the same error. How do I troubleshoot this issue? Setting verify to false is not an option.

Code Run

Works

curl -X GET https://api.me.com/admin/ -H 'Authorization: Token 4ae5'

Doesn't work

requests.get('https://api.me.com/admin/', headers={'Authorization': 'Token 4ae5'}, verify='/etc/ssl/certs/root.pem')

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 345, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 844, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connection.py", line 326, in connect
    ssl_context=context)
  File "/usr/local/lib/python3.5/dist-packages/urllib3/util/ssl_.py", line 325, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.5/ssl.py", line 377, in wrap_socket
    _context=self)
  File "/usr/lib/python3.5/ssl.py", line 752, in __init__
    self.do_handshake()
  File "/usr/lib/python3.5/ssl.py", line 988, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib/python3.5/ssl.py", line 633, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 376, in send
    timeout=timeout
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
    raise SSLError(e)
requests.packages.urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/requests/api.py", line 67, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 480, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 588, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 447, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

`requests.get(..., verify='/etc/ssl/certs/root.pem')` - are you sure you are using the correct path? Given your description of using `update-ca-certificates` I guess you are using Debian or similar (i.e. Ubuntu). In this case the relevant path should be `/etc/ssl/certs/ca-certificates.crt`. — Steffen Ullrich, Feb 05 '19 at 11:57
God, this is embarrassing. You are absolutely right, I had to use the bundled certificate. I kept on trying the absolute paths i specified in the /etc/ca-certificates.conf and kept getting errors! :D Please post it as an answer, I will accept it. — NerdSec, Feb 05 '19 at 12:24

score 3 · Accepted Answer · answered Feb 05 '19 at 12:35

3

... requests.get(..., verify='/etc/ssl/certs/root.pem')

Given your description of using update-ca-certificates it looks like you are using Debian or similar (i.e. Ubuntu). In this case the relevant path should be /etc/ssl/certs/ca-certificates.crt.

answered Feb 05 '19 at 12:35

Steffen Ullrich

114,247
10
131
172

score 1 · Answer 2 · answered Feb 09 '21 at 01:04

The reason is that Python Requests uses certificates from the python-certifi package., not those of the underlying operating system.

certifi includes all CA certificates from Mozilla, and it is also relatively straightforward to add missing certificates to it.

See this post for details on adding certificates to certifi: https://stackoverflow.com/a/66111417/516699

The good thing is also that you do not tweak the SSL of the underlying system, just that of the Python environment your are working in.