How to make Python use CA certificates from Mac OS TrustStore?

Question

I need to use custom root certificates on the company intranet and loading them in the Mac OS TrustStore (KeyChain) does solve the problem for all browsers and GUI apps.

It seems that it works even with the version of curl that ships with Mac OS X but it doesn't work with python, even the version that ships with Mac OS 10.12 Sierra (Python 2.7.10)

Still, it seems that I would be hit by:

urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>

How can I solve this?

Because I encounter this issue in lots and lots of Python tools. I would really appreciate if I find a way to avoid it without having to patch them.

Providing the custom CA certificate myself is not an option because I cannot patch tens of Python tools that I use.

Most of the tools are using the requests library but, there are a few that are using the native ssl support in Python directly.

Answer 1

This is also a problem in Python 3.6 with MacOS Sierrra. I know your use case is different. But I stumbled upon this thread while investigating this problem. So if anyone is also having this article is worth checking out:

http://www.cdotson.com/2017/01/sslerror-with-python-3-6-x-on-macos-sierra/

In a nutshell: Python 3.6 does not rely on MacOS' openSSL anymore. It comes with its own openSSL bundled and doesn't have access on MacOS' root certificates.

You have two options:

Run an install command shipped with Python 3.6

cd /Applications/Python\ 3.6/
./Install\ Certificates.command

or

Install the certifi package with

pip install certifi

I chose the first option and it worked.

Answer 2

Run this to set the appropriate variables. This is a combination of the answers that have already been given here. Put it in your ~/.bash_profile to make it permanent.

CERT_PATH=$(python -m certifi)
export SSL_CERT_FILE=${CERT_PATH}
export REQUESTS_CA_BUNDLE=${CERT_PATH}

Answer 3

Mac brew install python env.

$ python3
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 16:52:21) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import certifi
>>> certifi.where()
'/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/certifi/cacert.pem'
>>> 

Or from the command line:

$ python -m certifi

then need link cacert.pem as cert.pem

$ ln -s /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/certifi/cacert.pem cert.pem
$ pwd
/Library/Frameworks/Python.framework/Versions/3.7/etc/openssl

rehash

then work fine.

Answer 4

This was intended as an edit to an existing question, but since the queue was already full, posting as a separate answer.

Tested on MacOS 12.3.1 with python 3.10 installed with MacPorts.

If you prefer to trust root CA according to your OS, export them from System Roots keychain into a single file:

security export -t certs -f pemseq -k /System/Library/Keychains/SystemRootCertificates.keychain -o bundleCA.pem

If, additionally, you want to trust some internal self-signed CAs, export them, too. They are probably stored under the System keychain:

security export -t certs -f pemseq -k /Library/Keychains/System.keychain -o selfSignedCAbundle.pem

Merge the two files:

cat bundleCA.pem selfSignedCAbundle.pem >> allCAbundle.pem

Export as bash variable

export REQUESTS_CA_BUNDLE=/path/to/allCAbundle.pem

Consider adding the last code snippet to your .bash_profile

Note that REQUESTS_CA_BUNDLE works only for a single file, and does not for a directory.

Answer 5

If you put the additional certificates in a PEM bundle file you can use these two environment variables to overwrite the default cert stores used by Python openssl and requests.

SSL_CERT_FILE=/System/Library/OpenSSL/cert.pem
REQUESTS_CA_BUNDLE=/System/Library/OpenSSL/cert.pem

Please note that this file does not exist, you need to build it yourself.

Answer 6

I see a lot of answers out there recommend to turn off certificate validation or to use certifi.where. While turning off SSL is obvious risk. certifi.where is also a risk, mainly if you intend to make this code a production code that will run in a customer env.

PEP describing why it is wrong. ssl.create_default_context is well integrate with linux and windows trustStore. the problem is, as in your case with mac. I solve this by loading the certificates using the integrated security commandline tool

def create_macos_ssl_context():
    import subprocess
    import ssl
    import tempfile
    ctx = ssl.create_default_context()
    macos_ca_certs = subprocess.run(["security", "find-certificate", "-a", "-p",
                                     "/System/Library/Keychains/SystemRootCertificates.keychain"],
                                    stdout=subprocess.PIPE).stdout

    with tempfile.NamedTemporaryFile('w+b') as tmp_file:
        tmp_file.write(macos_ca_certs)
        ctx.load_verify_locations(tmp_file.name)
    print(ctx.get_ca_certs())

Note that this gives you the systemRoot certificates. if you need the user than simply change the value in the security command

Answer 7

As an update and datapoint, I ran into this issue running Python 3.7.0 on macOS 10.13.4:

$ ipython
Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 26 2018, 23:26:24)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.0.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import bokeh.sampledata

In [2]: bokeh.sampledata.download()
Using data directory: /Users/me/.bokeh/data

...
SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1045)

Instructions for solving the problem are in /Applications/Python\ 3.7/ReadMe.rtf

Following the suggestion there and running /Applications/Python\ 3.7/Install\ Certificates.command solved the problem:

From the terminal:

$ /Applications/Python\ 3.7/Install\ Certificates.command

Re-starting IPython...

$ ipython
>>> import bokeh.sampledata

>>> bokeh.sampledata.download()
Using data directory: /Users/me/.bokeh/data
Downloading: CGM.csv (1589982 bytes)
   1589982 [100.00%]
...

Answer 8

For me /Applications/Python\ 3.6/./Install\ Certificates command fails on pip certifi install. I am on mac High Sierra and use python3 so pip somewhat fails and I have to use pip3 instead.

So here what I did:

1. Manually ran pip3 install --update certify in a shell
2. Remove the install certifi line from the command script
3. Reran the script and everything was fine.

Note that you will end up with a cert.pem symbolic link in: /Library/Frameworks/Python.framework/Versions/3.6/etc/openssl/

Answer 9

Solution for MacOS or Linux with latest Python versions installed either as standalone or via port or brew

Download certificates from Certifi project at https://github.com/certifi/python-certifi/blob/master/certifi/cacert.pem. FYI Certifi is a 3rd party library that provides Mozilla’s curated collection of Root Certificates for validating the trustworthiness of SSL certificates while verifying the identity of TLS hosts.

Then add to your ~/.zshrc on the latest MacOS or ~/.bash_profile or similar:

export SSL_CERT_FILE=/pathtodownloadedfile/cacert.pem
export REQUESTS_CA_BUNDLE=/pathtodownloadedfile/cacert.pem

Answer 10

In my case only installing "Install Certificates.command" solved this issue using MAC OS

Update SSL certificate with certifi (MacOS only)

All we would have to do is to run command with the following piece of code:

- Press "command + space" button or open Spotlight
- type "Install Certificates.command" 

What this command does is update our system’s SSL certificate directory for MacOS.