159

I just installed Python3 from python.org and am having trouble installing packages with pip. By design, there is a man-in-the-middle packet inspection appliance on the network here that inspects all packets (ssl included) by resigning all ssl connections with its own certificate. Part of the GPO pushes the custom root certificate into the Windows Keystore.

When using Java, if I need to access any external https sites, I need to manually update the cacerts in the JVM to trust the Self-Signed CA certificate.

How do I accomplish that for python? Right now, when I try to install packages using pip, understandably, I get wonderful [SSL: CERTIFICATE_VERIFY_FAILED] errors.

I realize I can ignore them using the --trusted-host parameter, but I don't want to do that for every package I'm trying to install.

Is there a way to update the CA Certificate store that python uses?

oz123
  • 27,559
  • 27
  • 125
  • 187
Eric B.
  • 23,425
  • 50
  • 169
  • 316
  • 10
    @rfkortekaas All those options all involve adding something new to the process. Python must make use of a default trust store stored somewhere on the system. I'd like to modify that trust store. I don't want to add extra variables, different ca stores, etc. In java, the jvm relies on its own trust store (separate from the OS). I suspect that python must be doing something similar since my root certificate is in my Windows store and not recognized by python. – Eric B. Sep 06 '16 at 21:38
  • 1
    @eric_b you should accept an answer. – pauljohn32 Apr 28 '21 at 04:57

11 Answers11

224

Self-Signed Certificate Authorities pip / conda

After extensively documenting a similar problem with Git (How can I make git accept a self signed certificate?), here we are again behind a corporate firewall with a proxy giving us a MitM "attack" that we should trust and:

NEVER disable all SSL verification!

This creates a bad security culture. Don't be that person.

UPDATE 2022 Python 3.10

Can experimentally use the system truststore. https://pip.pypa.io/en/latest/user_guide/#using-system-trust-stores-for-verifying-https

# Requires Python 3.10 or later
python --version
Python 3.10.4

# Install the 'truststore' package from PyPI
python -m pip install truststore


# Use '--use-feature=truststore' flag to enable
python -m pip install SomePackage --use-feature=truststore

tl;dr: Configuring your own trust store CA bundle

pip config set global.cert path/to/ca-bundle.crt
pip config list
conda config --set ssl_verify path/to/ca-bundle.crt
conda config --show ssl_verify

# Bonus while we are here...
git config --global http.sslVerify true
git config --global http.sslCAInfo path/to/ca-bundle.crt

But where do we get ca-bundle.crt?

Get an up to date CA Bundle

cURL publishes an extract of the Certificate Authorities bundled with Mozilla Firefox

https://curl.haxx.se/docs/caextract.html

I recommend you open up this cacert.pem file in a text editor as we will need to add our self-signed CA to this file.

Certificates are a document complying with X.509 but they can be encoded to disk a few ways. The below article is a good read but the short version is that we are dealing with the base64 encoding which is often called PEM in the file extensions. You will see it has the format:

----BEGIN CERTIFICATE----
....
base64 encoded binary data
....
----END CERTIFICATE----

https://support.ssl.com/Knowledgebase/Article/View/19/0/der-vs-crt-vs-cer-vs-pem-certificates-and-how-to-convert-them

Getting our Self Signed Certificate

Below are a few options on how to get our self signed certificate:

  • Via OpenSSL CLI
  • Via Browser
  • Via Python Scripting

Get our Self-Signed Certificate by OpenSSL CLI

https://unix.stackexchange.com/questions/451207/how-to-trust-self-signed-certificate-in-curl-command-line/468360#468360

echo quit | openssl s_client -showcerts -servername "curl.haxx.se" -connect curl.haxx.se:443 > cacert.pem

Get our Self-Signed Certificate Authority via Browser

Thanks to this answer and the linked blog, it shows steps (on Windows) how to view the certificate and then copy to file using the base64 PEM encoding option.

Copy the contents of this exported file and paste it at the end of your cacerts.pem file.

For consistency rename this file cacerts.pem --> ca-bundle.crt and place it somewhere easy like:

# Windows
%USERPROFILE%\certs\ca-bundle.crt

# Linux/macOS
$HOME/certs/cabundle.crt

Get our Self-Signed Certificate Authority via Python

Thanks to all the brilliant answers in:

How to get response SSL certificate from requests in python?

I have put together the following to attempt to take it a step further.

https://github.com/neozenith/get-ca-py

Finally

Set the configuration in pip and conda so that it knows where this CA store resides with our extra self-signed CA.

# Windows
pip config set global.cert %USERPROFILE%\certs\ca-bundle.crt
conda config --set ssl_verify %USERPROFILE%\certs\ca-bundle.crt

OR

# Linux / macOS
pip config set global.cert $HOME/certs/ca-bundle.crt
conda config --set ssl_verify $HOME/certs/ca-bundle.crt

THEN

pip config list
conda config --show ssl_verify

# Hot tip: use -v to show where your pip config file is...
pip config list -v
# Example output for macOS and homebrew installed python
For variant 'global', will try loading '/Library/Application Support/pip/pip.conf'
For variant 'user', will try loading '/Users/jpeak/.pip/pip.conf'
For variant 'user', will try loading '/Users/jpeak/.config/pip/pip.conf'
For variant 'site', will try loading '/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/pip.conf'

Troubleshooting

Based on a great comment below

I've tried this and still get a SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')) error. Any suggestions?

here is a troubleshooting guide:

This is the normal error message when the certificates authorities are not yet correctly setup.

It could be a variety of factors to check:

  • The paths to your ca-bundle.crt have the right path separators for your OS (it has stung me),
  • you may not have the latest CAs to verify normal certificates,
  • you may not have added your CA in the right encoding.

Python is effectively doing those 3 steps:

  • Find my CA store,
  • read all entries,
  • look up this certificate against my trust store.

If any of those fail you get this error message from experience.

Check this answer linked from below to display and check your ssl_cert_dir using:

python -c "import ssl; print(ssl.get_default_verify_paths())"

References

Josh Peak
  • 5,898
  • 4
  • 40
  • 52
  • pip config set global.trusted-host XXXXX.com – zzzz zzzz Apr 08 '19 at 09:51
  • In windows, it works for me using the .pem format for the certificate. – Daniel Argüelles Apr 30 '19 at 10:30
  • 1
    @DanielArgüelles yeah that's right. Most often you can get away with not merging the Certificate Authority bundle but I have had enough times where the full bundle is needed so that pip or conda can validate certificates for other servers. Ultimately a bundle is still a text file with the contents of lots of pem files. Glad it worked and you didn't need to disable verification! :D – Josh Peak May 03 '19 at 04:41
  • I've tried this and still get a `'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)'))'` error. Any suggestions? – Leonardo May 19 '21 at 19:00
  • @Leonardo That is the normal error message when the certificates authorities are not yet correctly setup. It could be a variety of factors to check: The paths to your ca-bundle.crt have the right path separators for your OS (it has stung me), you may not have the latest CAs to verify normal certificates, you may not have added your CA in the right encoding. Python is doing those 3 steps: Find my CA store, read all entries, look up this certificate against my trust store. If any of those fail you get this error message from experience. – Josh Peak May 20 '21 at 01:26
  • 1
    @JoshPeak, is there any way to try and debug this to see where exactly it's failing? I've [posted this question](https://stackoverflow.com/questions/67609733/pip-custom-root-ca-usual-solutions-not-working) with the description of what I did. – Leonardo May 20 '21 at 15:20
  • 1
    Brilliant! having to combine the self-signed certificate required for the proxy and the pypi.org package store site certificate into *ONE* certificate file was the solution I've been trying to find for days! – beneM Nov 17 '21 at 17:34
  • 1
    @beneM Yeah I tried multiple files and it looks like it should recognise a folder of CAs but appending it all into one file was the only one that worked for me. – Josh Peak Nov 18 '21 at 00:29
  • should `pip config set` show up in `python -c "import ssl; print(ssl.get_default_verify_paths())"` otherwise, why doing it all? – Stefan Jan 24 '22 at 15:21
  • 1
    @JoshPeak Can you amend this answer to include Pip's new experimental support for using system trust stores via a library called "truststore"? https://pip.pypa.io/en/latest/user_guide/#using-system-trust-stores-for-verifying-https – sethmlarson Jul 27 '22 at 14:45
  • 1
    Just a note that when you are getting your own self-signed CA cert from the you may find that there are multiple if you use the browser method. You will likely need to add all of them – born_naked Apr 27 '23 at 19:52
54

Run: python -c "import ssl; print(ssl.get_default_verify_paths())" to check the current paths which are used to verify the certificate. Add your company's root certificate to one of those.

The path openssl_capath_env points to the environment variable: SSL_CERT_DIR.

If SSL_CERT_DIR doesn't exist, you will need to create it and point it to a valid folder within your filesystem. You can then add your certificate to this folder to use it.

Stevoisiak
  • 23,794
  • 27
  • 122
  • 225
rfkortekaas
  • 6,049
  • 2
  • 27
  • 34
  • 12
    on my Windows system this returns '/usr/local/ssl/certs' which is not available on Windows. – Colin Talbert Sep 20 '16 at 14:20
  • 2
    I finally got around to doing this as well after being yanked onto another project, and similar to @ColinTalbert it points to a non-existant folder `/usr/local/ssl/certs`. – Eric B. Sep 20 '16 at 21:25
  • I've editted my question and hopefully this solves the case. – rfkortekaas Sep 21 '16 at 18:45
  • 1
    @rfkortekaas Updating the SSL_CERT_FILE or the SSL_CERT_DIR variables didn't work. I just created a [new SO question](http://stackoverflow.com/q/39356413/827480) for this problem as it may not be simply a question of how to update a PEM file, but rather how to get python to access the right paths in cygwin/Windows. – Eric B. Sep 21 '16 at 20:42
  • Have you restarted your shell/python after you created the environment variabel? – rfkortekaas Sep 22 '16 at 18:20
  • 2
    I tried. I ended up creating a pip.conf file in `~/.config/pip/pip.conf` with the necessary settings. [See this answer](http://stackoverflow.com/a/39626495/827480). – Eric B. Sep 23 '16 at 21:14
  • I was doing it in a temporary or 'per session' manner. i.e. set SSL_CERT_FILE=<> – Colin Talbert Sep 27 '16 at 15:14
  • can anyone tell why the settings in "pip.ini" are not reflected here? It looks like as "pip.ini" is not taken into accout. – Stefan Jan 24 '22 at 15:06
23

Alternative solution on Windows is to install python-certifi-win32 that will allow Python to use Windows Certificate Store.

pip install python-certifi-win32
nt86
  • 1,480
  • 12
  • 12
  • 10
    first time you want to do `pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org python-certifi-win32` – k1m190r Mar 02 '21 at 16:31
  • simple, but saved my day! – lpounng Oct 12 '21 at 01:45
  • 1
    Not sure how my install was different, but doing this (including the suggested `--trusted-host` flags) completed successfully, but broke `pip` .. all subsequent calls to `pip` crashed, I couldn't even `pip uninstall python-certifi-win32`. I renamed the files manually so pip would work again and prove this was the culprit. – CodeShane Apr 19 '22 at 03:26
  • 7
    `python-certifi-win32` has been [replaced](https://gitlab.com/alelec/python-certifi-win32/-/issues/16) by [`pip-system-certs`](https://pypi.org/project/pip-system-certs/). – sourcream Aug 02 '22 at 12:57
  • Wow!!! This simple one line code solved my SSL error problem. I am not IT expert so don't know what it does but this solution really made my day. Thanks @nt86 – Manish Jan 04 '23 at 07:46
  • my code was working with python-certifi-win32 but when I removed this package and installed pip-system-certs I now get certificate handshake error. – Ali Aug 27 '23 at 07:05
10

Not best answer but you can reuse an already created ca bundle using --cert option of pip, for instance:

pip install SQLAlchemy==1.1.15 --cert="C:\Users\myUser\certificates\my_ca-bundle.crt"
aturegano
  • 936
  • 15
  • 30
10

On Windows, I solved it by creating a pip.ini file in %APPDATA%\pip\

e.g. C:\Users\asmith\AppData\Roaming\pip\pip.ini

In the pip.ini I put the path to my certificate:

[global]
cert=C:\Users\asmith\SSL\teco-ca.crt

https://pip.pypa.io/en/stable/user_guide/#configuration has more information about the configuration file.

Alex
  • 305
  • 3
  • 8
  • Hi Alex, how did you "put" the path to your certificate in the pip.ini file? From the command line? Did you type the file path into something like notepad and save it as a text file? I am using the anaconda prompt, but I think it is similar to windows. – spacedustpi Jul 31 '18 at 16:27
  • I created a text file with Notepad and then changed the file extension from "txt" to "ini". – Alex Aug 02 '18 at 00:02
  • Ah. Okay. Do you have to also have to type in "[gobal]" above the path? Do you know where I can go for a tutorial for this type of thing? Thanks. – spacedustpi Aug 02 '18 at 00:25
  • Yes, you need to type in "[global]" too. Sorry, I don't know of any tutorial but https://pip.pypa.io/en/stable/user_guide/#configuration has more information. – Alex Aug 03 '18 at 01:19
  • Thanks, I tried it out both ways and yes, "[global]" doesn't hurt. – spacedustpi Aug 04 '18 at 12:30
  • Praise for simplicity. Why don't you need to make certificate bundle as suggested in other answer,https://stackoverflow.com/a/52961564/1086346 – pauljohn32 Apr 28 '21 at 05:00
2

I think nt86's solution is the most appropriate because it leverages the underlying Windows infrastructure (certificate store). But it doesn't explain how to install python-certifi-win32 to start with since pip is non functional.

The trick is to use --trustedhost to install python-certifi-win32 and then after that, pip will automatically use the windows certificate store to load the certificate used by the proxy.

So in a nutshell, you should do:

pip install python-certifi-win32 -trustedhost pypi.org

and after that you should be good to go

General Grievance
  • 4,555
  • 31
  • 31
  • 45
TonyM
  • 21
  • 1
1

In my scenario, on Ubuntu using python 3.8, I wanted to see what I could change in my local system (rather than code or a installing a new package) that would allow me to utilize my custom CA and my self-signed certs.

So I just traced where python was looking:

strace python my_prog 2> out.txt

Then grepped the output for pem (you may need to look for "crt" "cert")

grep -i pem out.txt

Even though certifi wasn't ever explicitely installed, the file it was looking for was:

/path/to/virtualenv/for/project/lib/python3.8/site-packages/certifi/cacert.pem

So I renamed the old file and linked it to my system's ca bundle (e.g., /etc/ssl/certs/ca-certificates.crt)

And now it worked.

healthybodhi
  • 306
  • 2
  • 4
1

I cannot comment on Josh's The Greatest Reply of All Time nor rfKorttekaas less epic but also useful one due to too little reputation points, so here's another hint. In my case this error:

SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')) error.

required me to find & add to the cacerts.pem / ca-bundle.crt

  • certificate for Intermediate Certification Authorities (for the Self-Signed Certificate I tried to add)
  • Intermediate root certificate for that Intermediate CA

I got them by exporting correct certs from certmgr.msc on my machine (it's a corporation-vpn-hell kinda situation, only pip & certify, methods listed in other answers are not available for me). Also, a friendly reminder to restart your machines if adding environment variables (SSL_CERT_DIR, SSL_CERT_FILE) does not seem to work ;)

fatcatpl
  • 43
  • 6
0

In this post:

https://github.com/jorgenschaefer/elpy/issues/1936

they advice:

M-x elpy-rpc-reinstall-virtualenv

It worked for me.

Vince
  • 3,979
  • 10
  • 41
  • 69
0

In a corporate environment where the firewall is using a certificate issued by internal CA : You need to append the content of the public certificate of your FW and CA chain into "C:\Python311\Lib\site-packages\pip_vendor\certifi\cacert.pem"

Alon Or
  • 798
  • 7
  • 7
-3

Open Anaconda Navigator.

Go to File\Preferences.

Enable SSL verification Disable (not recommended)

or Enable and indicate SSL certificate path(Optional)

Update a package to a specific version:

Select Install on Top-Right

Select package click on tick

Mark for update

Mark for specific version installation

Click Apply

itsergiu
  • 662
  • 6
  • 11