SSL: CERTIFICATE_VERIFY_FAILED with Python3

Question

I apologize if this is a silly question, but I have been trying to teach myself how to use BeautifulSoup so that I can create a few projects.

I was following this link as a tutorial: https://www.youtube.com/watch?v=5GzVNi0oTxQ

After following the exact same code as him, this is the error that I get:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 1240, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1083, in request
    self._send_request(method, url, body, headers)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1128, in _send_request
    self.endheaders(body)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1079, in endheaders
self._send_output(message_body)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 911, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 854, in send
    self.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1237, in connect
server_hostname=server_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 376, in wrap_socket
_context=self)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 747, in __init__
self.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 983, in do_handshake
    self._sslobj.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 628, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "WorldCup.py", line 3, in <module>
    x = urllib.request.urlopen('https://www.google.com')
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 162, in urlopen
    return opener.open(url, data, timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 465, in open
    response = self._open(req, data)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 483, in _open
'_open', req)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 443, in _call_chain
    result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 1283, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 1242, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED]     certificate verify failed (_ssl.c:645)>

Can someone help me figure out how to fix this?

What URL are you trying? Does the host use a self-signed certificate? — Jason S, Feb 23 '16 at 05:04
www.google.com, like he did in the video. My code is the same as his. — PafflesWancakes, Feb 23 '16 at 05:11
I get the same error with any other website that I use, i.e. youtube.com, wikipedia.com etc — PafflesWancakes, Feb 23 '16 at 05:13
Maybe http://stackoverflow.com/questions/35275540/sslerror-certificate-verify-failed-for-every-domain-url — Jason S, Feb 23 '16 at 05:15
I got the same error as earlier. It's not working "/ I know it was workinge arlier because I was able to use it on another website. — PafflesWancakes, Feb 23 '16 at 05:21

score 181 · Answer 1 · edited May 25 '23 at 03:27

181

In my case, I used the ssl module to "workaround" the certification like so:

import ssl

ssl._create_default_https_context = ssl._create_unverified_context

Then to read your link content, you can use:

urllib.request.urlopen(urllink)

Note that whilst this will make the code work, it removes security protections.

edited May 25 '23 at 03:27

falsePockets

3,826
4
18
37

answered Mar 08 '18 at 13:31

Jia

2,417
1
15
25

2

Great solution. Worked and helped me figure out what was going on in the first place. – Aus_10 May 02 '18 at 17:55
worked perfectly well, i couldn't find the "Install Certificates.command", but this solution worked perfectly well for certificate issue. – Sumit Kumar Saha Jan 27 '19 at 12:01
1

NameError: name 'urllib' is not defined – Brian D Feb 17 '19 at 20:55
@BrianD check your version of Python. `urllib` import path has changed in recent versions of Python. – Mikaeil Orfanian Feb 18 '19 at 16:08
This worked for me as well. I wasn't able to find the "Install Certificates.command" file either. I'm on Windows. – jerbear Mar 01 '19 at 16:21
Works with pandas too. – markroxor May 21 '19 at 03:15
For a weird reason certificates didn't work for me, but this workaround did the job. – Noohone Jan 21 '20 at 21:19
@Jia I face this error when trying to upload my package to artifactory. Where should I add these two lines since my package has many modules? – Adrian Mar 05 '21 at 06:50
3

How do I get the page content from `urllib.request.urlopen(urllink)` ? In other words, what is equivalent of `requests.get(MY_URL).content`? – Willy Mar 17 '21 at 23:53
Worked for me - easy fix. – user7611503 Sep 20 '21 at 14:44

score 136 · Answer 2 · edited Dec 24 '22 at 20:59

136

Go to the folder where Python is installed, e.g., in my case (Mac OS) it is installed in the Applications folder with the folder name 'Python 3.6'. Now double click on 'Install Certificates.command'. You will no longer face this error.

For those not running a mac, or having a different setup and can't find this file, the file merely runs:

pip install --upgrade certifi

edited Dec 24 '22 at 20:59

starball

20,030
7
43
238

answered May 08 '17 at 19:04

Bhavik

1,477
1
9
9

9

What is the significance of the code - what change did it apply - care to explain ? – Naveen Vijay Mar 13 '18 at 01:02
5

To the people saying they can't find this file - this solution is only Apple products. – Kaja Nov 18 '19 at 09:32
Thank you very much! Also it worked on Python3.7 :) – Elinaldo Monteiro Dec 10 '19 at 01:42
This and only this command worked. It installed the right certificates. I have tried numuours and had lost trust with stackoverflwo. – TheOnlyAnil Aug 20 '20 at 15:04

score 35 · Answer 3 · answered Oct 15 '19 at 12:22

35

On Debian 9 I had to:

$ sudo update-ca-certificates --fresh
$ export SSL_CERT_DIR=/etc/ssl/certs

I'm not sure why, but this enviroment variable was never set.

answered Oct 15 '19 at 12:22

Dan Walters

1,218
1
18
30

2

Thank you, I faced the same issue on Raspbian Stretch after manually installing Python 3.8, and solved setting that environment variable. – Robert Hume Oct 17 '19 at 18:55
3

It might not even be necessary to update certificates - in my case, just exporting the path worked. – Kaja Nov 18 '19 at 09:30
I started having this problem after installing a new pip package. As @Kaja mentioned just exporting the path worked for me as well on Ubuntu 20.04. – Brett K. Sep 26 '20 at 19:32
The 2nd line worked on Mac OS Big Sur for me. – TrigonaMinima Oct 08 '21 at 20:53
This worked for me on Ubuntu20.04 after Dropbox was failing to install. Exactly what I was looking for. – Tony Barganski Mar 27 '22 at 21:48
thanks. export SSL_CERT_DIR=/etc/ssl/certs did the trick on ubuntu 20.04 – mondjunge Apr 06 '22 at 10:14

score 7 · Answer 4 · answered Oct 26 '21 at 14:05

7

This has changed in recent versions of the ssl library. The SSLContext was moved to it's own property. This is the equivalent of Jia's answer in Python 3.8

import ssl

ssl.SSLContext.verify_mode = ssl.VerifyMode.CERT_OPTIONAL

answered Oct 26 '21 at 14:05

deltree

3,756
1
30
51

'SSLContext' object attribute 'verify_mode' is read-only – Özgün Jan 11 '22 at 17:26
Whilst I wish there was a better way than using the [protected member in Jia's answer](https://stackoverflow.com/a/49174340/866333), your new answer doesn't substitute for it. In python 3.8 I receive: `urllib.error.URLError: – John Jan 24 '22 at 12:43
I posted this answer because Jia's answer didn't work for me and this did. – deltree Jan 26 '22 at 18:28
Stack Overflow is a wiki. Instead of writing a new answer that references another answer, you should just edit that other answer to say "if version x, do y, else do z". – falsePockets May 25 '23 at 03:29

score 4 · Answer 5 · answered Mar 07 '22 at 12:06

4

As a workaround (not secure), you can turn certificate verification off by setting PYTHONHTTPSVERIFY environment variable to 0:

export PYTHONHTTPSVERIFY=0

answered Mar 07 '22 at 12:06

Adil

2,418
7
34
38

score 3 · Answer 6 · answered Jan 24 '22 at 13:49

Building on the update to Jia's 2018 answer in deltree's late 2021 one I was able to achieve equivalent functionality with:

import urllib.request
import ssl


def urllib_get_2018():
    # Using a protected member like this is not any more fragile
    # than extending the class and using it. I would use it.
    url = 'https://localhost:6667/my-endpoint'
    ssl._create_default_https_context = ssl._create_unverified_context
    with urllib.request.urlopen(url = url) as f:
        print(f.read().decode('utf-8'))


def urllib_get_2022():
    # Finally! Able to use the publice API. Happy happy!
    url = 'https://localhost:6667/my-endpoint'
    scontext = ssl.SSLContext(ssl.PROTOCOL_TLS)
    scontext.verify_mode = ssl.VerifyMode.CERT_NONE
    with urllib.request.urlopen(url = url, context=scontext) as f:
        print(f.read().decode('utf-8'))

I needed to use CERT_NONE instead of CERT_OPTIONAL as well as creating a ssl.SSLContext(ssl.PROTOCOL_TLS) to pass to urlopen.

It's important to keep using urllib as it makes sense when working with small container images where pip might not be installed, yet.

score 2 · Answer 7 · answered Sep 11 '19 at 14:25

2

When you are using a self signed cert urllib3 version 1.25.3 refuses to ignore the SSL cert

To fix remove urllib3-1.25.3 and install urllib3-1.24.3

pip3 uninstall urllib3

pip3 install urllib3==1.24.3

Tested on Linux MacOS and Window$

answered Sep 11 '19 at 14:25

AAber

1,562
10
14

i tried this but it did not work for me (still getting the same error). In my case I think the certificates are valid but are not being used. – szeitlin Sep 20 '19 at 21:26
If your target has a valid certificate you don't need this fix. To check if you site has a valid certificate run: `curl https://target.web.site/` If you get a message "SSL certificate problem: self signed certificate" you have a self signed certificate on your target. If you get a proper answer from the site then the certificate is valid. – AAber Sep 22 '19 at 13:47
Yeah the error I'm seeing is that my local machine's certificates are not being used. The target's certificates are fine. – szeitlin Sep 23 '19 at 17:42

score 1 · Answer 8 · answered Jan 10 '21 at 09:35

I have a lib what use https://requests.readthedocs.io/en/master/ what use https://pypi.org/project/certifi/ but I have a custom CA included in my /etc/ssl/certs.

So I solved my problem like this:

# Your TLS certificates directory (Debian like)
export SSL_CERT_DIR=/etc/ssl/certs
# CA bundle PATH (Debian like again)
export CA_BUNDLE_PATH="${SSL_CERT_DIR}/ca-certificates.crt"
# If you have a virtualenv:
. ./.venv/bin/activate
# Get the current certifi CA bundle
CERTFI_PATH=`python -c 'import certifi; print(certifi.where())'`

test -L $CERTFI_PATH || rm $CERTFI_PATH
test -L $CERTFI_PATH || ln -s $CA_BUNDLE_PATH $CERTFI_PATH

Et voilà !

Thanks! This seems like the best solution - I'm using using python 3.11. — djchapm, Jul 13 '23 at 23:34

score 1 · Answer 9 · answered Dec 01 '21 at 07:40

I faced the same issue with Ubuntu 20.4 and have tried many solutions but nothing worked out. Finally I just checked openssl version. Even after update and upgrade, the openssl version showed OpenSSL 1.1.1h [22 Sep 2020]. But in my windows system, where the code works without any issue, openssl version is OpenSSL 1.1.1k 25 Mar 2021.

I decided to update the openssl manually and it worked! Thank God!!!

Steps are as follows(Ubuntu 20.4):

*To check openssl version

openssl version -a

*To update openssl:

sudo apt install build-essential checkinstall zlib1g-dev

cd /usr/local/src/

sudo wget https://www.openssl.org/source/openssl-1.1.1k.tar.gz

sudo tar -xf openssl-1.1.1k.tar.gz

cd openssl-1.1.1k

sudo ./config --prefix=/usr/local/ssl --openssldir=/usr/local/ssl shared zlib
sudo make
sudo make test
sudo make install

cd /etc/ld.so.conf.d/
sudo nano openssl-1.1.1k.conf

*Type /usr/local/ssl/lib and save

sudo ldconfig -v
sudo nano /etc/environment

*Add ':/usr/local/ssl/bin' to the path

source /etc/environment
echo $PATH

*Now check openssl version

openssl version -a

score 0 · Answer 10 · answered Aug 20 '22 at 07:30

0

you might exec command: pip install --upgrade certifi or you might opened charles/fiddler, just close it

answered Aug 20 '22 at 07:30

yaqin2015

36
3

score -2 · Answer 11 · answered Feb 11 '20 at 20:13

-2

I had this problem in MacOS, and I solved it by linking the brew installed python 3 version, with

brew link python3

After that, it worked without a problem.

answered Feb 11 '20 at 20:13

Kostas Tsiligkiris

49
4

1

While just installing python3 via Brew may resolve this issue, it does not address the root cause. FYI for anyone checking out this answer. – A. Weatherwax Apr 21 '21 at 19:19

SSL: CERTIFICATE_VERIFY_FAILED with Python3

11 Answers11

Linked