urrlib Error - SSL: CERTIFICATE_VERIFY_FAILED - in CONDA environment

Question

I am running a mac and using a conda environment, trying to replicate results from a research paper that uses a script with urllib.

urllib.request.urlretrieve()

fails, with error: URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)>

Why this error? Is it due to the target site? Trying to download from this lab:

https://taylor0.biology.ucla.edu/birdDBQuery/

and even tried to check that website here: https://www.digicert.com/help/

but I see an error:

Error: taylor0.biology.ucla.edu/birdDBQuery/ is not a fully qualified public domain name or public IP address.

Cannot tell if that is the culprit.

Looking on stack overflow, they hints to run the Install certificate.command script which is in /Application/python[3.your version] folder:

urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)

I did it and did it as well for the folder where my conda env is.

I doubled checked and ensure the script has the correct folder in the beginning (see below).

I also tried with ip install certifi and conda -c install certifi

All of the above says it is fine:

Requirement already satisfied: certifi in ./Sites/miniconda3/lib/python3.9/site-packages (2022.12.7)
 -- removing any existing file or link
 -- creating symlink to certifi certificate bundle
 -- setting permissions
 -- update complete

but no, the running urllib.request.urlretrieve() from my jupyter notebook still fails.

How to solve this??

Should I replace urlretrieve with requests library ?

--

Install certificate.command

--

#!/bin/sh

/Users/me/Sites/miniconda3/bin/python3.9  << "EOF"

# install_certifi.py
#
# sample script to install or update a set of default Root Certificates
# for the ssl module.  Uses the certificates provided by the certifi package:
#       https://pypi.org/project/certifi/

import os
import os.path
import ssl
import stat
import subprocess
import sys

STAT_0o775 = ( stat.S_IRUSR | stat.S_IWUSR | stat.S_IXUSR
             | stat.S_IRGRP | stat.S_IWGRP | stat.S_IXGRP
             | stat.S_IROTH |                stat.S_IXOTH )

def main():
    openssl_dir, openssl_cafile = os.path.split(
        ssl.get_default_verify_paths().openssl_cafile)

    print(" -- pip install --upgrade certifi")
    subprocess.check_call([sys.executable,
        "-E", "-s", "-m", "pip", "install", "--upgrade", "certifi"])

    import certifi

    # change working directory to the default SSL directory
    os.chdir(openssl_dir)
    relpath_to_certifi_cafile = os.path.relpath(certifi.where())
    print(" -- removing any existing file or link")
    try:
        os.remove(openssl_cafile)
    except FileNotFoundError:
        pass
    print(" -- creating symlink to certifi certificate bundle")
    os.symlink(relpath_to_certifi_cafile, openssl_cafile)
    print(" -- setting permissions")
    os.chmod(openssl_cafile, STAT_0o775)
    print(" -- update complete")

if __name__ == '__main__':
    main()
EOF

---

EDIT

Answer was closed but it took me 1h to figure out how to solve. Thanks for the hint anyway.

For anyone in troubles:

1. Go to https://www.ssllabs.com/ssltest/analyze.html?d=[TARGET_SITE]

2. Open the Certification path section, and download the chain certificate. There is a light grey icon for download.

3. Save the certificate. It might be a .pem. If so, 'mycertificate.pem'

4. Now if you use requests

   requests.get(TARGET_URL, verify='path/to/mycertificate.pem')

If using urllib:

import ssl
# add self_signed cert
myssl = ssl.create_default_context()
myssl.load_verify_locations("../../mycertificate.pem")
# send request
response = urllib.request.urlopen(TARGET_URL,context=myssl)