How to change the 'cafile' argument in the ssl module in Python3?

Question

I am in the process of porting my Python 2 code to Python 3. This specific section deals with my local machine posting to a webpage when running my code in Python 3 (everything works great in Python 2). I believe the root cause of my issue is the certificate on my side is failing to be authenticated by the server. Below is the error I see.

Error(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1056)')))
request Error: HTTPSConnectionPool(host='my/personal.server', port=443): Max retries exceeded with url: my/personal/url/and/data (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1056)')))

The error occurs on my requests.post(url, headers) line in my program. I've made sure the server URL's are correct.

I believe the issue is as follows. In macOS's native Python 2 environment (what I had used to code before), the ssl module is different than the one located in the directory Python 3 uses for its imports. My questions are:

• Is it possible to use LibreSSL with my Python 3 instance?
• If not, how can I pass the cafile and other path arguments when I import the ssl module in Python 3?

FWIW, I downloaded Python 3 from Python.org.

Python 2

⇒  python
Python 2.7.10 (default, Feb 22 2019, 21:55:15)
[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.37.14)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl, sys
>>> sys.executable
'/usr/bin/python'
>>> ssl.OPENSSL_VERSION
'LibreSSL 2.2.7'
>>> ssl.get_default_verify_paths()
DefaultVerifyPaths(cafile='/private/etc/ssl/cert.pem', capath='/private/etc/ssl/certs', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/private/etc/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/private/etc/ssl/certs')

Python 3

⇒  python3
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 16:52:21)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl, sys
>>> sys.executable
'/usr/local/bin/python3'
>>> ssl.OPENSSL_VERSION
'OpenSSL 1.1.0j  20 Nov 2018'
>>> ssl.get_default_verify_paths()
DefaultVerifyPaths(cafile='/Library/Frameworks/Python.framework/Versions/3.7/etc/openssl/cert.pem', capath=None, openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/Library/Frameworks/Python.framework/Versions/3.7/etc/openssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/Library/Frameworks/Python.framework/Versions/3.7/etc/openssl/certs')

I believe this is the issue why I have this error, though I am not 100% sure. Any other pointers or input would be welcome!

EDIT

Following ivan_pozdeev's suggested duplicate question, I am able to run openssl s_client -connect mywebsite.com:443 -verify 9 and I get Verify return code: 0 (ok). Though I am not directly calling openssl in my python/bash scripts, I have been able to verify my Python 2 configuration works.

⇒  openssl
OpenSSL> version
LibreSSL 2.6.5
OpenSSL> ca
Using configuration from /private/etc/ssl/openssl.cnf

I am looking into how to utilize what I did above with my Python 3 interpreter and its libraries. However, this seem to be more of a test to see if I can talk to the server rather than tackling the arguments/environment issue.

EDIT 2

Perhaps this is a case of looking into this too long, but I have found something strange. The Python 2/3 instances above were done in zsh and not in my program. When I run my script with these print statements, I get this ssl information:

Python 2

import ssl
print ssl.OPENSSL_VERSION
print ssl.get_default_verify_paths()

Output

LibreSSL 2.2.7
DefaultVerifyPaths(cafile='/usr/local/etc/openssl/cert.pem', capath='/private/etc/ssl/certs', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/private/etc/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/private/etc/ssl/certs')

Python 3

import ssl
print(ssl.OPENSSL_VERSION)
print(ssl.get_default_verify_paths())

Output

OpenSSL 1.1.0j  20 Nov 2018
DefaultVerifyPaths(cafile=None, capath=None, openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/Library/Frameworks/Python.framework/Versions/3.7/etc/openssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/Library/Frameworks/Python.framework/Versions/3.7/etc/openssl/certs')

The Python 2 script prints what I would expect. The Python 3 script has my cafile argument set to None, which is different than what ssl.get_default_verify_paths() returns when I run it in my zsh. I use python and python3 aliases in my bash script to call my Python scripts and am absolutely sure they reference the same interpreters as my zsh. What exactly is going on here?

Answer 1

Found out it is possible to set arguments in the environment through the bash shell script I was wrapping around my python script.

export SSL_CERT_FILE=/usr/local/etc/openssl/cert.pem
export REQUESTS_CA_BUNDLE=/usr/local/etc/openssl/cert.pem

I guess my script was just looking at an empty certificate directory before? Regardless, it works now. The path I've written is the location where OpenSSL has its certificates by default.

Answer 2

Problem

I was behind a corporate firewall.

I had SSL errors for boto3, SharePoint Online API, etc (snippet):

SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))

I found out through my network admin (i.e. IT Security) that our Netskope tool uses its own proxy certificates.

Easiest solution

Some possible solutions I found included Duke of CitizenKanedom's bash wrapper, copying the certificates into given folders, or 'verify="path" ' for requests. But I wanted a simple solution that didn't move any system files around or require new dependencies, and would be easy enough to scale up with later Docker container usage. I figured out you can set session environments like so:

import os
os.environ["REQUESTS_CA_BUNDLE"] = 'C:/ProgramData/Netskope/STAgent/download/nscacert.pem'
os.environ["SSL_CERT_FILE"] = 'C:/ProgramData/Netskope/STAgent/download/nscacert.pem'