How to make pipenv install use SSL certificate of firewall?

Question

Sitting behind a very strict firewall with SSL decryption, I usually install python packages (on macOS 10.15.) with these options pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org <packagename>. But pipenv install --trusted-host pypi.org --trusted-host files.pythonhosted.org <packagename> doesn't work:

pipenv.vendor.requirementslib.exceptions.RequirementError: Failed parsing requirement from '--trusted-host'

Since ignoring SSL didn't work, I tried to place the certificate of the firewall into a folder and set REQUESTS_CA_BUNDLE=/path/to/company/certificates.pem but without success (maybe I did it wrong). User @Shanti made a promising comment in this question, but I don't know how he accomplished feeding the certificate to pipenv.

So on the bottom line I am looking for a way to make pipenv use my firewall's certificate.

EDIT: here's the output when running pipenv install:

Creating a virtualenv for this project…
Pipfile: /Users/admin/Code/test/Pipfile
Using /Users/admin/.pyenv/versions/3.8.6/bin/python3.8 (3.8.6) to create virtualenv…
⠧ Creating virtual environment...created virtual environment CPython3.8.6.final.0-64 in 404ms
  creator CPython3Posix(dest=/Users/admin/.local/share/virtualenvs/test-NSydZlln, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/admin/Library/Application Support/virtualenv)
    added seed packages: pip==20.2.4, setuptools==50.3.2, wheel==0.35.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

✔ Successfully created virtual environment! 
Virtualenv location: /Users/admin/.local/share/virtualenvs/test-NSydZlln
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Building requirements...
Resolving dependencies...
✘ Locking Failed! 
Traceback (most recent call last):
  File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/vendor/urllib3/contrib/pyopenssl.py", line 488, in wrap_socket
    cnx.do_handshake()
  File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/OpenSSL/SSL.py", line 1934, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/OpenSSL/SSL.py", line 1671, in _raise_ssl_error
    _raise_current_error()
  File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/vendor/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/vendor/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/vendor/urllib3/connectionpool.py", line 976, in _validate_conn
    conn.connect()
  File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/vendor/urllib3/connection.py", line 361, in connect
    self.sock = ssl_wrap_socket(
  File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/vendor/urllib3/util/ssl_.py", line 377, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/vendor/urllib3/contrib/pyopenssl.py", line 494, in wrap_socket
    raise ssl.SSLError("bad handshake: %r" % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/vendor/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/vendor/urllib3/connectionpool.py", line 724, in urlopen
    retries = retries.increment(
  File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/vendor/urllib3/util/retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/wheel/json (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/resolver.py", line 807, in <module>
    main()
  File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/resolver.py", line 802, in main
    _main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
  File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/resolver.py", line 785, in _main
    resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages)
  File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/resolver.py", line 746, in resolve_packages
    results, resolver = resolve(
  File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/resolver.py", line 728, in resolve
    return resolve_deps(
  File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/utils.py", line 1378, in resolve_deps
    results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
  File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/utils.py", line 1096, in actually_resolve_deps
    results = resolver.clean_results()
  File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/utils.py", line 1002, in clean_results
    collected_hashes = self.collect_hashes(ireq)
  File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/utils.py", line 885, in collect_hashes
    r = session.get(pkg_url, timeout=10)
  File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/vendor/requests/sessions.py", line 543, in get
    return self.request('GET', url, **kwargs)
  File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/vendor/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/vendor/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/vendor/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/wheel/json (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))

if you want the package and installed it fast, just a trick, download package from the pypi and install it locally — sahasrara62, Oct 29 '20 at 11:57
try to see if this helps you https://github.com/pypa/pipenv/issues/2110 — BeeFriedman, Oct 29 '20 at 12:41
@CForce99 I placed the certificate of the firewall into a folder and set `REQUESTS_CA_BUNDLE=/path/to/company/certificates.pem` but without success. User @Shanti made a promising comment here https://stackoverflow.com/questions/40684543/how-to-make-python-use-ca-certificates-from-mac-os-truststore#comment-111589453, but I don't know how to accomplish his hint. — Madamadam, Oct 29 '20 at 22:02
@Madamadam unfortunately don't seem to understand what he did.... — BeeFriedman, Oct 29 '20 at 22:47

Stephan Schlecht · Accepted Answer · 2020-11-02T15:21:13.150

As already stated in the comments, setting the environment variable would solve the problem. It should look like this:

export REQUESTS_CA_BUNDLE=/path/to/certificates.pem

Complete Chain

In certificates.pem you must have a complete chain that includes the root certificate. Therefore certificates.pem should look like this:

-----BEGIN CERTIFICATE-----
MII... 
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MII... 
-----END CERTIFICATE-----
...

You can split the file into single files with suffix .pem including the begin and end marker like so:

-----BEGIN CERTIFICATE-----
MII... 
-----END CERTIFICATE-----

In Finder you can now select the individual .pem files, enter <alt> + <tab> so that you can see the contents of each certificate. The chain must be complete, e.g. you should find the corresponding signing certificate for each certificate when you look in the 'Issuer' section under 'Common Name'.

If one or more are missing, use the Keychain Access application (/Applications/Utilities/) to search for the certificate with the missing 'Common Name', export the cert in .PEM format and simply append the resulting file to the end of your certificates.pem file.

Test

Locally tested like this:

setting a HTTPS proxy (in this case Charles)
save the Charles certificate in a .pem file
try to call pipenv install requests (or any other package), it fails with a SSLCertVerificationError
set REQUESTS_CA_BUNDLE environment variable
call pipenv install requests again -> works

Screenshot

due to your step-by-step guide, I realised, that macOS Catalina uses zsh shell and I set environment variable in bash — now everything is working. Thanks! — Madamadam, Nov 02 '20 at 16:33

How to make pipenv install use SSL certificate of firewall?

1 Answers1

Linked