29

I'm trying to connect to my corporate's internal webpages through the requests package, but since python does not use the windows default trusted certificates the connection is denied. I found out that wincertstore can be used to fetch the windows default certificates. But I'm still not sure how to use that along with the my request. Below is the code I have tried so far.............

import requests, socket, atexit, ssl, wincertstore
from requests.auth import HTTPBasicAuth
certfile = wincertstore.CertFile()
certfile.addstore("CA")
certfile.addstore("ROOT")
atexit.register(certfile.close)
ssl_sock = ssl.wrap_socket(s,ca_certs=certfile.name, 
cert_reqs=ssl.CERT_REQUIRED)
requests.get(url)

I get the following error................... requests.exceptions.SSLError: HTTPSConnectionPool(host='myhost', port=443): Max retries exceeded with url: myurl (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))

I am able to use wget on the same url and download the content.

wget --no check certificate --user=my username --password=my password URL

But I am not interested in downloading the content as I only need to scrape a small portion of the webpage content.

Pythin version = 3.6.5

Wincertstore link - Link

Thanks in advance for your help..............

adimessi30
  • 415
  • 1
  • 4
  • 9
  • Please add the `wget` command that works, and a link to where you found this `wincertstore` code, and which Python version you're using. (If `wincertstore` is [this project](https://pypi.org/project/wincertstore/), it's just a backport of the code in Python 3.4+ to 2.3-3.3, which hasn't been updated since 3.4 came out, so I doubt it would be much help for a problem in 3.6, but in 2.7 it might.) – abarnert May 20 '18 at 19:13
  • Edit that all into the question, not in comments. – abarnert May 20 '18 at 19:27
  • Anyway, your `wget` command isn't using your Windows certs, it's _ignoring_ them, and not doing any validation. So (a) do you know your certs are installed correctly in the first place? And (b) have you tried telling `requests` to ignore validation as well? If so, show us the code, and what happened. And also, tell us whether making that work would be acceptable. – abarnert May 20 '18 at 19:29
  • @abarnert yes if I could make requests ignore validation and return the pages HTML it is completely acceptable......I will post the request code as soon as possible..... – adimessi30 May 20 '18 at 19:34

3 Answers3

56

I had a similar issue and fixed it using the python-certifi-win32 package (now out of maintenance):

As of 2022 (as mentioned by Briareos386 in the comments)

pip install pip-system-certs

Original answer (out of maintenance)

pip install python-certifi-win32

now you can just use:

requests.get(url, verify=True)

and the certificate is checked using the Windows Certificate Store.

Note: This only works if the certificate is installed in the Windows Certificate Store...

sourcream
  • 210
  • 1
  • 11
clfaster
  • 1,450
  • 19
  • 26
  • 3
    or with anaconda saving grace `conda install -c conda-forge python-certifi-win32` – imbr Jun 03 '20 at 17:17
  • pip reported "Requirement already satisfied: certifi..." but running setup.py must have somehow fixed an issue as my SSLCertVerificationError went away. – Alias_Knagg Nov 02 '20 at 09:30
  • 1
    This solved my problem where the cert I needed was in the win cert store (python 3.7.8) – Faustin Carter Jan 05 '21 at 23:06
  • 1
    After hours of searching, this still seems to be the most straightforward solution. Contrary to what [this answer says](https://stackoverflow.com/a/50438960/119527), I was unable to see any sort of fallback-to-OS behavior. And I didn't want to modify the 3rd-party code, so [this answer](https://stackoverflow.com/a/50215614/119527) didn't work for me. Thank Guido for monkey-patching! – Jonathon Reinhart Feb 25 '22 at 21:45
  • 2
    Unfortunately `python-certifi-win32` seems to be [out-of-maintenance](https://stackoverflow.com/a/72647960/172525). `pip-system-certs` (by the same author) works for me as a drop-in replacement very well. – Briareos386 Jun 16 '22 at 15:11
  • 1
    Adding to what @Briareos386 said, `python-certifi-win32` currently breaks `pip` if you are using Python 3.10 on Windows. – sourcream Jul 01 '22 at 12:52
11

This is all explained in the SSL Cert Verification section of the requests docs.

By default, requests uses the certs from certifi if present, falling back to whatever urllib3 thinks is your OS cert store, which itself falls back on whatever Python thinks it is (although in older versions it often didn't).

Your company apparently has a private, maybe even self-signed, cert, which isn't going to be in certifi. It might be in the Windows cert store—in which case urllib3 should automatically pick it up—but I suspect that it isn't. Maybe the cert is installed directly into some custom browser setup your IT department forces you to use, instead of into the OS store. Or maybe it's not installed at all. (You didn't mention being able to access this site in a browser without seeing a broken-lock icon…)

You're passing --no check certificate (or, more likely, --no-check-certificate?) to wget, so you're just not verifying SSL. And if you want to do the same thing in requests, that's just:

requests.get(url, verify=False)

If you're pretty sure that you do have the cert installed, even though wget can't find it… well, your code isn't going to work as written. Here's what would work:

  • Ignore the cert and just disable validation, as shown above.
  • Figure out where the relevant cert actually is installed and how to load it, and:
    • Pass it as the verify argument on every requests call.
    • Set it up somewhere statically and pass it in an environment variable.
    • Install it into your default cert store so everything works automatically.
    • Write an HTTPAdapter that installs it into your requests session.

First, your code is just trying to get the default cert in exactly the same way Python already does. That wincertstore module is just a backport of what's already builtin to Python 3.4+.

Second, all your code is doing is getting a cert, using it to create an SSL socket, ignoring that socket, and telling requests to do its normal thing. That isn't going to help anything. If you want to pass a cert to requests, you either do this:

requests.get(url, verify='/path/to/cert')

… or put it in the environment variable REQUESTS_CA_BUNDLE

… or do the HTTPAdapter code that I showed you in chat (and which you found an old, non-working version of somewhere unspecified). See HTTPAdapter in the docs if you actually want to do that.

abarnert
  • 354,177
  • 51
  • 601
  • 671
  • 2
    The link to the `requests` documentation is now dangling. Perhaps https://requests.readthedocs.io/en/master/user/advanced/#ssl-cert-verification would be a suitable replacement, but it doesn’t have any of the impressive detail here about `urllib3` and wotnot. – Martin Dorey Aug 01 '20 at 02:42
  • SSL doc link no longer works. – nate Aug 05 '22 at 11:54
1

In my case (Windows 10 + Python 3.10.2 + Elasticsearch 8.0.1)

When I ran the code below

requests.get('https://192.168.1.3:9200')

I got this error

Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:997)')))

I tried previous solutions but none of them worked for me. I could fix the problem after adding Kibana's CA SSL certificate into Python's default certificate store.

  1. Kibana's CA SSL can be found in your Kibana config file (kibana.yml > elasticsearch.ssl.certificateAuthorities)
  2. Python's default certificate store can be found with this Python code certifi.where()
  3. Then copy the Kibana's CA certificate content and paste it into the cacert.pem file.

Then it worked for me.

ihsany
  • 870
  • 10
  • 12