python requests to google.com throwing SSLError

Question

While trying to use an old script I had for using the Google Drive python API, the authentication started to give me SSL errors. After a bit of investigation I narrowed the cause down to either the python requests module, or pyOpenSSL which it uses. I am able to show the problem with just this code:

import requests

requests.get('https://google.com/')

The resulting error is, in full:

Traceback (most recent call last):
  File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\urllib3\contrib\pyopenssl.py", line 444, in wrap_socket
    cnx.do_handshake()
  File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\OpenSSL\SSL.py", line 1907, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\OpenSSL\SSL.py", line 1639, in _raise_ssl_error
    _raise_current_error()
  File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\OpenSSL\_util.py", line 54, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\urllib3\connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\urllib3\connectionpool.py", line 849, in _validate_conn
    conn.connect()
  File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\urllib3\connection.py", line 356, in connect
    ssl_context=context)
  File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\urllib3\util\ssl_.py", line 359, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\urllib3\contrib\pyopenssl.py", line 450, in wrap_socket
    raise ssl.SSLError('bad handshake: %r' % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\requests\adapters.py", line 445, in send
    timeout=timeout
  File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\urllib3\connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\urllib3\util\retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='google.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\User\Documents\Email2notes\requtest.py", line 3, in <module>
    requests.get('https://google.com/')
  File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\requests\api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\requests\api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\requests\sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\requests\sessions.py", line 622, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\requests\adapters.py", line 511, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='google.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))

I think I can assume that Google's certs are ok, and the code is a single line, so I think the exception must be caused by one of the modules. I have seen other issues of this type that reccommend to update them (for example). However I already am using the latest version of requests and urllib3 and pyOpenSSL (updated them using pip), I can't update them any further, so how might I fix this?

I have requests[security] installed as is suggested elsewhere. Also suggested there is to downgrade to an older version of certifi, but pip could not find the version suggested.

My python version is 3.5.0 and I'm working on Windows 10. Note that this is not a duplicate question; in other questions with this problem the issue was with the website's SSL certs. If there is a problem with Google's SSL certs that is beyond me.

Also, using verify=False does obviate this error for this snippet of code. However in my actual code (using google-auth-oauthlib which itself uses requests) there is nowhere for me to apply that - I just call e.g. google_auth_oauthlib.flow.run_local_server(), per here.

Is it somehow related to this? https://stackoverflow.com/questions/50246084/django-paypalrestsdk-error-openssl-ssl-error-ssl-routines-tls-process-s — Julien Cochennec, Sep 06 '18 at 13:07
Very possibly. Python indeed lists my OpenSSL version as `'OpenSSL 1.0.2d 9 Jul 2015'`. But even after installing version 1.0.2p and version 1.1.0i, and updating several environment variables, Python won't recognise them; it still gives that 1.0.2d version. Any idea how I could fix that? Any answer I can find talks about OS X and brew, not much use to me. — Cycloneblaze, Sep 06 '18 at 15:53
I eventually updated python itself to 3.6.6. Now my OpenSSL install is reporting as `'OpenSSL 1.0.2o 27 Mar 2018'`, which should be new enough. But the original SSL error is still happening in the same way, so I think we can rule out OpenSSL's version as a cause. — Cycloneblaze, Sep 06 '18 at 23:30

score 1 · Accepted Answer · answered Sep 08 '18 at 00:36

I figured it out! TL;DR: my antivirus was doing some HTTPS/TLS filtering.

I had a look at the answer that @Juilen Cochennec linked in his comment, again, and noticed it suggested using openssl directly from the commendline to examine what certificate chain was actually being returned.

So I ran

openssl s_client -showcerts -connect googleapis.com:443

and I did not get the certificate chain I expected; the first intermediate certificate was not issued by Google but by Bullguard. It was

i:/C=GB/ST=Hounslow/L=Heathrow/O=BullGuard Ltd./OU=DevelTeam/CN=BullGuard SSL Proxy CA

Bullguard is the name of my antivirus, so I suspected that it was somehow blocking my requests. After some research I found the precise cause, namely, Bullguard's Safe Browsing feature. It scans websites and blocks them if it thinks they are unsafe.

A subset of this is safe results, which shows checkmarks (or X's) next to results on search engines, such as Google. It appears to use HTTPS/TLS filtering to do this; it inserts itself between you and your search results so that it can modify the page with the checkmarks. Looking now, this was not even secure in Firefox; the checkmarks don't display because it can't get the safe browsing results.

The upshot is that this filtering was causing SSL errors across all google domains, which was tripping up requests. To fix all I had to do was disable safe results for Google; not even safe browsing itself, just safe results. This disabled the filtering and my requests now work again.

I hope that this helps anyone else with this problem!

P.S. I found this useful to see what the SSL certificate chain was supposed to be, which helped me to identify that it was wrong.

python requests to google.com throwing SSLError

1 Answers1