python httplib2 certificate verify failed

Question

I have tried everything I can find to get this to work...

I'm working on a plugin for a python-based task program (called GTG). I'm running Gnome on Opensuse Linux.

Code (Python 2.7):

def initialize(self):
    """
    Intialize backend: try to authenticate. If it fails, request an authorization.
    """
    super(Backend, self).initialize()
    path = os.path.join(CoreConfig().get_data_dir(), 'backends/gtask', 'storage_file-%s' % self.get_id())
    # Try to create leading directories that path
    path_dir = os.path.dirname(path)
    if not os.path.isdir(path_dir):
        os.makedirs(path_dir)

    self.storage = Storage(path)
    self.authenticate()

def authenticate(self):
    """ Try to authenticate by already existing credences or request an authorization """
    self.authenticated = False

    credentials = self.storage.get()
    if credentials is None or credentials.invalid == True:
        self.request_authorization()
    else:
        self.apply_credentials(credentials)
        # Request periodic import, avoid waiting a long time
        # self.start_get_tasks()

def apply_credentials(self, credentials):
    """ Finish authentication or request for an authorization by applying the credentials """


    http = httplib2.Http(ca_certs = '/etc/ssl/certs/ca_certs.pem', disable_ssl_certificate_validation=True)

    http = credentials.authorize(http)

    # Build a service object for interacting with the API.
    self.service = build_service(serviceName='tasks', version='v1', http=http, developerKey='AIzaSyAmUlk8_iv-rYDEcJ2NyeC_KVPNkrsGcqU')
    # self.service = build_service(serviceName='tasks', version='v1')
    self.authenticated = True

def _authorization_step2(self, code):
    credentials = self.flow.step2_exchange(code)
    # credential = self.flow.step2_exchange(code)

    self.storage.put(credentials)
    credentials.set_store(self.storage)

    return credentials

def request_authorization(self):
    """ Make the first step of authorization and open URL for allowing the access """

    self.flow = OAuth2WebServerFlow(client_id=self.CLIENT_ID,
        client_secret=self.CLIENT_SECRET,
        scope='https://www.googleapis.com/auth/tasks',
        redirect_uri='http://localhost:8080',
        user_agent='GTG')


    oauth_callback = 'oob'
    auth_uri = self.flow.step1_get_authorize_url(oauth_callback)
    # credentials = self.flow.step2_exchange(code)



    # url = self.flow.step1_get_authorize_url(oauth_callback)
    browser_thread = threading.Thread(target=lambda: webbrowser.open_new(auth_uri))
    browser_thread.daemon = True
    browser_thread.start()

    # Request the code from user
    BackendSignals().interaction_requested(self.get_id(), _(
        "You need to <b>authorize GTG</b> to access your tasks on <b>Google</b>.\n"
        "<b>Check your browser</b>, and follow the steps there.\n"
        "When you are done, press 'Continue'."),
        BackendSignals().INTERACTION_TEXT,
        "on_authentication_step")


def on_authentication_step(self, step_type="", code=""):

    if step_type == "get_ui_dialog_text":
        return _("Code request"), _("Paste the code Google has given you"
                "here")
    elif step_type == "set_text":
        try:

            credentials = self._authorization_step2(code)
        except FlowExchangeError, e:
            # Show an error to user and end
            self.quit(disable = True)
            BackendSignals().backend_failed(self.get_id(), 
                        BackendSignals.ERRNO_AUTHENTICATION)
            return

        self.apply_credentials(credentials)
        # Request periodic import, avoid waiting a long time
        self.start_get_tasks()

The browser window opens up and I am presented with a code from Google. The program opens a small window where I can enter the code from Google.When that happens I get this in the console :

 No handlers could be found for logger "oauth2client.util"
Created new window in existing browser session.
[522:549:0108/063825:ERROR:nss_util.cc(821)] After loading Root Certs, loaded==false: NSS error code: -8018

but the SSL icon is green in Chrome...

then when I submit the code, I get :

    Exception in thread Thread-10:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/site-packages/GTG/backends/backend_gtask.py", line 204, in on_authentication_step
    credentials = self._authorization_step2(code)
  File "/usr/lib/python2.7/site-packages/GTG/backends/backend_gtask.py", line 151, in _authorization_step2
    credentials = self.flow.step2_exchange(code)
  File "/usr/lib/python2.7/site-packages/oauth2client/util.py", line 132, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/oauth2client/client.py", line 1283, in step2_exchange
    headers=headers)
  File "/usr/lib/python2.7/site-packages/httplib2/__init__.py", line 1586, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/lib/python2.7/site-packages/httplib2/__init__.py", line 1328, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/lib/python2.7/site-packages/httplib2/__init__.py", line 1250, in _conn_request
    conn.connect()
  File "/usr/lib/python2.7/site-packages/httplib2/__init__.py", line 1037, in connect
    raise SSLHandshakeError(e)
SSLHandshakeError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)

The file is called backend_gtask.py...

I have tried importing the certificate as stated here : How to update cacerts.txt of httplib2 for Github?

I have tried to disable verification (httplib2.Http(disable_ssl_certificate_validation=True)) as stated all over the web,

I have updated the python packages (which seemed to make things worse)

I have copied ca_certs.pem back and forth between /etc/ssl... and /usr/lib/python2.7/...

When I visit the auth page in a browser, it says the certificate is verified...

What else can I possibly check?

SHORT TEST CODE :

from oauth2client.client import OAuth2WebServerFlow
from oauth2client.tools import run
from oauth2client.file import Storage

CLIENT_ID = 'id'
CLIENT_SECRET = 'secret'

flow = OAuth2WebServerFlow(client_id=CLIENT_ID,
                           client_secret=CLIENT_SECRET,
                           scope='https://www.googleapis.com/auth/tasks',
                           redirect_uri='http://localhost:8080')

storage = Storage('creds.data')

credentials = run(flow, storage)

print "access_token: %s" % credentials.access_token

Found that here: https://github.com/burnash/gspread/wiki/How-to-get-OAuth-access-token-in-console%3F

Please provide more details: code, platform, python version, detailed error messages, site you are trying to access... — Steffen Ullrich, Jan 07 '15 at 11:15
I edited my question adding the missing details. Thanks for your reply. — brainstormtrooper, Jan 08 '15 at 05:47
It is not obvious from your large code, which are the relevant parts for the problem and how to reproduce it. Please make an example which is as small as possible but still makes reproducing the problem possible (maybe just the HTTP request?). — Steffen Ullrich, Jan 08 '15 at 06:33
I added a little test I found on github. It produces the same error... — brainstormtrooper, Jan 08 '15 at 10:03
I've tried your test program on Ubuntu 14.04 with default python (2.7.6) and it is working for me. Which leaves the questions: platform, python version (like 2.7.9, not only 2.7), detailed error messages. — Steffen Ullrich, Jan 08 '15 at 19:25
yes, it's python version 2.7.9. I've also done some more testing. httplib2 0.9-1.2 seems to be at fault. The flow package uses it and the plugin I'm writing does too. If I pass a ca_cert parameter to httplib2 directly in my plugin then some of the errors go away, but the flow step 2 still fails with the error `SSLHandshakeError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)` — brainstormtrooper, Jan 09 '15 at 05:48
2.7.9 enabled certificate validation by default. Earlier versions of python just ignored certificate errors and thus disabled the protection SSL should offer. Looks like you don't have the necessary certificates on your system, so the question remains which OS you are using and where did you got your python installation. — Steffen Ullrich, Jan 09 '15 at 07:05
I'm running openSUSE 13.2 (64bit). All the packages come from official repositories. certs are in /etc/ssl. There is a ca-bundle.pem and a certs folder symlink to the system ssl package. I manually added the google cert to the .pem using instructions mentioned above... I also directly passed the path to the ca-bundle.pem to the httplib constructor. Now I get this error from ssl.py : `File "/usr/lib64/python2.7/ssl.py", line 507, in __init__ self._context.load_verify_locations(ca_certs) IOError: [Errno 2] No such file or directory` — brainstormtrooper, Jan 09 '15 at 08:15
'No such file or directory' the path you have given must be wrong. — Steffen Ullrich, Jan 09 '15 at 08:25
duh... indeed, fixed the path and now it works... So, now that I have done so much fiddling with the code, is there not a risk that the "plugin" is no longer portable? is it normal to have to create an http object in advance to give it to google client library and to ssl? Will older versions of httplib not work with the plugin anymore?... anyway, Thanks LOADS for your help... — brainstormtrooper, Jan 09 '15 at 08:44
I would suggest you make a small example of just httplib2 trying to access the google URL with no explicit cafile setting. Since python 2.7.9 should use the system CA path and such a common root certificate should be in this path it would be a bug in SUSE 13.2 if this fails. I've tried on Ubuntu with a fresh install of python 2.7.9 and it works there as expected without any fiddling with cafile. — Steffen Ullrich, Jan 09 '15 at 10:40

score 2 · Answer 1 · edited May 23 '17 at 11:52

OK... Big thanks to Steffen Ullrich.

httplib2 version 0.9 tries to use the system certificates and not the certs.txt file that used to be shipped with it. It also enforces verification.

httplib2 can take a couple of useful parameters - notably ca_certs. Use it to point to the actual *.pem file in you ssl installation. I cannot be a folder, must be a real file.

I use the following in the initialization of the plugin :

self.http = httplib2.Http(ca_certs = '/etc/ssl/ca-bundle.pem')

Then, for all subsequent calls to httplib or google client libraries, I pass my pre-built http object as a parameter like this:

credentials = self.flow.step2_exchange(code, self.http)

self.http = credentials.authorize(self.http)

Now ssl connections work with the new httplib2...

I will eventually have to make sure the plugin can find certificates on any system, but at least I know what the problem was.

Thanks again to Steffen Ullrich for walking me through this.

I'm using mac 10.15.4 (19E266), this worked for me. – Nishant Nawarkhede May 21 '20 at 15:37 — Nishant Nawarkhede, May 21 '20 at 15:37

score 0 · Answer 2 · answered Aug 26 '22 at 21:21

0

See this answer for an easier fix without touching your code: just set your certificate bundle pem file path in an environment variable:

export HTTPLIB2_CA_CERTS="\path\to\your\ca-bundle"

answered Aug 26 '22 at 21:21

Amit Naidu

2,494
2
24
32

python httplib2 certificate verify failed

2 Answers2

Linked