keep getting SSL error when making a get request in python

Question

I'm making a simple get request as follows:

import requests

response = requests.get('url')

And it errors out with the following output:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 417, in wrap_socket
cnx.do_handshake()
File "/usr/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1426, in do_handshake
self._raise_ssl_error(self._ssl, result)
File "/usr/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1174, in _raise_ssl_error
_raise_current_error()
File "/usr/lib/python3.6/site-packages/OpenSSL/_util.py", line 48, in exception_from_error_queue
raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 594, in urlopen
chunked=chunked)
File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 350, in _make_request
self._validate_conn(conn)
File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 835, in _validate_conn
conn.connect()
File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/connection.py", line 323, in connect
ssl_context=context)
File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/util/ssl_.py", line 324, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 424, in wrap_socket
raise ssl.SSLError('bad handshake: %r' % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 423, in send
timeout=timeout
File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 624, in urlopen
raise SSLError(e)
requests.packages.urllib3.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.6/site-packages/requests/api.py", line 70, in get
return request('get', url, params=params, **kwargs)
File "/usr/lib/python3.6/site-packages/requests/api.py", line 56, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 488, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 609, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 497, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)
>>>

I know that to avoid it I can simply disable verification but it's definitely not the right thing to do and not the way to go, so I'd like to know if there're any other, better workarounds.

I would appreсiate any help!

EDIT:

After running a test on https://www.ssllabs.com I found out that the server's certificate chain is incomplete but I have absolutely no idea how to handle this issue.

score 0 · Answer 1 · answered Jan 18 '17 at 23:06

0

I remember that they have something like:

requests.get(URL, verify='/path/to/certfile')

You can check

http://docs.python-requests.org/en/master/user/advanced/#ssl-cert-verification

to find the proper way to do your cert-verification.

answered Jan 18 '17 at 23:06

Chong

335
2
9

I clean forgot to mention it, but it's not my site that I'm trying to access. I'm just writting a primitive scrapper that is supposed to scrap some info off the site. As far as I understand, first I have to retrieve the certificate and save it in a file, right? I tried doing it with `openssl s_client -connect {HOSTNAME}:{PORT} -showcerts` but it failed saying that 'no peer certificate available' and 'SSL handshake has read 0 bytes and written 0 bytes'... – Albert Jan 18 '17 at 23:27
Um, if you are not testing your own site, I feel things get to be easier. So why not just do `requests.get(url, verify=False)`? If all you gonna do it to get the information on their site, I think you do not need to do the cert-verification unless they require you to log in before getting the information? – Chong Jan 19 '17 at 00:04
Maybe you can take a look at this question http://stackoverflow.com/questions/24457408/openssl-command-to-check-if-a-server-is-presenting-a-certificate or do some more Google on your openssl failure. – Chong Jan 19 '17 at 00:10
@Albert also, if possible, could you tell me which site you are trying to get information from? – Chong Jan 19 '17 at 00:11
well I've googled through a lot of stuff by this moment but unfortunately didn't find a solution...perhaps it has to do with the fact that I don't quite understand how openssl and all the related stuff work. Well from what I've read, setting `verify=False` is not secure and my scrapper will scrap the same piece if info periodically, not just once. Yeah sure, the site is this one https://www.metro-cc.ru – Albert Jan 19 '17 at 08:26
As far as I understand I have to somehow download the missing chain certificate. But where do I find it?!.... – Albert Jan 19 '17 at 18:36
Yes, set `verify=False`is not secure. But for most cases, the security is for yourself (your information will be private when it is sent to this site), but you only want their data, right? If you set `verify=False, does your scrapper work? Have your tried to export the certificate from browser like Chrome? I think this might be a way. – Chong Jan 19 '17 at 19:31
Feel like I can get information from that website if I set `verify=False`. – Chong Jan 19 '17 at 19:52
you mean exporting a cert and then doing something like this `>>> requests.get('https://kennethreitz.org', cert=('/path/client.cert', '/path/client.key'))` But where do I get the private key (aka client.key)?!I'm sorry for a silly question but I'm absolutely new to all this security stuff... Well yes, it is possible to get information from it like this, but I wouldn't like to resort to this method for security reasons. – Albert Jan 19 '17 at 20:12
I am sorry that I am not familiar with the https stuff either. My friend suggests me to take a look at [link](https://gist.github.com/mtigas/952344). Got no idea whether this can help~ I don't disagree with your comment about "security reasons" but I really don't think it is something you need to care about when writing a website scraper to get the data you want. I feel like the most important thing of tools like scraper should be the functionality because you almost have no information to lose when you just copy the data from a website. Life is short so I feel better to make things work first – Chong Jan 19 '17 at 22:45

keep getting SSL error when making a get request in python

1 Answers1