Using urllib gives SSL error

Question

Running a request with urllib but continuously receive this error:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1400, in connect
    server_hostname=server_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 814, in __init__
    self.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1068, in do_handshake
    self._sslobj.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<pyshell#15>", line 1, in <module>
    products = amazon.search_n(1, Keywords='kindle', SearchIndex='All')
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/amazon/api.py", line 288, in search_n
    return list(islice(items, n))
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/amazon/api.py", line 544, in __iter__
    for page in self.iterate_pages():
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/amazon/api.py", line 561, in iterate_pages
    yield self._query(ItemPage=self.current_page, **self.kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/amazon/api.py", line 573, in _query
    response = self.api.ItemSearch(ResponseGroup=ResponseGroup, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/bottlenose/api.py", line 274, in __call__
    {'api_url': api_url, 'cache_url': cache_url})
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/bottlenose/api.py", line 235, in _call_api
    return urllib2.urlopen(api_request, timeout=self.Timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1320, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)>

Any ideas how this can be resolved? Evidently the SSL verification is not working correctly, but I do not know how to fix the issue.

Using Python 3.6.1 on Mac Os

Thanks

score 0 · Answer 1 · answered Jan 02 '18 at 23:00

0

I had this issue not so long ago. Try giving the following commands a go:

pip install requests[security]

If that doesn't work, try the following:

pip install pyOpenSSL
pip install cryptography
pip install idna

answered Jan 02 '18 at 23:00

JulieWinnf

21
1

I’ll give this a go and let you know the outcome – ProCode Jan 02 '18 at 23:13

score 0 · Answer 2 · answered Jan 02 '18 at 23:04

Try the requests module. I am using a windows, so try the mac equivalent of a pip install. With Python 3.6, I believe that urllib has been having some issues. If you are trying to webscrape, the requests module is better for Python 3, than the urllib module.

Below is an example from the other day when I was webscraping Wikipedia for stock tickers:

          response = requests.get("https://en.wikipedia.org/wiki/List_of_S26P_500_companies")
          soup = bs4.BeautifulSoup(response.text)

I got the url by using the requests module, and then was able to work with the text using beautiful soup. From what I gathered, urllib is to be avoided at all times for Python 3+ and is more suitable for Python 3. Try to do exactly what you're doing, but via the requests module instead of urllib.

I actually prefer to use requests for that very reason. But I need to use an API for Amazon and the only way I know how to is to use pre-existing code for the API which uses urllib. If by any chance you know of any other ways to work with the Amazon API it would be helpful. — ProCode, Jan 02 '18 at 23:13

Using urllib gives SSL error

2 Answers2

Linked