Pandas read_csv from web acting different between Python 3.8 and 3.10?

Question

It seems that pandas is acting differently when trying to read a CSV from the web between Python 3.8 and Python 3.10. It works with 3.8, but appears to fail with 3.10.

I am using the call:

w1 = pd.read_csv("https://www.ncei.noaa.gov/data/local-climatological-data/access/2021/72254013904.csv")

To try to download weather data from the NOAA website. It works with Python 3.8:

But the same thing fails with Python 3.10:

The full error output is here:

>>> w1 = pd.read_csv("https://www.ncei.noaa.gov/data/local-climatological-data/access/2021/72254013904.csv", low_memory=False)
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 1348, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 975, in send
    self.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 1454, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 512, in wrap_socket
    return self.sslsocket_class._create(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 1070, in _create
    self.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 1341, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jrhodes/Library/Python/3.10/lib/python/site-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/Users/jrhodes/Library/Python/3.10/lib/python/site-packages/pandas/io/parsers/readers.py", line 680, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/Users/jrhodes/Library/Python/3.10/lib/python/site-packages/pandas/io/parsers/readers.py", line 575, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/Users/jrhodes/Library/Python/3.10/lib/python/site-packages/pandas/io/parsers/readers.py", line 933, in __init__
    self._engine = self._make_engine(f, self.engine)
  File "/Users/jrhodes/Library/Python/3.10/lib/python/site-packages/pandas/io/parsers/readers.py", line 1217, in _make_engine
    self.handles = get_handle(  # type: ignore[call-overload]
  File "/Users/jrhodes/Library/Python/3.10/lib/python/site-packages/pandas/io/common.py", line 670, in get_handle
    ioargs = _get_filepath_or_buffer(
  File "/Users/jrhodes/Library/Python/3.10/lib/python/site-packages/pandas/io/common.py", line 339, in _get_filepath_or_buffer
    with urlopen(req_info) as req:
  File "/Users/jrhodes/Library/Python/3.10/lib/python/site-packages/pandas/io/common.py", line 239, in urlopen
    return urllib.request.urlopen(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 519, in open
    response = self._open(req, data)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 1391, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 1351, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>

Did something change in the SSL perhaps between 3.8 and 3.10?

@S3DEV it looks like ```Python 3.8``` has ```urllib3 1.26.6``` and ```Python 3.10``` has ```urllib3 1.26.8``` installed. Installing ```urllib3 1.26.6``` for ```Python 3.10``` did not fix the problem. — joshdr83, Feb 13 '22 at 17:32

score 1 · Answer 1 · answered Jun 01 '22 at 09:38

I've just run into this too. There is indeed increased security in Python 3.10. I've opened a bug report, but in the meantime a workaround based on this post is:

import ssl
from urllib.request import urlopen

import pandas as pd

url = ("https://www.ncei.noaa.gov/"
       "data/local-climatological-data/"
       "access/2021/72254013904.csv")

context = ssl.create_default_context()
context.set_ciphers("DEFAULT")
result = urlopen(url, context=context)
df = pd.read_csv(result)

Pandas read_csv from web acting different between Python 3.8 and 3.10?

1 Answers1