Python3.6.8 urllib.request.urlopen Error "Connection reset by peer"

Question

I am trying to fetch geocode information from maps.googleapis.com using urllib.request.urlopen.

I am facing strange issue where urllib.request.urlopen is working with proxy server enabled but not working without proxy.

I am getting connection reset error during ssl hadnshake,

Traceback (most recent call last):
  File "/usr/lib64/python3.6/urllib/request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/usr/lib64/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/lib64/python3.6/http/client.py", line 974, in send
    self.connect()
  File "/usr/lib64/python3.6/http/client.py", line 1415, in connect
    server_hostname=server_hostname)
  File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket
    _context=self, _session=session)
  File "/usr/lib64/python3.6/ssl.py", line 773, in __init__
    self.do_handshake()
  File "/usr/lib64/python3.6/ssl.py", line 1033, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib64/python3.6/ssl.py", line 645, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/usr/lib64/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/usr/lib64/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib64/python3.6/urllib/request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib64/python3.6/urllib/request.py", line 1320, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 104] Connection reset by peer>

I was getting similar issue with requests module but when I updated it as below I am able to get response using requests.get method,

pip install requests[security]

I checked with our Network Security team and all the IPs for maps.googleapis.com are whitelisted.

I checked with cURL command and its working fine as well.

Do anyone know what is the issue with the urllib.request.urlopen ?

I am using python 3.6.8 version.

I think it has something to do with user agent, try changing that https://bugs.python.org/issue21896 — Jatin Mehrotra, Sep 03 '20 at 14:54

Jatin Mehrotra · Answer 1 · 2020-09-03T15:52:18.873

0

1)an auto-bot detection mechanism is most likely dropping your connection. You should provide a User-Agent header to fake a browser visit.

headers={'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36'})

While installing requests library it skips few of optional security packages ('pyOpenSSL', 'ndg-httpsclient', and 'pyasn1') which are required for the SSL/Https connection. You can fix it by either running this command
```
pip install "requests[security]"
```

or

pip install pyopenssl ndg-httpsclient pyasn1

if it doesn't work

You can (sometimes) correct this with a time.sleep(0.01) placed strategically.

Refer this https://stackoverflow.com/a/383816/13126651

edited Sep 03 '20 at 15:52

answered Sep 03 '20 at 14:54

Jatin Mehrotra

9,286
4
28
67

I had tried it; but unfortunately even it's not working. I am thinking there is issue with SSL verification but I am not getting how to fix it. – Swapnil Dengale Sep 03 '20 at 15:38
You said you are working with maps.google.api, I think its authentication problem, because it requires API key, hence SSL issue. – Jatin Mehrotra Sep 03 '20 at 15:45
Updated my answer, please check – Jatin Mehrotra Sep 03 '20 at 15:52
1

Thanks Jatin, yes, as I had mentioned in question 'pip install "requests[security]"' solves problem with requests module. But it is not solving issue with urllib.requests.openurl. Also as per many answers I had tried installing pyopenssl ndg-httpsclient pyasn1 and that also didn't work. I am not using any threads so time.sleep wont be helpful. Also the API key is correct as the same URL I use works in browers/cURL or with urllib.requests.openurl with proxy. It is only failing with urllib.requests.openurl when request going through firewall (even though IP is whitelisted). – Swapnil Dengale Sep 03 '20 at 17:00
What if you use this https://github.com/googlemaps/google-maps-services-python and if you want to stick with urlib usage of it https://docs.python.org/3.1/howto/urllib2.html – Jatin Mehrotra Sep 03 '20 at 17:23
Yeah, I didn't want to change existing code which is using urllib. It was working fine few days back and suddenly it stopped working. So wanted to check if someone has faced similar issues because of some firewall or network settings. I checked from one other server and same code works fine there. So code wise there seems to be no issue. If I don't get to know whats the exact issue I will use requests module (Same is used in google maps client). – Swapnil Dengale Sep 03 '20 at 19:38

Python3.6.8 urllib.request.urlopen Error "Connection reset by peer"

1 Answers1