112

When using Python 2.7 with urllib2 to retrieve data from an API, I get the error [Errno 104] Connection reset by peer. Whats causing the error, and how should the error be handled so that the script does not crash?

ticker.py

def urlopen(url):
    response = None
    request = urllib2.Request(url=url)
    try:
        response = urllib2.urlopen(request).read()
    except urllib2.HTTPError as err:
        print "HTTPError: {} ({})".format(url, err.code)
    except urllib2.URLError as err:
        print "URLError: {} ({})".format(url, err.reason)
    except httplib.BadStatusLine as err:
        print "BadStatusLine: {}".format(url)
    return response

def get_rate(from_currency="EUR", to_currency="USD"):
    url = "https://finance.yahoo.com/d/quotes.csv?f=sl1&s=%s%s=X" % (
        from_currency, to_currency)
    data = urlopen(url)
    if "%s%s" % (from_currency, to_currency) in data:
        return float(data.strip().split(",")[1])
    return None


counter = 0
while True:

    counter = counter + 1
    if counter==0 or counter%10:
        rateEurUsd = float(get_rate('EUR', 'USD'))

    # does more stuff here

Traceback

Traceback (most recent call last):
  File "/var/www/testApp/python/ticker.py", line 71, in <module>
    rateEurUsd = float(get_rate('EUR', 'USD'))
  File "/var/www/testApp/python/ticker.py", line 29, in get_exchange_rate
    data = urlopen(url)
  File "/var/www/testApp/python/ticker.py", line 16, in urlopen
    response = urllib2.urlopen(request).read()
  File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 406, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 438, in error
    result = self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 625, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python2.7/urllib2.py", line 406, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 438, in error
    result = self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 625, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python2.7/urllib2.py", line 400, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 418, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1207, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1180, in do_open
    r = h.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1030, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 407, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 365, in _read_status
    line = self.fp.readline()
  File "/usr/lib/python2.7/socket.py", line 447, in readline
    data = self._sock.recv(self._rbufsize)
socket.error: [Errno 104] Connection reset by peer
error: Forever detected script exited with code: 1
Athena Wisdom
  • 6,101
  • 9
  • 36
  • 60
  • On arch linux, get_rate works fine for me. Are you sure you aren't being filtered? Can you load that url in a browser? – korylprince Dec 13 '13 at 14:00
  • @korylprince It works fine in a browser, and the script runs fine for a while before the error starts appearing. If I can't avoid the error, how should the error be handled so that it does not crash, and probably use the most recent value retrieved? – Athena Wisdom Dec 13 '13 at 14:02

4 Answers4

155

"Connection reset by peer" is the TCP/IP equivalent of slamming the phone back on the hook. It's more polite than merely not replying, leaving one hanging. But it's not the FIN-ACK expected of the truly polite TCP/IP converseur. (From other SO answer)

So you can't do anything about it, it is the issue of the server.

But you could use try .. except block to handle that exception:

from socket import error as SocketError
import errno

try:
    response = urllib2.urlopen(request).read()
except SocketError as e:
    if e.errno != errno.ECONNRESET:
        raise # Not error we are looking for
    pass # Handle error here.
Community
  • 1
  • 1
Bunyk
  • 7,635
  • 8
  • 47
  • 79
  • 8
    Is it true that the administrator of the server usually uses this method to block the potential scraping requests by a client or is it more likely to be just an unintentional bug? Now I wonder if I'm blocked intentionally or not... – Blaszard Dec 28 '16 at 19:34
  • 1
    In some cases, it may happen due to a bug in another part of the system. In my case, there were many CLOSE_WAIT tcp connections in my server side and the number was bigger than server application could provide (java allows max 50 connections at a time). So my server side application was just denying new connection attempts by resetting the connection after approximately 50 CLOSE_WAIT hung connections. – u.unver34 Feb 18 '20 at 10:20
  • not a native english speaker. What does "slamming the phone back on the hook." mean? – Charlie Parker Sep 21 '20 at 22:49
  • 7
    @Pinocchio I'm also not native speaker, but this also exists in my language and refers to obsolete phone technology: https://en.wikipedia.org/wiki/Telephone_hook With old phones, when you pick up a phone, the hook automatically turned on the connection. When you put the phone back on the hook - it disconnected. So basically in modern language, it means "to press disconnect/red button, leave the call". – Bunyk Sep 22 '20 at 11:31
  • so if I have control over the server and the client--what can I do to fix this on the server end? Thank you! – Mike Sandstrom Mar 17 '21 at 18:07
  • @MikeSandstrom I think it very much depends on your server (maybe it's process got terminated because of some unexpected error), so it's better to ask a separate questions about this, providing more information about your server and request you are sending to it, etc... – Bunyk Mar 17 '21 at 19:04
  • no question privileges until next month sadly – Mike Sandstrom Mar 17 '21 at 19:48
20

You can try to add some time.sleep calls to your code.

It seems like the server side limits the amount of requests per timeunit (hour, day, second) as a security issue. You need to guess how many (maybe using another script with a counter?) and adjust your script to not surpass this limit.

In order to avoid your code from crashing, try to catch this error with try .. except around the urllib2 calls.

Alexander
  • 2,925
  • 3
  • 33
  • 36
Felix Martinez
  • 339
  • 2
  • 3
5

There is a way to catch the error directly in the except clause with ConnectionResetError, better to isolate the right error. This example also catches the timeout.

from urllib.request import urlopen 
from socket import timeout

url = "http://......"
try: 
    string = urlopen(url, timeout=5).read()
except ConnectionResetError:
    print("==> ConnectionResetError")
    pass
except timeout: 
    print("==> Timeout")
    pass
Cyril
  • 157
  • 2
  • 2
  • 1
    This is not a correct answaer because pass an exception is like put to silent an critical error in your code. Reference to the Most Diabolic Anti Pattern in Pythn article https://realpython.com/the-most-diabolical-python-antipattern/ – Franz Kurt Mar 16 '22 at 16:47
3

there are 2 solution you can try.

  1. request too frequently. try sleep after per request
time.sleep(1)
  1. the server detect the request client is python, so reject. add User-Agent in header to handle this.
    headers = {
        "Content-Type": "application/json;charset=UTF-8",
        "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)"
    }
    try:
        res = requests.post("url", json=req, headers=headers)
    except Exception as e:
        print(e)
        pass

the second solution save me

geosmart
  • 518
  • 4
  • 15