Python Requests package: lost connection while streaming

Question

I'd like to use the Requests package to connect to the streaming API of a web service. Suppose I use the following code to send a request, receive the response and iterate through the lines of response as they arrive:

import requests

r = requests.get('http://httpbin.org/stream/20', stream=True)
for line in r.iter_lines():
    if line:
        print line

While waiting to receive new data, we are basically waiting for r.iter_lines() to generate a new piece of data. But what if I lose internet connection while waiting? How can I find out so I can attempt to reconnect?

score 3 · Accepted Answer · edited May 23 '17 at 12:22

You can disconnect from your network to have a try. Requests raise such error:

requests.exceptions.ConnectionError: HTTPConnectionPool(host='httpbin.org', port=80): Max retries exceeded with url: /stream/20 (Caused by : [Errno -3] Temporary failure in name resolution)

The error message shows Requests already retries for network error. You can refer to this answer for setting the max_retries. If you wants more customization (e.g. waits between retries), do it in a loop:

import socket
import requests
import time

MAX_RETRIES = 2
WAIT_SECONDS = 5

for i in range(MAX_RETRIES):
    try:
        r = requests.get('http://releases.ubuntu.com/14.04.1/ubuntu-14.04.1-desktop-amd64.iso',
                         stream=True, timeout=10)
        idx = 1
        for chunk in r.iter_content(chunk_size=1024):
            if chunk:
                print 'Chunk %d received' % idx
                idx += 1
        break
    except requests.exceptions.ConnectionError:
        print 'build http connection failed'
    except socket.timeout:
        print 'download failed'
    time.sleep(WAIT_SECONDS)
else:
    print 'all tries failed'

EDIT: I tested with a large file. I used iter_content instead, because it's a binary file. iter_lines is based on iter_content (source codes), so I believe the behaviour is same. Procedure: run the codes with network connected. After receiving some chunks, disconnect. Wait 2-3 seconds, reconnect, the downloading continued. So requests package DOES retry for connection lost in the iteration.

Note: If no network when build the connection (requests.get()), ConnectionError is raised; if network lost in the iter_lines / iter_content, socket.timeout is raised.

Thanks for the reply. I don't think this works. No exception is raised by iter_lines when I disconnect. I think the ConnectionError exception is only raised if the requests.get() call fails. The situation I'm interested in is when the request returns successfully, then iter_lines() begins generating the lines, and *then* the connection is lost. What I see in that situation is that the script simply hangs. No error message, no exceptions raised. — navidoo, Sep 16 '14 at 19:23
I run a test on it. Requests package does retry for connection lost in the iter_lines(). And you can catch `socket.timeout` if retries failed. Please see the updated answer — ZZY, Sep 17 '14 at 03:26
You're right. I upgraded to the latest version and now iter_lines is raising the exception. I think it was missing in the older version I had. — navidoo, Sep 17 '14 at 23:26

Python Requests package: lost connection while streaming

1 Answers1