unbuffered urllib2.urlopen

Question

I have client for web interface to long running process. I'd like to have output from that process to be displayed as it comes. Works great with urllib.urlopen(), but it doesn't have timeout parameter. On the other hand with urllib2.urlopen() the output is buffered. Is there a easy way to disable that buffer?

Very similar question at http://stackoverflow.com/questions/107705/python-output-buffering — synthesizerpatel, Oct 08 '10 at 08:40
@synthesizerpatel: well, urlopen() returns object with file-like interface, but it's not a file. — vartec, Oct 08 '10 at 08:51

score 0 · Answer 1 · answered Apr 30 '13 at 17:07

urllib2 is buffered when you just call read()

you could define a size to read and therefore disable buffering.

for example:

import urllib2

CHUNKSIZE = 80

r = urllib2.urlopen('http://www.python.org')
while True:
    chunk = r.read(CHUNKSIZE)
    if not chunk:
        break
    print(chunk)

this would print the response after each chunk is read from the socket, not buffer until the entire response is received.

score 0 · Accepted Answer · answered Oct 08 '10 at 08:23

0

A quick hack that has occurred to me is to use urllib.urlopen() with threading.Timer() to emulate timeout. But that's only quick and dirty hack.

answered Oct 08 '10 at 08:23

vartec

131,205
36
218
244

unbuffered urllib2.urlopen

2 Answers2

Related