python: streaming httpRequest / loading website partially

Question

I would like to know if there is a way in python to "stream" httpRequests in order to avoid loading the whole page.

What I´m currently doing to get the html data of a given url is this:

req = urllib2.Request(url)
response = urllib2.urlopen(req)
return response.read()

This way I´m always loading the whole website, but since I only need a small part of it I´m using more bandwith then I need to. If I could stop loading the website after I found a specific value / expression, or even better if I could specify where to start / end loading the website eg. starting at character #3000 loading until #5000 I´d save a lot of bandwith.

thanks in advance tschery

score 1 · Accepted Answer · edited May 23 '17 at 10:32

1

This stackoverflow answer shows how to do partial HTTP loading in Python. You can also use response.read(N) (N being the number of bytes to read) but there is no guarantee that the exact amount you specify is downloaded.

edited May 23 '17 at 10:32

Community

1
1

answered Apr 14 '14 at 19:11

Elektito

3,863
8
42
72

figured it out myself by now, but I appreciate your answer and I hope it helps others. – tschery Apr 23 '14 at 16:28

python: streaming httpRequest / loading website partially

1 Answers1