23

Is there a way to download huge and still growing file over HTTP using the partial-download feature?

It seems that this code downloads file from scratch every time it executed:

import urllib
urllib.urlretrieve ("http://www.example.com/huge-growing-file", "huge-growing-file")

I'd like:

  1. To fetch just the newly-written data
  2. Download from scratch only if the source file becomes smaller (for example it has been rotated).
Paul D. Waite
  • 96,640
  • 56
  • 199
  • 270
Konstantin
  • 6,061
  • 8
  • 40
  • 48

3 Answers3

44

It is possible to do partial download using the range header, the following will request a selected range of bytes:

req = urllib2.Request('http://www.python.org/')
req.headers['Range'] = 'bytes=%s-%s' % (start, end)
f = urllib2.urlopen(req)

For example:

>>> req = urllib2.Request('http://www.python.org/')
>>> req.headers['Range'] = 'bytes=%s-%s' % (100, 150)
>>> f = urllib2.urlopen(req)
>>> f.read()
'l1-transitional.dtd">\n\n\n<html xmlns="http://www.w3.'

Using this header you can resume partial downloads. In your case all you have to do is to keep track of already downloaded size and request a new range.

Keep in mind that the server need to accept this header for this to work.

Nadia Alramli
  • 111,714
  • 37
  • 173
  • 152
  • 2
    Also you have to check Content-Range header (it may differ from range you requested) and probably be ready to parse multipart/byteranges body. – Denis Otkidach Nov 26 '09 at 15:22
  • 2
    Checked into the multipart/byteranges aspect. The spec explicitly disallows multipart/byteranges responses to a single range request. – David Blevins Aug 31 '10 at 05:32
  • 2
    For retrieving the rest from a position on (a typical case), just use `"bytes=%d-"` (i. e. just without the end value). – Alfe May 10 '12 at 13:12
2

This is quite easy to do using TCP sockets and raw HTTP. The relevant request header is "Range".

An example request might look like:

mysock = connect(("www.example.com", 80))
mysock.write(
  "GET /huge-growing-file HTTP/1.1\r\n"+\
  "Host: www.example.com\r\n"+\
  "Range: bytes=XXXX-\r\n"+\
  "Connection: close\r\n\r\n")

Where XXXX represents the number of bytes you've already retrieved. Then you can read the response headers and any content from the server. If the server returns a header like:

Content-Length: 0

You know you've got the entire file.

If you want to be particularly nice as an HTTP client you can look into "Connection: keep-alive". Perhaps there is a python library that does everything I have described (perhaps even urllib2 does it!) but I'm not familiar with one.

Conrad Meyer
  • 2,851
  • 21
  • 24
  • 1
    If you roll your own solution with TCP sockets then you end up getting rid of all the features in urllib2, like following redirects and handling proxy settings. – Andrew Dalke Nov 26 '09 at 07:45
  • Absolutely. I didn't remember urllib2 supported setting arbitrary Request headers. It is (of course) the right way to go here. – Conrad Meyer Nov 26 '09 at 08:56
-1

If I understand your question correctly, the file is not changing during download, but is updated regularly. If that is the question, rsync is the answer.

If the file is being updated continually including during download, you'll need to modify rsync or a bittorrent program. They split files into separate chunks and download or update the chunks independently. When you get to the end of the file from the first iteration, repeat to get the appended chunk; continue as necessary. With less efficiency, one could just repeatedly rsync.

mpez0
  • 2,815
  • 17
  • 12