0

Currently I use TIdHTTP in Delphi 11 to automatically download updates of my software. My installer is relatively big (about 100 MB) and the download takes a couple of minutes or even more.

Supposing I have the same installation file on different servers, is there a way to use all these servers to improve the download speed, something like Torrent does?

AmigoJack
  • 5,234
  • 1
  • 15
  • 31
delphirules
  • 6,443
  • 17
  • 59
  • 108
  • Where is the bottleneck? The server that is server is bg the file, or your domestic broadband. If it's the latter then all you can do is get better broadband. Find the bottleneck. – David Heffernan Nov 30 '22 at 04:09
  • 1
    One can also start multiple downloads from the **same** server if it only limits bandwidth per download instead of per requesting client: download 3 parts parallelly and have 3 times the speed, should the client have such a big capacity. More to read: [Difference between Content-Range and Range headers?](https://stackoverflow.com/q/716680/4299358) and [HTTP Range header](https://stackoverflow.com/q/3303029/4299358) and [What byte range 0- means](https://stackoverflow.com/q/65063701/4299358) and [HTTP range request for last byte](https://stackoverflow.com/q/39697023/4299358) – AmigoJack Nov 30 '22 at 09:17
  • 1
    and [Does a HTTP resource that accepts range requests always specify content-length?](https://stackoverflow.com/q/27083155/4299358) as well as all questions tagged [HTTP Range](https://stackoverflow.com/questions/tagged/http-range) and [HTTP Content-Length](https://stackoverflow.com/questions/tagged/http-content-length). – AmigoJack Nov 30 '22 at 09:20
  • @DavidHeffernan The bottleneck is the server, because it's a shared server, that's why i want to try the 'torrent' method – delphirules Nov 30 '22 at 11:03

1 Answers1

2

Torrent works by downloading separate pieces of a file from multiple sources in parallel, and then putting the pieces together into the final file.

You can do that with TIdHTTP too, if all of the servers (or even just 1 server) support the HTTP Range request header. So, for example, you could download the file in 1KB chunks by downloading byte ranges 0-1023, 1024-2047, 2048-3071, and so on until the final chunk.

If your server(s) support that, then you can do the following:

  1. Create a file on disk, and presize it to the total size of the final file (see How do you pre-allocate space for a file in C/C++ on Windows?).
  2. For each piece of the file you want to download:
    • Create a TIdHTTP.
    • Create a TFileStream 1 to the final file, requesting and sharing read/write access/rights, and then seek it to the desired start offset for the piece in the file.
    • Download the piece into the TFileStream, setting the TIdHTTP.Request.Range property to 'bytes=<start>-<end>', where start is the starting offset, and end is the ending offset, of the piece in the file.

1 UPDATE: Oh wait, I forgot that TIdHTTP (more specifically, TIdIOHandler.ReadStream()) resizes the given TStream to the size of the data being downloaded, if that size is reported by the server (which it would be in this situation). You DON'T want that to happen when you have already presized the target file ahead of time, otherwise it will get truncated/corrupted if you download multiple TFileStreams into the same file. So, you can't use a standard TFileStream here. What you could do, though, is derive a new class from TFileStream and override its virtual SetSize() methods to do nothing. That should work, I think.

Alternatively:

  1. For each piece you want to download:
    • Create a TIdHTTP.
    • Create a TFileStream (a standard TFileStream is fine here) to a separate temp file, requesting write-only access and sharing no rights, and do not seek it.
    • Download the piece to the TFileStream, setting TIdHTTP.Request.Range as described above.
  2. When all pieces are downloaded, create a new file on disk and copy the bytes of each temp file, in order, into this final file (see How do I concatenate two files into one?), and delete the temp files.
Remy Lebeau
  • 555,201
  • 31
  • 458
  • 770
  • In all the years I never read or came up with the term "presize" to accurately describe that a file should reserve more space than it's about to write at that time (see [How do you pre-allocate space for a file in C/C++ on Windows?](https://stackoverflow.com/q/7970333/4299358)) – AmigoJack Nov 30 '22 at 09:26
  • Oh, wait, I forgot the presize approach will run into a problem when using a standard `TFileStream`. I have updated my answer to address that. – Remy Lebeau Nov 30 '22 at 18:19