Creating a Download Accelerator

Question

I am referring to this article to understand file downloads using C#.

Code uses traditional method to read Stream like

((bytesSize = strResponse.Read(downBuffer, 0, downBuffer.Length)) > 0

How can I divide a file to be downloaded into multiple segments, so that I can download separate segments in parallel and merge them?

using (WebClient wcDownload = new WebClient())
{
    try
    {
        // Create a request to the file we are downloading
        webRequest = (HttpWebRequest)WebRequest.Create(txtUrl.Text);
        // Set default authentication for retrieving the file
        webRequest.Credentials = CredentialCache.DefaultCredentials;
        // Retrieve the response from the server
        webResponse = (HttpWebResponse)webRequest.GetResponse();
        // Ask the server for the file size and store it
        Int64 fileSize = webResponse.ContentLength;

        // Open the URL for download 
        strResponse = wcDownload.OpenRead(txtUrl.Text);
        // Create a new file stream where we will be saving the data (local drive)
        strLocal = new FileStream(txtPath.Text, FileMode.Create, FileAccess.Write, FileShare.None);

        // It will store the current number of bytes we retrieved from the server
        int bytesSize = 0;
        // A buffer for storing and writing the data retrieved from the server
        byte[] downBuffer = new byte[2048];

        // Loop through the buffer until the buffer is empty
        while ((bytesSize = strResponse.Read(downBuffer, 0, downBuffer.Length)) > 0)
        {
            // Write the data from the buffer to the local hard drive
            strLocal.Write(downBuffer, 0, bytesSize);
            // Invoke the method that updates the form's label and progress bar
            this.Invoke(new UpdateProgessCallback(this.UpdateProgress), new object[] { strLocal.Length, fileSize });
        }
    }

FYI, using `UpdateProgressCallback` looks like this is WinForms code. — John Saunders, Sep 19 '12 at 16:44
@Dan-o: but it would not be consistent with the "asp.net" tag. Hence the question. — John Saunders, Sep 19 '12 at 18:28
@JohnSaunders: dunno how I missed that. I scanned the tags for winforms, didnt see it and just missed the aspnet tag. Silliness. — Sam Axe, Sep 19 '12 at 19:05
If you have one NIC, what improvement will multiple connections likely make? It's one thing to keep track of chunks of data so that a download may be re-started but doing more than one thing through a single pipe often doesn't accelerate performance. — Peter Ritchie, Sep 20 '12 at 19:12
@PeterRitchie Speaking from abstract perspective only, this is what commercial download accelarator has been doing. i.e divide files into multiple segment and download them parallely and merge them at client pc. — Abhijeet, Sep 21 '12 at 14:31
@autrevo and the consensus is that download accelerators don't work http://bit.ly/SeOWgy — Peter Ritchie, Sep 21 '12 at 15:26

score 1 · Accepted Answer · answered Sep 20 '12 at 19:07

you need several threads to accomplish that. first you start the first download thread, creating a webclient and getting the file size. then you can start several new thread, which add a download range header. you need a logic which takes care about the downloaded parts, and creates new download parts when one finished.

http://msdn.microsoft.com/de-de/library/system.net.httpwebrequest.addrange.aspx

I noticed that the WebClient implementation has sometimes a strange behaviour, so I still recommend implementing an own HTTP client if you really want to write a "big" download program.

ps: thanks to user svick

Creating a Download Accelerator

1 Answers1

Linked