How to change this code to download file bigger than 2GB?

Question

I'm using this code it works good for smaller files but fails to download files with filesize over 2GB. I have tried to use webclient as well but it does not suit my code well or not working as this code works just trying to figure out how to download 2GB files with this one. Thanks

System.Net.HttpWebRequest request = (System.Net.HttpWebRequest)System.Net.WebRequest.Create(url_);
//request.Proxy = WebRequest.GetSystemWebProxy();
System.Net.HttpWebResponse response = (System.Net.HttpWebResponse)request.GetResponse();
ServicePoint sp = request.ServicePoint;
sp.ConnectionLimit = MAX_THREADS;
response.Close();
// gets the size of the file in bytes
Int64 iSize = response.ContentLength;
// keeps track of the total bytes downloaded so we can update the progress bar
Int64 iRunningByteTotal = 0;
UpdateStatus("File OK", count);
// use the webclient object to download the file
using (System.Net.WebClient client = new System.Net.WebClient())
{
    // open the file at the remote URL for reading
    using (System.IO.Stream streamRemote = client.OpenRead(new Uri(VideoUrl)))
    {
        // using the FileStream object, we can write the downloaded bytes to the file system
        using (Stream streamLocal = new FileStream(sFilePathToWriteFileTo, FileMode.Create, FileAccess.Write, FileShare.None))
        {
            // loop the stream and get the file into the byte buffer
            int iByteSize = 0;
            byte[] byteBuffer = new byte[iSize];<---------throws error here
            while ((iByteSize = streamRemote.Read(byteBuffer, 0, byteBuffer.Length)) > 0)
            {
                if (isCanceled == true) return;
                if (manualResetEvent.WaitOne(0, false)) return;
                // write the bytes to the file system at the file path specified
                streamLocal.Write(byteBuffer, 0, iByteSize);
                iRunningByteTotal += iByteSize;

                // calculate the progress out of a base "100"
                double dIndex = (double)(iRunningByteTotal);
                double dTotal = (double)byteBuffer.Length;
                double dProgressPercentage = (dIndex / dTotal);
                int iProgressPercentage = (int)(dProgressPercentage * 100);
                UpdateProgress((int)iProgressPercentage, count);
            }

            // clean up the file stream
            streamLocal.Close();
        }
        // close the connection to the remote server
        streamRemote.Close();
    }
}

Exception:

System.OverflowException was caught
  HResult=-2146233066
  Message=Arithmetic operation resulted in an overflow.
  Source=Downloader
  StackTrace:
       at MetroFramework.Demo.frmMain.FileDownloader(Object state) in frmMain.cs:line 595--->byte[] byteBuffer = new byte[iSize];
  InnerException:

Define "fails". Please read [ask] and include all relevant information in your question, and share your research. You can start by not trying to allocate a buffer as large as the file, which kind of beats the purpose of the buffer. — CodeCaster, Jan 10 '16 at 14:25
I bet it fails at `byte[] byteBuffer = new byte[iSize];` with an out of memory exception, since `Int64 iSize = response.ContentLength;`. Try copying the streams in smaller chunks, see [How do I copy the contents of one stream to another?](http://stackoverflow.com/questions/230128/how-do-i-copy-the-contents-of-one-stream-to-another). — dbc, Jan 10 '16 at 14:27
"Fails" is not an error. Please [edit] your question, include the full exception details and **share your research**. You're not the first to encounter an OutOfMemoryException. — CodeCaster, Jan 10 '16 at 14:33
It might me a noob question for many but i don't understand down votes. — m.qayyum, Jan 10 '16 at 14:40
@m.qayyum - I'm not a downvoter, but the reason may be that you didn't include the full `ToString()` output of the exception, or even the exception message. That information would have made answering your question easier, and be useful people who have the same problem in the future -- so why not include it? Also, please give the exception as text, not a screen capture, so future readers with the same problem can find it by searching. — dbc, Jan 10 '16 at 14:42

score 2 · Accepted Answer · answered Jan 10 '16 at 14:33

You seem to have done everything correctly at first glance:

You've allocated a buffer
You've created a loop that fills the buffer
It then flushes the buffer to the file, before it iterates for the next buffer
You've also included running totals and progress updates

Everything... save one thing:

byte[] byteBuffer = new byte[iSize];

This allocates the buffer to be the size of the file to download. In other words, if the file is 1GiB in size, you allocate a 1 GiB buffer, and then you try to fill the whole buffer in one call. This filling may return fewer bytes but you've still allocated the whole buffer. Note that the maximum length of a single array in .NET is a 32-bit number which means that even if you recompile your program for 64bit and actually have enough memory available, it will still fail for files bigger than 2GiB.

So do this:

byte[] byteBuffer = new byte[65536];

Use a 64KiB buffer or some other sane buffer size.

score 2 · Answer 2 · edited May 23 '17 at 11:59

Apart from your buffer problem fixed by @Lasse's answer, the actual problem is that you're reinventing the wheel.

Use WebClient.DownloadFile() and subscribe to the DownloadProgressChanged event:

using (System.Net.WebClient client = new System.Net.WebClient())
{
    client.DownloadProgressChanged += WebClient_DownloadProgressChanged;

    client.DownloadFile(sourceUrl, targetFile);

    client.DownloadProgressChanged -= WebClient_DownloadProgressChanged;
}

private void WebClient_DownloadProgressChanged(object sender, DownloadProgressChangedEventArgs e)
{
    // e.BytesReceived, 
    // e.TotalBytesToReceive,
}

How to change this code to download file bigger than 2GB?

2 Answers2

Linked