Downloading files asynchronously from httpwebrequest throws timeout error

Question

I have the following method to download files from a web address using httpwebrequest. I am downloading 150 files contained in a list. This may take say 30 mins max. When I run my service my webrequests keep timeing out and I'm not sure why? I'm guessing it is creating 150 tasks and processing as many as it can but timing out after the timeout period on some of the tasks. What is going on in my method?

try
{
     //Download File here
     Task t = new Task(() =>
     {
         byte[] lnBuffer;
         byte[] lnFile;
         HttpWebRequest httpWebRequest = (HttpWebRequest)HttpWebRequest.Create(string.Format("{0}/{1}", Settings1.Default.WebPhotosLocation, f.FileName));
//Breaks on the line below
         using (HttpWebResponse httpWebResponse = (HttpWebResponse)httpWebRequest.GetResponse())
         {
              httpWebRequest.Timeout = 14400000;
              httpWebRequest.KeepAlive = true;
              using (BinaryReader binaryReader = new BinaryReader(httpWebResponse.GetResponseStream()))
              {
                    using (MemoryStream memoryStream = new MemoryStream())
                    {
                          lnBuffer = binaryReader.ReadBytes(1024);
                          while (lnBuffer.Length > 0)
                          {
                                memoryStream.Write(lnBuffer, 0, lnBuffer.Length);
                                lnBuffer = binaryReader.ReadBytes(1024);
                          }
                          lnFile = new byte[(int)memoryStream.Length];
                          memoryStream.Position = 0;
                          memoryStream.Read(lnFile, 0, lnFile.Length);
                    }
               }
         }
         using (System.IO.FileStream lxFS = new FileStream(string.Format(@"{0}\{1}", fileDirectory, f.FileName), FileMode.Create))
         {
               lxFS.Write(lnFile, 0, lnFile.Length);
         }
         Log.WriteLine(string.Format("Downloaded File: {0}", f.FullName), Log.Status.Success);
         filesDownloaded++;
     });
     t.Start();
     listOfTasks.Add(t);
}
catch (Exception ex)
{
     Log.WriteLine(string.Format("There has been an error Downloading File {0}. Error: {1}", f.FullName, ex), Log.Status.Error);
     throw;
}

EDIT

Exception thrown at line stated in code:

Exception: An exception of type 'System.Net.WebException' occurred in System.dll but was not handled in user code

Additional information: The operation has timed out

The error is thrown after 3 minutes and 20 seconds after the first download begins

How are you calling this method? Are you sure your server can take 150 concurrent calls? More so, you dont need threads to do IO bound work. — Yuval Itzchakov, Aug 11 '14 at 09:25
I'm calling this method in a foreach loop on my client side. I'm not sure even how to check if my server can take 150 concurrent calls and also why do I not need threads to do IO work? I thought my thread would wait on every download before moving onto the next? — JKennedy, Aug 11 '14 at 09:34
Sidenote: Replace all of that copy algorithm with `Stream.Copy` directly to the `FileStream`. Or: `new WebClient().DownloadFile(url, path)`. — usr, Aug 11 '14 at 10:49
I got my download algorithm from [Here](http://stackoverflow.com/questions/2368115/how-to-use-httpwebrequest-to-pull-image-from-website-to-local-file/2368505#2368505) but I will look at the DownloadFile method. The connection limit is that set server side or client side? 2 minutes sounds about right to when my program times out — JKennedy, Aug 11 '14 at 10:54
This should not time out before the timeout you set. Please find out *what* exactly is timing out. Post the full exception as well as the code location it is happening. Set the debugger to break on all exceptions. — usr, Aug 11 '14 at 10:59
I have now updated my question with the relevant information — JKennedy, Aug 11 '14 at 11:50
You set the timeout before you call GetResponse. Change that, and increase the connection limit to a high number such as 1000000 for testing purposes. — usr, Aug 11 '14 at 11:53
Ok I will change that and get back to you. I also call `Task.WaitAll(listOfTasks.ToArray());` at the end of the program. I was wondering whether this method has a timeout and could be causing the operations to fail? — JKennedy, Aug 11 '14 at 11:58
If that method does not throw it cannot possibly have caused this timeout. The exception tells you what timed out: **Web**Exception. — usr, Aug 11 '14 at 12:03
So turned out the error was exactly what you said. I was setting the timeout after calling `GetResponse()` I put the timeout above the `GetResponse()` call and it works a charm. Thank you — JKennedy, Aug 11 '14 at 13:11

score 0 · Accepted Answer · answered Aug 11 '14 at 13:13

Turns out I was setting the Timeout value after calling GetResponse. Putting the timeout above this method call worked perfectly.

HttpWebRequest httpWebRequest = (HttpWebRequest)HttpWebRequest.Create(string.Format("{0}/{1}", Settings1.Default.WebPhotosLocation, f.FileName));
httpWebRequest.Timeout = 14400000;
httpWebRequest.KeepAlive = true;
using (HttpWebResponse httpWebResponse = (HttpWebResponse)httpWebRequest.GetResponse())
{
    //Do stuff
}

Downloading files asynchronously from httpwebrequest throws timeout error

1 Answers1