What is the best way to download files via HTTP using .NET?

Question

In one of my application I'm using the WebClient class to download files from a web server. Depending on the web server sometimes the application download millions of documents. It seems to be when there are lot of documents, performance vise the WebClient doesn't scale up well.

Also it seems to be the WebClient doesn't immediately close the connection it opened for the WebServer even after it successfully download the particular document.

I would like to know what other alternatives I have.

Update: Also I noticed that for each and every download WebClient performs the authentication hand shake. I was expecting to see this hand shake once since my application only communicate with a single web server. Shouldn't the subsequent calls of the WebClient reuse the authentication session?

Update: My application also calls some web service methods and for these web service calls it seems to authentication session is reused. Also I'm using WCF to communicate with the web service.

Are you using asynchronous webclient calls or synchronous ones? — call me Steve, Mar 24 '10 at 05:38
I'm using synchronous calls however multiple threads are calling different WebClient calls at the same time. — Shamika, Mar 24 '10 at 07:53
I would suggest testing with only one Webclient and asynchronous calls to avoid the redundant authentication phases. I think it'son purpose that the webclient does not immediately close the connection so you can queue another request. I would see the WebClient abstraction as a Web Browser, within a web browser one started you can iniate multiple request without re-authenticate. however if you close your browser and re-open another one each time you want to download a file (even if you share the cookies) it might get slow. — call me Steve, Mar 25 '10 at 13:33

score 5 · Answer 1 · answered Mar 24 '10 at 05:40

5

I think you can still use "WebClient". However, you are better off using the "using" block as a good practice. This will make sure that the object is closed and is disposed off:-

using(WebClient client = new WebClient()) {
// Use client
}

answered Mar 24 '10 at 05:40

Ashish Gupta

14,869
20
75
134

I don't think that I gain anything by calling Dispose method for WebClient. Dispose method is just there because WebClient inherits from Component class. – Shamika Mar 24 '10 at 07:56
4

@Shamika: do it anyway. What if a future version decides to take advantage of IDisposable? Don't depend on implementation details. – John Saunders Mar 24 '10 at 09:40

score 2 · Answer 2 · answered Mar 24 '10 at 05:53

I bet you are running into the default limit of 2 connections per server. Try running this code at the beginning of your program:

var cme = new System.Net.Configuration.ConnectionManagementElement();
cme.MaxConnection = 100;
System.Net.ServicePointManager.DefaultConnectionLimit = 100;

score 0 · Answer 3 · edited May 23 '17 at 12:07

I have noticed the same behavior with the session in another project I was working on. To solve this "problem" I did use a static CookieContainer (since the session of the client is recognized by a value saved in a cookie).

public static class SomeStatics
{ 
    private static CookieContainer _cookieContainer;
    public static CookieContainer CookieContainer
    {
         get
         {
             if (_cookieContainer == null)
             {
                 _cookieContainer = new CookieContainer();
             }
            return _cookieContainer;
         }
    }
}

public class CookieAwareWebClient : WebClient
{ 
    protected override WebRequest GetWebRequest(Uri address)
    {
        WebRequest request = base.GetWebRequest(address);
        if (request is HttpWebRequest)
        {
            (request as HttpWebRequest).CookieContainer = SomeStatics.CookieContainer;
            (request as HttpWebRequest).KeepAlive = false;
        }
        return request;
    }
}

//now the code that will download the file
using(WebClient client = new CookieAwareWebClient())
{
    client.DownloadFile("http://address.com/somefile.pdf", @"c:\\temp\savedfile.pdf");
}

The code is just an example and inspired on Using CookieContainer with WebClient class and C# get rid of Connection header in WebClient.

The above code will close your connection immediately after the file is download and it will reuse the authentication.

I tried and still I'm seeing the WebClient authentication handshake for each download. FYI I'm communicating with a SharePoint server with integrated windows authentication. — Shamika, Mar 26 '10 at 08:16
What about: client.Credentials = new NetworkCredential("username", "password"); ? — Joop, Mar 26 '10 at 09:11

score -1 · Answer 4 · answered Mar 24 '10 at 05:19

WebClient is probably the best option. It doesn't close the connection straight away for a reason: so it can use the same connection again, without having to open a new one. If you find that it's not reusing the connection as expected, that's usually because you're not Close()ing the response from the previous request:

var request = WebRequest.Create("...");
// populate parameters

var response = request.GetResponse();
// process response

response.Close(); // <-- make sure you don't forget this!

I'm using WebClient.DownloadData() to download content. Not sure whether I need to explicitly close any response object here. I suppose the method DownloadData do it for me internally. — Shamika, Mar 24 '10 at 07:51

What is the best way to download files via HTTP using .NET?

4 Answers4