Downloading multiple files WebClient

Question

I'm trying to download multiple files but it's not working as I hoped. Can someone tell me what's wrong with this script, because I've tried a lot of things and really don't know what to do anymore.

public static void DownloadFile(string url)
        {
            WebClient client = new WebClient();
            var name = url.Substring(url.LastIndexOf('/')).Remove(0, 1);
            foreach (var item in urls)
            {
                client.DownloadFile(item, "C:\\" + name);
            }
        }

        private void btnGo_Click(object sender, EventArgs e)
        {
            urls.Add("url1");
            urls.Add("url2");
            urls.Add("url3");
            Parallel.ForEach(urls,
               new ParallelOptions { MaxDegreeOfParallelism = 10 }, 
               DownloadFile);
        }

---

using (var sr = new StreamReader(HttpWebRequest.Create(url).GetResponse().GetResponseStream()))
            {
                using (var sw = new StreamWriter(url.Substring(url.LastIndexOf('/'))))
                {
                    sw.Write(sr.ReadToEnd());
                }
            }

Answer 1

I would use a System.Net.HttpWebRequest instead.

This is what the code would look like:

private List<string> urls = new List<string>();

private void btnGo_Click(object sender, EventArgs e)
{
    urls.Add("http://199.91.152.106/ua0p3fbc5nlg/gg2w2fq4ljc1nnd/MicroCraft_Beta.zip");
    Parallel.ForEach(urls, new ParallelOptions { MaxDegreeOfParallelism = 10 }, DownloadFile);
}

public static void DownloadFile(string url)
{
    var req = (HttpWebRequest)WebRequest.Create(url);
    var name = url.Substring(url.LastIndexOf('/') + 1);
    using (var res = (HttpWebResponse)req.GetResponse())
    using (var resStream = res.GetResponseStream())
    using (var fs = new FileStream("C:\\" + name, FileMode.Create, FileAccess.Write, FileShare.None))
    {
        // Save to file
        var buffer = new byte[8 * 1024]; // 8 KB buffer
        int len; // Read count
        while ((len = resStream.Read(buffer, 0, buffer.Length)) > 0)
            fs.Write(buffer, 0, buffer.Length);
    }
}

Because the URL you told me in the comment isn't using a proper implementation of the HTTP protocol. You'll have to add this to your config file in order for it to work (either App.config or Web.config, depending on if it's an ASP.Net site or offline application):

<system.net>
    <settings>
        <httpWebRequest useUnsafeHeaderParsing="true" />
    </settings>
</system.net>

As to your problem with names colliding which you said in your comment, this should be resolved by changing your var name = url.Substring(url.LastIndexOf('/')).Remove(0, 1); into something else.

If you want to have an incremental filename, you could use this:

// Inside your class:
private static int counter = 0;

// In your method:
var name = "file" + System.Threading.Interlocked.Increment(ref counter) + ".html";

Answer 2

You are downloading all files to the same file in your DownloadFile code which assumes single call to this function downloads all files.

Fixes:

Option 1: Don't use Parallel.ForEach and simply call DownloadFile once. Specify unique file names for each download. I.e. by taking part of Url that you are downloading or simply using random/temporary file names.

Something like this (assuming urls is some sort of IEnumerable<string>)

foreach (var item in urls)
{
   var name = item.Substring(item.LastIndexOf('/')).Remove(0, 1);
   client.DownloadFile(item, "C:\\" + name);
}

Option 2: Use Parallel.ForEach but change DownloadFile code to download only single file:

public static void DownloadFile(string url)
{
    WebClient client = new WebClient();
    var name = url.Substring(url.LastIndexOf('/')).Remove(0, 1);
    client.DownloadFile(url, "C:\\" + name);
}