Webclient.DownloadStringAsync multiple url downloading too slow

Question

Im trying to download files at the same time.

-Download string on the main Link

-Take the subLinks from it

-Download subLinks at the same time( 590~ subLinks to download) Like a tree.

    static async Task Direct()
    {
        WebClient wc2 = new WebClient() { UseDefaultCredentials = true, Encoding = Encoding.GetEncoding(1251) };
        HtmlDocument hd = new HtmlDocument();
        wc2.Proxy.Credentials = CredentialCache.DefaultCredentials;
        hd.LoadHtml(wc2.DownloadString(link));
        foreach (var item in hd.DocumentNode.SelectNodes(xpath).Select(x => x.GetAttributeValue("href", "")))
        {
            Task.Run(() =>
                {
                    using (WebClient wc = new WebClient() { UseDefaultCredentials = true, Encoding = Encoding.GetEncoding(1251) })
                    {
                        wc.DownloadStringCompleted += Over;
                        wc.Proxy.Credentials = CredentialCache.DefaultCredentials;
                        wc.DownloadStringAsync(new Uri(item));
                    }
                });
        }
        Console.Title = "ready";
    }
    static Stopwatch sw = new Stopwatch();
    static void Over(object sender, DownloadStringCompletedEventArgs e)
    {
        Task.Run(()=>overTask((sender as WebClient)));
    }
    static async Task overTask(WebClient sender)
    {
        if (sw.IsRunning)
        {
            sw.Stop();
            Console.WriteLine(sw.ElapsedMilliseconds);
            sw.Reset();
        }
        sw.Start();
        sender.Dispose();
    }

1-This code i have, it downloads the subLinks but not the same time, slower than it. It takes DownloadStringAsync request and I hope maximum 20 seconds to all strings, but my big Stopwatch says min. 35 sec.

2-This 1000ms, i expect that max. 20 ms. because they have to started at the same time, They have not much difference between them. Same website,different pages.

I make a few changes in the first code. There is some improvements, but still, slower than i excepted.The Result.

as @Liam says, I tried to make it via thread.

static async Task Direct()
    {
        WebClient wc2 = new WebClient() { UseDefaultCredentials = true, Encoding = Encoding.GetEncoding(1251) };
        HtmlDocument hd = new HtmlDocument();
        wc2.Proxy.Credentials = CredentialCache.DefaultCredentials;
        hd.LoadHtml(wc2.DownloadString(link));
        foreach (var item in hd.DocumentNode.SelectNodes(xpath).Select(x => x.GetAttributeValue("href", "")))
        {
            /* Task.Run(() =>
                {
                    using (WebClient wc = new WebClient() { UseDefaultCredentials = true, Encoding = Encoding.GetEncoding(1251) })
                    {
                        wc.DownloadStringCompleted += Over;
                        wc.Proxy.Credentials = CredentialCache.DefaultCredentials;
                        wc.DownloadStringAsync(new Uri(item));
                    }
                });*/

            new Thread(() =>
            {
                using (WebClient wc = new WebClient() { UseDefaultCredentials = true, Encoding = Encoding.GetEncoding(1251) })
                {
                    wc.DownloadStringCompleted += Over;
                    wc.Proxy.Credentials = CredentialCache.DefaultCredentials;
                    wc.DownloadStringAsync(new Uri(item));
                }
            }).Start();
        }
        Console.Title = "ready";
    }
    static Stopwatch sw = new Stopwatch();
    static void Over(object sender, DownloadStringCompletedEventArgs e)
    {
        new Thread(() =>
        {
            if (sw.IsRunning)
            {
                sw.Stop();
                Console.WriteLine(sw.ElapsedMilliseconds);
                sw.Reset();
            }
            sw.Start();
        }).Start();

the output ms(the ms between two downloads) is here

EDIT:Guys, the foreach loop must started them via little differences. It question hasn't got the answer. I said "If the tasks started at the same time(at least, little differences between them)and they download things that have same length of string(20-30 char is changed), so why is the difference between them is too much? Is it depends on the website? Can we make that faster?

Putting an async call into a `Task` somewhat defeats the object of an async call. Why don't you fire them, collect the `Tasks` returned in an array then `await` on these? — Liam, Apr 21 '17 at 11:29
Possible duplicate of [Running multiple async tasks and waiting for them all to complete](http://stackoverflow.com/questions/25009437/running-multiple-async-tasks-and-waiting-for-them-all-to-complete) — Liam, Apr 21 '17 at 11:30
I'd also have a read of [Using the Stopwatch with Async methods](http://stackoverflow.com/questions/19381426/using-the-stopwatch-with-async-methods) — Liam, Apr 21 '17 at 11:32
The `ServicePointManager` limits the number of concurrent connections to a single computer so you may investigate increasing the max connection limit. — Crowcoder, Apr 21 '17 at 11:44
@Liam I make an taskList and add every task in it. Then, I tried to waitall method, Program freezes on that line. — GBursali, Apr 21 '17 at 12:54
Yes, that's what async is. Your telling it to wait (`WaitAll()`). The clues in the name :) — Liam, Apr 21 '17 at 13:14
If you want fire and forget don't use `async`, use threading for that. Again, see here [Fire-and-forget with async vs “old async delegate”](http://stackoverflow.com/questions/12803012/fire-and-forget-with-async-vs-old-async-delegate). BTW I'd imagine your doing what everyone does, async != threading. async parks threads, threading spawns new threads. They have different use cases — Liam, Apr 21 '17 at 13:16

Webclient.DownloadStringAsync multiple url downloading too slow

0 Answers0