Parallel async HttpWebRequests with different proxies. How to maximize performance?

Question

I'm writing an app that needs to do a lot of parallel httpwebrequests to bookmaker site from different proxies. The reason why i'm using different proxies is that bookmaker can ban ip if there are many requests from it. My goal is to get freshest site content as fast as possible. Here is my code with all settings that I have:

        ServicePointManager.DefaultConnectionLimit = 1000;
        ServicePointManager.Expect100Continue = false;
        ServicePointManager.UseNagleAlgorithm = false;

        for (var i = 0; i < proxyCollection.Count; i++)
        {
            var proxyLocal = proxyCollection[i];
            var iLocal = i;
            Task.Run(async () =>
                {
                    var httpWebRequest = (HttpWebRequest) WebRequest.Create(String.Format("https://bookmaker.com/page{0}", iLocal));
                    httpWebRequest.Proxy = proxyLocal;
                    httpWebRequest.PreAuthenticate = true;
                    httpWebRequest.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;

                    using (var httpWebResponse = (HttpWebResponse) await httpWebRequest.GetResponseAsync())
                    using (var responseStream = httpWebResponse.GetResponseStream())
                    using (var streamReader = new StreamReader(responseStream))
                    {
                        var stringContent = await streamReader.ReadToEndAsync();
                        //Here i'm processing new data. It works pretty fast, so this isn't a problem
                        ProcessStringContent(stringContent);
                    }
                });            
         }

And this code works not so fast than I expected.

First problem is that in strange reason all requests don't start at one time. As I can see in task manager loading has two or more maximums. More over I have one realy fast proxy and proxyCollection contains it. But if I use await Task.Delay(5000) after code above, in some cases to the moment when 5000ms have passed the request with my fast proxy doesn't even start!

Second problem is that total time of all task execution is slow. I expected that if one request needs 200-300ms to execute, than 100 requests in parallel and async needs a little more time. But some times this "more" is 10-20 times. I have a suspicion, that something is wrong.

Third problem is that when I'm running this code it freezes UI (not full freeze, but UI lags). I read that WebRequest.Create is processing synchronously and can get some time (dns lookup, proxy settings e.t.c) and if I'm making a lot of requests they can simply fill all my threads (UI thread too) for creating WebRequests. But I tried to create requests to direct ip address (WebRequest.Create(String.Format("https://10.10.10.1/page{0}", iLocal)) - nothing changed, and i'm setting proxy (so auto detect proxy isn't needed), so I don't understand why can creating take so much time (and is problem with creating or maybe with smth else?).

Please can someone point me what i'm doing wrong? I'm lost in all ServicePointManager settings (all what I tried didn't help). Can .Net deal with such type of task? Or maybe I need to use nodejs for example for best performance?

P.S: Collection of proxies is not such big (50-100 different proxies).

score 0 · Answer 1 · edited May 23 '17 at 11:43

0

Back to (Parallel) Basics: Don't Block Your Threads, Make Async I/O Work For You

Example:

Parallel.For(0, 10, delegate(int i) {
    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(
        new Uri("http://www.mysi.com"));

    string dataToSend = "Data";
    byte[] buffer = System.Text.Encoding.GetEncoding(1252).
        GetBytes(dataToSend);
    request.Method = "POST";
    request.ContentType = "application/x-www-form-urlencoded";
    request.ContentLength = buffer.Length;

    request.Host = "www.mysite.com";

    Stream requestStream = request.GetRequestStream();
    requestStream.Write(buffer, 0, buffer.Length);
    requestStream.Close();

    HttpWebResponse response = (HttpWebResponse)request.GetResponse();
});

Send multiple WebRequest in Parallel.For

edited May 23 '17 at 11:43

Community

1
1

answered Apr 01 '15 at 08:45

Taras Kovalenko

2,323
3
24
49

Can you please explain what did you mean with this post? Thank you. – Юрий Бржозовский Apr 01 '15 at 08:50
If you use async/await we need use Parallel but not Task.Run – Taras Kovalenko Apr 01 '15 at 08:52
Hm. It there any difference between Parallel and Task.Run? I'm not sure, but parallel works with tasks too. How can it helps in my case? Thank you. – Юрий Бржозовский Apr 01 '15 at 09:55
Interesting. Can you point me please where I can read about that Task.Run and async/await does not work well? It's really surprising for me. Thank you. – Юрий Бржозовский Apr 01 '15 at 11:12
You can also try using Task.WhenAll instead Task.Run https://msdn.microsoft.com/en-us/library/system.threading.tasks.task.whenall%28v=vs.110%29.aspx – Taras Kovalenko Apr 01 '15 at 11:20
In the end I of cause use `Task.WhenAll`. But I need to start tasks anyway - that's why I use `Task.Run` – Юрий Бржозовский Apr 01 '15 at 11:24

Parallel async HttpWebRequests with different proxies. How to maximize performance?

1 Answers1