3

I have to send 100,000 http requests every few seconds and with the code provided below it takes 19 seconds to send the requests and its longer than my interval.

During this time the CPU usage and Ethernet usage is 100% . I have tried it in a dual processor computer with a higher band width but the same result.

Is there any other way to have a better performance.

protected async void btnStartCallWhenAll_Click(object sender, EventArgs e)
{
    t1 = DateTime.Now;
    // Make a list of web addresses. 
    List<string> urlList = SetUpURLList(Convert.ToInt32(txtNoRecordsToAdd.Text));

    // One-step async call. 
    await ProcessAllURLSAsync(urlList);

    t2 = DateTime.Now;
    double spent = t2.Subtract(t1).TotalMilliseconds;
    txtTimeElapsed.Text = " Time Spent :" + spent.ToString() ;
}
private List<string> SetUpURLList(int No)
{
    List<string> urls = new List<string>
    {
    };

    for (int i = 1; i <= No; i++)
        urls.Add("http://msdn.microsoft.com/library/windows/apps/br211380.aspx");

    return urls;
}
private async Task ProcessAllURLSAsync(List<string> urlList)
{
    ServicePointManager.UseNagleAlgorithm = true;
    ServicePointManager.Expect100Continue = true;
    ServicePointManager.CheckCertificateRevocationList = true;
    ServicePointManager.MaxServicePointIdleTime = 10000;
    ServicePointManager.DefaultConnectionLimit = 1000;

    IEnumerable<Task> CallingTasksQuery =
        from url in urlList select CallURLAsync(url);

    Task[] CallingTasks = CallingTasksQuery.ToArray();

    await Task.WhenAll(CallingTasks);
}

private async Task CallURLAsync(string url)
{
    var content = new MemoryStream();
    var webReq = (HttpWebRequest)WebRequest.Create(url);
    using (WebResponse response = await webReq.GetResponseAsync())
    {
    }
}
albert sh
  • 1,095
  • 2
  • 14
  • 31
  • Have you tried using `Parallel.For` or `Parallel.Foreach`? – Jamie Rees Sep 01 '15 at 15:13
  • 7
    100'000 http requests is a lot for a single tcp stack, are you sure the network card is not the issue here? – Philippe Paré Sep 01 '15 at 15:16
  • @PhilippeParé, Thanks for your note, would you please let me know how can I increase the TCP Stack or what kind of network card is appropriate. – albert sh Sep 01 '15 at 15:32
  • 2
    What OS are you running this on? Have you considered partitioning this between more than one physical machine? You should consider that the network or web server may be the issue as well, the internet is only so fast, and a 30-50 millisecond response time (quick) multiplied by 100,000 requests means it takes some time to complete. – Ron Beyer Sep 01 '15 at 15:33
  • 1
    @RonBeyer has your answer, what you need is multiple machines, not a single dual core machine. If I'm not mistaken, the tcp stack is cpu and network card bound. The only way you're going to achieve 100k requests is by fragmenting your requests on different machines. At some point, your network will also be affected, the upload speed of your connection will become a bottleneck too. – Philippe Paré Sep 01 '15 at 15:36
  • 4
    100,000 * 1246 KB (average size web page) web requests is 124.6 GB, that's some serious bandwidth to download that in 19 seconds, not to mention that the question says this is too slow. I have to assume that these are a few byte web requests instead of https web page requests. Also the code listed uses DateTime.Now and builds a loop of identical urls included in the timing. This can not be representative of distinct web calls using accurate timing. – James Sep 01 '15 at 15:43
  • A question may be "why the heck are you sending so many requests in such a short space of time?" there may be alternative solutions to your problem. – Clint Sep 01 '15 at 15:57
  • You may want to look at this: http://stackoverflow.com/questions/4145365/how-to-handle-up-to-100k-http-connections-in-net/4145474#4145474 and this: http://stackoverflow.com/questions/648282/any-nio-frameworks-for-net – Paddy Sep 01 '15 at 16:01
  • @PhilippeParé 100k connections are *nothing* for a Windows server. Also, the NIC does not even know what a connection is. The kernel does that. – usr Sep 01 '15 at 16:19
  • @usr I'm not talking about the server, I'm talking about the client making the requests. – Philippe Paré Sep 01 '15 at 17:21
  • @PhilippeParé no trouble there either. A TCP connection is just a data structure. Early kernels had non-linear scaling data structures. That's gone for a long time. – usr Sep 01 '15 at 17:24
  • @PhilippeParé, in order to change TCP stack, I changed TcpTimedWaitDelay =1 and MaxUserPort=32768 in registry but it had no effect, do you have any other recommendation? – albert sh Sep 01 '15 at 17:25
  • 1
    "During this time the CPU usage and Ethernet usage is 100%", clearly this is too much for the system – Philippe Paré Sep 01 '15 at 17:25
  • @albertsh you need to put this code on multiple machines, this one is not powerful enough for this – Philippe Paré Sep 01 '15 at 17:26
  • @albertsh I would code a local server-client connection to receive orders and execute these requests... Let's say you have the "dispatcher" and 10 "workers". The dispatcher receives the urls, splits them into 10 and gives the requests to the 10 workers. By the way, do you need the response page? where is it stored? – Philippe Paré Sep 01 '15 at 17:32
  • @RonBeyer, I am testing in windows server 2012. Unfortunately partitioning is not a good solution for me as the number of requests per second is only 3000 and I would need 30 computers to cover that. – albert sh Sep 01 '15 at 17:37
  • @PhilippeParé, Thank you for taking time.No I don't need the response. these are just the Confirmation URLs that must be called for a third party.to decrease amount of data transfer I also used : webReq.Method = "HEAD"; – albert sh Sep 01 '15 at 17:45
  • @Clint, These are the confirmation URLs that must be sent for third party and I don't need their response . – albert sh Sep 01 '15 at 17:54
  • @albertsh sure this is a good improvement (using webReq.Method = "HEAD"). If you don't need the response, you could very easily create the client-server code to be only one way, the clients always read incoming requests and executes them as soon as possible. – Philippe Paré Sep 01 '15 at 18:07
  • @PhilippeParé, unfortunately using webReq.Method="Head" had no effect. – albert sh Sep 01 '15 at 18:18
  • @JamieRees, Yes I also tried parallel.for but it was slower than this solution – albert sh Sep 01 '15 at 18:19

1 Answers1

0

Referring to HttpWebRequest is extremely slow!

Ensure you've set your:

webReq.Proxy = null;

It appears as though the startup of WebRequest will look for the default proxy.

Community
  • 1
  • 1
Alex KeySmith
  • 16,657
  • 11
  • 74
  • 152