Threading HTTPWebRequests

Question

I have a program which is permanently sending http requests to a server and analyzing the responses. As I need to send a lot of requests I would like to do parallel http requests. I tried a few things like Parallel.ForEach, creating Tasks with the longrunningoption, etc. but I always had the same problem, it was too slow. I'm using the httpwebrequest class and for testing I compared the times which one thread needed for doing 50 requests, and then the same with more threads. 1 Thread took 1500ms for these 50 requests, and 5 threads already took 1900-2500ms. It slowed down with each new thread/task..

    public void test()
    {
        Stopwatch stopWatch = new Stopwatch();

        stopWatch.Start();

        for (int i = 0; i < 50; i++)
        {
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create("www.thetestingurl.com");

        request.Method = "GET";

        var response = request.GetResponse();

        response.Close();            
        }

        stopWatch.Stop();

        MessageBox.Show(stopWatch.ElapsedMilliseconds.ToString());
    }

So what am I doing wrong? I already know about ServicePointManager.DefaultConnectionLimit, but this doesnt help.

EDIT:

This is how I try to create the tasks:

for (int i = 0; i < taskCount; i++)
{
    var task = new Task(() => test(),
        TaskCreationOptions.LongRunning);
    task.Start();
}
Task.WaitAll();

EDIT:

So, to make it clear what I mean: When I run this code on 1 Thread it takes 1500ms, when I use tasks, threads, threadpool etc. each thread needs more than 1500ms. I want to create x Threads, they should send the requests parallel and each thread should take 1500ms for the 50 requests, just as the 1 threaded application does. Is this not possible?

Should be a lot faster with multiple threads as you're I/O bound. Please show your parallelized code. Maybe your locking pattern is not correct... — Pragmateek, Nov 03 '13 at 14:44
As said above, I tried a few things like Parallel.ForEach, Threads or creating Tasks. E.g. I just do this in a for loop: var task = new Task(() => test(), TaskCreationOptions.LongRunning); task.Start(); — John Smith, Nov 03 '13 at 15:33
But how do you read the result of each task? Maybe you're synchronizing too early... — Pragmateek, Nov 03 '13 at 15:35
Are you set System.Net.ServicePointManager.DefaultConnectionLimit = 100000? Try to get async response. — sh1ng, Nov 03 '13 at 18:58
http://stackoverflow.com/questions/202481/how-to-use-httpwebrequest-net-asynchronously — allonhadaya, Nov 03 '13 at 19:00

score 2 · Answer 1 · answered Nov 03 '13 at 18:11

I suggest using ThreadPool instead of Task or Parallel.ForEach, because they also uses ThreadPool and you have no control on it. I prepared simple example based on your code, you can compile it as console app:

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Net;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using System.Windows.Forms;

namespace Test1
{
    public class Class1
    {
        public void test()
        {
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.dir.bg/");

            request.Method = "GET";

            var response = request.GetResponse();

            response.Close();
        }

        public static void Main()
        {
            CountdownEvent countDown = new CountdownEvent(50);

            Stopwatch sw = new Stopwatch();
            sw.Start();


            for (int i = 0; i < 50; i++)
            {
                //11646 ms
                //new Class1().test();

                // 502 ms
                ThreadPool.QueueUserWorkItem(DoRequest, countDown);
            }

            countDown.Wait();

            sw.Stop();

            MessageBox.Show(sw.ElapsedMilliseconds.ToString());


        }

        private static void DoRequest(Object stateInfo)
        {
            CountdownEvent countDown = (CountdownEvent)stateInfo;

            new Class1().test();

            countDown.Signal();
        }
    }


}

In my tests time was decreased from 11646 ms to 502 ms.

Thank you, but isn't it possible to get the same speed for each thread as the speed of the synchronous http request? I mean you need 11646ms for the 50 requests, so ~232,92ms for each request. Although your suggestion decreases the overall time, the time for each request is approx. 502ms. So is it possible to create 50 Tasks/Threads which need the same time for the request as the non multitheaded code example (so approx 232ms)? — John Smith, Nov 03 '13 at 18:33

Threading HTTPWebRequests

1 Answers1