Need help in deciding when is it good idea to limit the 'number of thread pool threads .net app consumes'?

Question

I have HTTP client which basically invokes multiple web requests against HTTP server. And I execute each HTTP request in a thread pool thread (synchronous call), and by default uses 30 TCP (using httpwebrequest.servicepoint - http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.servicepoint.aspx ). And based on the system i am managing, there can be ~500/1000 thread pool threads waiting for I/O (http response)

Now, I am wondering do I need to limit the number of threads I use as well? (for ex, http://msdn.microsoft.com/en-us/library/ee789351(v=vs.110).aspx System.Threading.Tasks - Limit the number of concurrent Tasks )

EDIT

Yes, I think I need to, limit the number of threads I use as even though these threads are in wait state they take up resources. This way I can control number of resources/threads I use up which makes it easier for my component to be integrated with others without causing them for starvation/contention for resource/threads.

EDIT 2

I have decided to completely embrace async model so that i won't be using thread pool threads to execute http requests, rather I can simply rely on "collaboration of OS Kernel and I/O completion port thread(s)" which will ensure that upon completion response will be sent in a callback (this way i can best use of cpu as well as resource). I am currently thinking of using (webclient.uploaddatataskasync) http://msdn.microsoft.com/en-us/library/system.net.webclient.uploaddatataskasync(v=vs.110).aspx, and update the code accordingly. (couple of references for details: HttpWebRequest and I/O completion ports, How does .NET make use of IO Threads or IO Completion Ports? )

EDIT 3

Basically i have used "async network I/O .net APIs as mentioned above" which essentially removed usage of my parallel library. For details, please see the below answer (i have added it for convenience, just in case if anyone is interested!).

psuedo code to give an idea how I am invoking web requests using webclient

//psudeo code to represents there can be varibale number of requests
//these can be ~500 to ~1000
foreach(var request in requests)
{
    //psudeo code which basically executes webrequest in threadpool thread
    //MY QUESTION: Is it OK to create as many worker threads as number rrequests
    //and simply let them wait on a semaphore, on should i limit the concurrency?
    MyThreadPoolConcurrentLibrary.ExedcuteAction(() =>
        {            
            var sem = new Semaphore(initialCount: 50, maximumCount: 50.Value);
            try
            {
                //using semaphore as the HTTP Server which i am taking to recommend 
                //to send '50' parallel requests in '30' TCP Connections
                sem.WaitOne();
                //using my custom webclient, so that i can configure 'tcp' connections 
                //(servicepoint connection limit) and ssl validation etc.
                using (MyCustomWebClient client = new MyCustomWebClient())
                {
                    //http://msdn.microsoft.com/en-us/library/tdbbwh0a(v=vs.110).aspx
                    //basically the worker thread simply waits here
                    client.UploadData(address: "urladdress", data: bytesdata);
                }
            }
            finally
            {
                sem.Release(1);
            }
        });
}
MyThreadPoolConcurrentLibrary.WaitAll(/*...*/);

Basically should I do something to limit the number of threads I consume, or let the thread pool take care of it (i.e. in case if my app reaches thread pool's maximum thread limit, it any way queues the request - so I can simply rely on it)

*pseudo code which should show my custom webclient where I configure tcp connections, ssl validation etc.

class MyCustomWebClient : WebClient
{
    protected override WebRequest GetWebRequest(Uri address)
    {
        HttpWebRequest request = (HttpWebRequest)base.GetWebRequest(address);            
        request.KeepAlive = true;
        request.Timeout = 300;
        request.ServicePoint.ConnectionLimit = TCPConnectionsLimit;
        request.ServerCertificateValidationCallback = this.ServerCertificateValidationCallback;
        return request;

    }
    private bool ServerCertificateValidationCallback(object sender, System.Security.Cryptography.X509Certificates.X509Certificate certificate, System.Security.Cryptography.X509Certificates.X509Chain chain, System.Net.Security.SslPolicyErrors sslPolicyErrors)
    {
        throw new NotImplementedException();
    }
}

Best Regards.

Answer 1

Since I am performing network I/O (http web requests), it is not good idea use 'synchronous' httpwebrequests and let the thread pool threads to block in sync calls. So, i have used 'async network i/o operations (web client's async task methods) as mentioned above in the question as per the suggestions from comments. It automatically removed usage of number of threads in my component - for details, please see below psudeo code snippet...

Here are some useful links that helped me to adapt to few of c# 5.0 async concepts easily (async/await):

Deep Dive Video (good explanation of async/await state machine) http://channel9.msdn.com/events/TechDays/Techdays-2014-the-Netherlands/Async-programming-deep-dive

http://blog.stephencleary.com/2013/11/there-is-no-thread.html

async/await error handling: http://www.interact-sw.co.uk/iangblog/2010/11/01/csharp5-async-exceptions , http://msdn.microsoft.com/en-us/library/0yd65esw.aspx , How to better understand the code/statements from "Async - Handling multiple Exceptions" article?

Nice book: http://www.amazon.com/Asynchronous-Programming-NET-Richard-Blewett/dp/1430259205

class Program
{        
    static SemaphoreSlim s_sem = new SemaphoreSlim(90, 90);
    static List<Task> s_tasks = new List<Task>();   
    public static void Main()
    {                     
        for (int request = 1; request <= 1000; request++)
        {                             
            var task = FetchData();
            s_tasks.Add(task);                
        }
        Task.WaitAll(s_tasks.ToArray());
    }        
    private static async Task<string> FetchData()
    {
        try
        {
            s_sem.Wait();
            using (var wc = new MyCustomWebClient())
            {
                string content = await wc.DownloadStringTaskAsync(
                    new Uri("http://www.interact-sw.co.uk/oops/")).ConfigureAwait(continueOnCapturedContext: false);
                return content;
            }
        }
        finally
        {
            s_sem.Release(1);
        }
    }
    private class MyCustomWebClient : WebClient
    {
        protected override WebRequest GetWebRequest(Uri address)
        {
            var req = (HttpWebRequest)base.GetWebRequest(address);
            req.ServicePoint.ConnectionLimit = 30;
            return req;
        }
    }
}

Regards.

Answer 2

You could always simply aim for the same limit that browsers run under. That way the server admins can't really hate on you too much.

Now, the RFC says that you should limit connections to 2 pr domain, but according to http://www.stevesouders.com/blog/2008/03/20/roundup-on-parallel-connections/

many browsers go as high as 6 or 8 parallel connections (and this was in 2008).

Browser HTTP/1.1    HTTP/1.0
IE 6,7         2    4
IE 8           6    6
Firefox 2      2    8
Firefox 3      6    6
Safari 3,4     4    4
Chrome 1,2     6    ?
Chrome 3       4    4
Chrome 4+      6    ?
iPhone 2       4    ?
iPhone 3       6    ?
iPhone 4        4   ?
Opera 9.63,     4   4
Opera 10.51+    8   ?