.Net max concurrent timer threads

Question

I am trying to load a queue of interval processes. In other words I have a queue and I want each item in the queue to run on an individual interval.

My problem is that I can't seem to get more than 25 threads to run at one time. I'm using .Net 4.5 on a 64bit machine which has a default max thread count of 32768.

How do I make my app run as many concurrent threads as my machine can handle?

Here is an example app that replicates the actual problem in my production code:

class Program
{
    static void Main(string[] args)
    {
        System.Threading.ThreadPool.SetMaxThreads(200, 200);
        test t = new test();

        t.LoadUrls("http://www.google.com");

        while (1 == 1)
        {
            System.Threading.Thread.Sleep(1000);//refresh every 5 seconds
            Console.WriteLine(System.Diagnostics.Process.GetCurrentProcess().Threads.Count);
        }
    }


    public class test
    {

        public void LoadUrls(string url)
        {
            for (int i = 0; i < 100; i++)
            {
                System.Threading.Timer t = new System.Threading.Timer(new System.Threading.TimerCallback(RunInterval), url, 0, 1000);
                Console.WriteLine("Loaded {0} feeds.", i);
            }
        }
        private static void RunInterval(object state)
        {
            string url = state as string;
            string data = "";

            using (System.Net.WebClient cl = new System.Net.WebClient())
            {
                Console.WriteLine("getting data for " + url);
                data = cl.DownloadString(url);
            }

            //do something with the data

        }
    }
}

This code should theoretically run 198 threads after 2 seconds or so.

By the way, this worked beautifully in my prototype app; it was written in node. But, now I can't get it to work correctly in c#...

ANSWER: The problem was actually with garbage collection and wasn't a threadpool issue at all; the pool is more than capable of spooling all the threads I'm throwing at it. The trick is to use the single parameter constructor of the System.Threading.Timer; this will make the timer use itself as a semaphore, thus avoiding gc.

class Program
{
    static void Main(string[] args)
    {
        for (int i = 0; i < 100; i++)
        {
            test t = new test();
            t.url = "http://www.google.com?" + i;
            System.Threading.Timer ti = new System.Threading.Timer(new System.Threading.TimerCallback(t.RunInterval));
            ti.Change(0, 1000);
        }

        while (1 == 1)
            System.Threading.Thread.Sleep(int.MaxValue);
    }


    public class test
    {
        public string url { get; set; }
        public void RunInterval(object state)
        {
            Console.WriteLine("getting data for " + this.url);
            string data = "";

            using (System.Net.WebClient cl = new System.Net.WebClient())
            {
                data = cl.DownloadString(this.url);
            }
        }
    }
}

I'm not for sure why you would ever want a timer to be collected by the gc, but hey what do I know.

This is for a polling service. It has to poll n feeds at specified intervals. I could make a fancy queue system and run n individual services that only poll 1 feed per service. But, I would like to make this meta; I would like the program to start each individual process. — Eulalie367, Jan 10 '13 at 19:48
Your answer is still wrong. Everything may be garbage-collected, nothing special to exclude timers. You should keep a reference to all the timers and then dispose of them when done. Otherwise, they may still be disposed of when garbage-collected. — TimTIM Wong, Oct 20 '21 at 23:34

H H · Accepted Answer · 2013-01-10T19:16:48.277

5

How do I make my app run as many concurrent threads as my machine can handle?

That's the wrong approach. Every thread is costly, create a few hundred and your system will seriously degrade.

Your code uses the ThreadPool. The pool has an algorithm to limit the number of threads. When you increase the Sleep() times you might see a few more threads.

A more direct way would be to set ThreadPool.MinThreads. But expect less performance, not more.

edited Jan 10 '13 at 19:16

answered Jan 10 '13 at 19:08

H H

263,252
30
330
514

It isn't that costly... I have 64GB of ram to work with, but it never gets over 10MB. I want it to use all 64GB. What would your suggestion be? – Eulalie367 Jan 10 '13 at 19:10
Also, it never uses more than 1% of my available CPU. – Eulalie367 Jan 10 '13 at 19:11
I've added a few suggestions but it is still the wrong approach. – H H Jan 10 '13 at 19:12
1

Curious how you measure memory, every thread should use 1 MB. And the Sleep() and I/O calls don't use CPU so that's why you see 1%. A good indicator that more threads won't help. – H H Jan 10 '13 at 19:13
What would be the correct approach? I basically want to load a queue to retrieve data from n rest services. Each service has it's own refresh limits, so they all have to have individual timers. – Eulalie367 Jan 10 '13 at 19:16
Just remove the `Sleep()` calls and run the I/O in the timer events. I don't see a direct reason to interfere. (When testing, Sleep() is a bad substitute). – H H Jan 10 '13 at 19:19
setting the min threads allows more threads, but threre is still something wrong. System.Threading.ThreadPool.SetMinThreads(200, 200); Does the thead.sleep() in Main put child threads to sleep as well? If so, how do I keep all threads alive indefinitely? – Eulalie367 Jan 10 '13 at 20:08
What is still wrong? And have you tried just using the Timer? Looks like you're trying too hard, and are just making it worse. And No, Sleep() in 1 thread does not affect other threads. – H H Jan 10 '13 at 20:20
If you run the code it only runs each thread once. In other words setting the minthreads worked, but the timers never re-fire. I was thinking that it might be that the timers are all in the main thread that is now sleeping. I changed the sleep(100) in Main to a console.readline, but get the same results. By the way, thanks for all the help (I feel silly having to actually post a question to stackoverflow) – Eulalie367 Jan 10 '13 at 20:31
Your timer instances are becoming Garbage. See http://stackoverflow.com/q/4962172/60761 – H H Jan 10 '13 at 20:36
But do give up on maximizing the threads. Fewer is better here. – H H Jan 10 '13 at 20:38
It looks like gc was the problem the whole time... I was using the state parameter as a passthrough param, when it is really a semaphore. – Eulalie367 Jan 10 '13 at 21:08

score 3 · Answer 2 · answered Jan 10 '13 at 19:15

According to System.Threading.Timer

Use a TimerCallback delegate to specify the method you want the Timer to execute. The timer delegate is specified when the timer is constructed, and cannot be changed. The method does not execute on the thread that created the timer; it executes on a ThreadPool thread supplied by the system.

Then in System.Threading.Threadpool

There is one thread pool per process. Beginning with the .NET Framework 4, the default size of the thread pool for a process depends on several factors, such as the size of the virtual address space. A process can call the GetMaxThreads method to determine the number of threads. The number of threads in the thread pool can be changed by using the SetMaxThreads method. Each thread uses the default stack size and runs at the default priority.

score 1 · Answer 3 · answered Jan 10 '13 at 19:12

1

you want to use property called MaxDegreeOfParallelism, but you will need to modify your code a bit. and then test it and figure out the optimal number of threads you want to run in parallel. here is the link: http://msdn.microsoft.com/en-us/library/system.threading.tasks.paralleloptions.maxdegreeofparallelism.aspx

answered Jan 10 '13 at 19:12

Alex

2,342
1
18
30

MaxDegree imposes a Ceiling. Not a minimum. – H H Jan 10 '13 at 19:14

.Net max concurrent timer threads

3 Answers3