C# - When to use standard threads, ThreadPool, and TPL in a high-activity server

Question

I've been reading a lot about threading lately as I am looking to develop a high-performance, scalable TCP server capable of handling up to 10,000-20,000 clients, each client of which is consistently communicating bidirectionally to the server with a command-based system. The server will receive a command, and execute either a single (or many) tasks as per the command. My question is how to appropriately make use of the .NET threading constructs for a variety of situations, executing tasks that could take between one minute to several hours, depending on the work being performed.

What's confusing me the most is the fact that everywhere I read, I see something like "use a manually created Thread (or custom thread pool) to handle 'long-running' tasks, and use TPL for short-lived tasks, or tasks that require parallel processing." What exactly is a long-running task? Is that 5 seconds, 60 seconds, an hour?

With what time frame should I be using each of these three methods of creating threads:

• Manually created Threads
• The .NET ThreadPool class
• TPL

Another issue I've contemplated is as follows--say my server does in fact have 20,000 clients connected, each of which sends 1 command (which could translate to one or many tasks) per second. Even with powerful hardware, isn't there a chance that I could be pushing too high of a workload into whatever thread pool / work item queue I have, thereby eventually generating an OutOfMemoryException after the queue slowly fills to the maximum?

Any insight would be greatly appreciated.

Answer 1

Actually, for that scenario all of those are secondary; the first thing you should look at is asyc-IO, aka .BeginRead(...) etc; this allows you to minimise the number of threads by waiting on IO completion ports - much more efficient.

Once you have a complete message, at that scale I would throw the message into a custom thread-pool/synchronized-queue. I would have a controlled number of regular threads (not pool threads or IOCP) servicing that queue to process each item.

As it happens I'm doing something similar (lower scale) at the moment; to prevent memory exploding, I have capped the work queue; if it gets full (i.e. the workers can't keep up) then you might block IOCP for a small while, perhaps with a timeout eventually that tells the client "too busy" at the IOCP layer.

Answer 2

What's confusing me the most is the fact that everywhere I read, I see something like "use a manually created Thread (or custom thread pool) to handle 'long-running' tasks, and use TPL for short-lived tasks, or tasks that require parallel processing."

Strange advice, or maybe you mis-quoted a little. A thread is also capable of parallel processing, and with the TPL you can create a Task with the LongRunning option. What remains is that you should not launch long tasks on the ThreadPool.

What exactly is a long-running task? Is that 5 seconds, 60 seconds, an hour?

The TPL runs on top of the ThreadPool and the TP will create new Threads at a max of 2 per second. So long-running is >= 500 ms

---

Even with powerful hardware, isn't there a chance that I could be pushing too high of a workload into whatever thread pool / work item queue I have,

Yes, no Threading tool can extend your actual capacity...

With 20k clients you will probably need a Server Farm, an option to include in your design early...

So you should probably give WCF a good look before getting to deep into sockets.

Answer 3

Marcs suggestion is the way I would do it. But if you tasks take longer than one second and the clients sends a request per second the queue would steadily increase.

In that case I would use one server as facade which gets all requests from the clients and send responses back to them in an asynchronous manner.

The server would put all requests in a message queue which is read by several other servers. Those servers handle the requests and put the response in another message queue which is read by the first server.

Another solution would be to use a load balancing server.

Answer 4

You appear to be building a server which will service thousands of concurrent requests, each long-running in terms of minutes to hours.

Typically, make thread workloads short enough to complete at most within a few seconds. Anything longer, you'll start hogging server resources and seriously affect your server's scalability. Having tens of thousands of threads block on long-running operations, or doing those long-running operations simultaneously, will definitely kill your scalability.

Not sure how much CPU time you're consuming in each long-running. This will affect your design, e.g.:

If each long-running is primarily blocking on I/O, you may use one thread to wait on overlapped I/O or I/O completion port, then wake up new threads to process completed I/O (up to a throttled limit). You'll need to have a limit number of threads to service the waiting connections.

If each long-running operation waits for other operations to complete, consider Windows Workflow Foundation.

If each long-running operation consumes CPU, you don't want too many of them to be running at any one time or it will thrash your server. In this case, use MSMQ and/or TPL to queue tasks and make sure only a few are running at the same time.

In all of these, it seems that you are keeping the client connection open. The worst thing to do is to keep one thread blocking for each connection. You'll need to implement thread-pooling strategies in order to use only a limited number of threads to service all outstanding connections.