3

In a project of mine, I noticed the server spiking in CPU usage as the number of clients connected increase.

10 clients: 0% mostly, spikes to 7% randomly.
15 clients: 0% mostly, spikes to 10% randomly.
25 clients: 10% mostly, spikes to 60% randomly.
50 clients: 50% mostly, spikes to 60%, CPU is overall at 100% (due to gameservers).
(Note: there are 8 logical cores on the CPU)

I narrowed down the problem to Thread.Yield, on this line: https://github.com/vercas/vProto/blob/master/vProto/Base%20Client/Package%20Sending.cs#L121
As soon as I comment that line out, CPU usage stays at 0% constantly even with 100 clients!

Why is Thread.Yield doing this?

user2864740
  • 60,010
  • 15
  • 145
  • 220
Vercas
  • 8,931
  • 15
  • 66
  • 106
  • Do you *also* comment out the `Thread.Sleep` line when commenting out `Thread.Yield`? That is, I don't believe the issue is strictly related to `Thread.Yield` "causing the spikes", but rather that `Thread.Sleep(10)` actually *reduces* the frequency of the loop and/or *contention*, and thus CPU usage consumed per thread.. – user2864740 Mar 08 '14 at 09:43
  • No actual sending was happening during my test. Swapping a few pointers couldn't have been that expensive... – Vercas Mar 08 '14 at 09:50
  • I don't doubt it's related to the scheduling in *some* fashion, but I suspect it's *not* simply because of a "context switch". I really do think it might be related to lock contention or other degenerate case with the scheduler. Does `Thread.Sleep(1)` also exhibit similar spikes? If not, what about throughput? – user2864740 Mar 08 '14 at 09:55
  • I could easily test by placing the sleep in an `else` statement attached to the previous if (checking if there are any queued packages). As for throughput, there is only a body-less package sent every 30 seconds per client (and per-thread). The same throughput exists now and doesn't exhibit the same problem. – Vercas Mar 08 '14 at 10:08
  • 700 clients and still 0% CPU usage. – Vercas Mar 08 '14 at 10:33
  • That's with removing the lock or using Thread.Sleep(1) only or a combination or ..? – user2864740 Mar 08 '14 at 10:52
  • `Thread.Sleep(1)`. 1200 clients and only goes to 1% sometimes. I'm benchmarking with the help of a friend. So far, results are very satisfying. Pings are ~70 for me. That's less than I get to our gameservers (on the same machine). – Vercas Mar 08 '14 at 10:58

2 Answers2

2

I don't know why this usage of Thread.Yield/Sleep1 might cause these spikes, however I refute that it is caused merely by "context switching". (I've no doubt it relates, but a stronger explanation is required.)

Thread.Sleep or Thread.Yield seems to give a satisfactory answer for when Yield and Sleep are used exclusively - basically that Yield, like Sleep(0), might not yield - although it may not directly apply the case of "Yield and Sleep if required"1 vs "always Sleep without trying to yield Yield", as presented in this question.

1The original CPU-spiking code presented used: if (!Thread.Yield()) Thread.Sleep(10);. (This is an example of why it is important to include relevant code in questions.)

My arguments against context switching being the problem follow.

  1. Windows use preemptive scheduling and context switches dozens of times per second, even when threads don't actively yield.

  2. Thread.Sleep(x), where x > 0, will always cause a context-switch; yet Thread.Sleep(1) is reported to not cause such spikes.

  3. Thread.Yield might not cause a context switch, yet it is reported to cause spikes.

    The operating system (read: Thread.Yield) will not switch execution if..

Community
  • 1
  • 1
user2864740
  • 60,010
  • 15
  • 145
  • 220
  • Well, as I mentioned before, the spikes were happening while sending nothing more than two packages every 30 seconds, and constant CPU usage increased exponentially after ~300 clients. All the updates of that file after removing `Thread.Yield()` never produced a spike, even under load. Especially the new `ThreadPool`-based method, it is amazing. I saturated the network connection before getting to 2% CPU usage. (~1800 vProto clients to 1 vProto server + ~250 clients connected to gameservers on the same server machine) Still, I'd love to get to the bottom of this problem. – Vercas Mar 31 '14 at 17:43
1

It is due to the way Thread.Yield releases processing. It forces the current process thread to release prematurely. This in turn sends out messages to all other processes telling them to do their own thing. Switching process context is expensive in terms of swapping out memory, loading cached processes, and moving through the process list out of sequence.

From MSDN:

If this method succeeds, the rest of the thread's current time slice is yielded. The operating system schedules the calling thread for another time slice, according to its priority and the status of other threads that are available to run.

Yielding is limited to the processor that is executing the calling thread. The operating system will not switch execution to another processor, even if that processor is idle or is running a thread of lower priority. If there are no other threads that are ready to execute on the current processor, the operating system does not yield execution, and this method returns false.

This method is equivalent to using platform invoke to call the native Win32 SwitchToThread function. You should call the Yield method instead of using platform invoke, because platform invoke bypasses any custom threading behavior the host has requested.

UPDATE

There has been some challenge to the statement that Thread.Yield causes expensive context switching. Here are additional references:

Difference between Thread.Sleep0 and Thread.Yield

Threading in C# - Joseph Albahari

MSDN - About Processes and Threads

MSDN - Multitasking Considerations

The recommended guideline is to use as few threads as possible, thereby minimizing the use of system resources. This improves performance. Multitasking has resource requirements and potential conflicts to be considered when designing your application. The resource requirements are as follows:

  • The system consumes memory for the context information required by both processes and threads. Therefore, the number of processes and threads that can be created is limited by available memory.
  • Keeping track of a large number of threads consumes significant processor time. If there are too many threads, most of them will not be able to make significant progress. If most of the current threads are in one process, threads in other processes are scheduled less frequently.
Community
  • 1
  • 1
Adam Zuckerman
  • 1,633
  • 1
  • 14
  • 20
  • And when the Windows scheduler does this, it's cheap as dirt (performance-wise), but when I do it, it eats away half a CPU core worth of performance..? – Vercas Mar 08 '14 at 09:27
  • Pretty much. Most of the expense is the forced swapping in and out of memory. – Adam Zuckerman Mar 08 '14 at 09:31
  • Don't do a Thread.Sleep(0) as it will do the exact same thing as Thread.Yield(). – Adam Zuckerman Mar 08 '14 at 09:37
  • I am aware. I use 1 millisecond. If the system it runs on happens to have higher thread resolution enabled, it'll be a gain. The otherwise extra 14 milliseconds won't hurt. I hope. – Vercas Mar 08 '14 at 09:39
  • I'm not sure how the MSDN link/quote addresses the spikes. Since windows is a *preemptive* OS, it will "expensive switching process context" anyway, even without Thread.Yield, will it not? (In fact, it does this many, many times/second already on my very-interactive desktop.) – user2864740 Mar 08 '14 at 09:41
  • I wish I could peak into Thread.YieldInternal in mscorlib.dll! – Vercas Mar 08 '14 at 09:43
  • Also, since `Thread.Sleep(10)` causes an "expensive switching process context" (albeit one that says "don't call me again before 10ms"), *why* would only `Thread.Yield` be affected with these spikes when not `Thread.Sleep(10)`? That is, I am challenging the assertion that the spikes are caused by "switching process context", as it happens in both cases. – user2864740 Mar 08 '14 at 09:51
  • When .NET invokes the Win 32 thread switch, I am fairly certain that doing so also pauses the .NET runtime. If there was a method for just pausing/yielding to other .NET threads, it would cut down a lot of swapping. – Adam Zuckerman Mar 08 '14 at 09:51
  • Thread sleep has at least two paths of processing. If you perform a sleep(0), it does exactly the same thing as yield. If you put a number > 0, it runs a different code path that doesn't always perform process switching. – Adam Zuckerman Mar 08 '14 at 09:55
  • How could it sleep without performing a context switch? Have an external reference/resource? – user2864740 Mar 08 '14 at 09:58
  • Oh, I see. That's what I *don't like* about this answer - it talks about *processes*, which I refute. Also, only `Thread.Yield` *may not* yield thread control, while `Thread.Sleep(x; x>0)` *will always* yield the process (which again is an argument *against* the claim in this reply). The MSDN link talks about *processors* (not *processes*), which may be a factor. – user2864740 Mar 08 '14 at 10:06
  • 2
    You seem to be conflating two very different things - context switching and memory swapping. Context switching does not necessarily require memory swapping, so the reasoning behind your answer would seem to break down. – Iridium Mar 08 '14 at 13:57