4

Simplified background:

My app runs a lot of tasks. Most of them are CPU intensive.
One task (which is actually a single thread running in a loop, listening to packets from the network), is very much a "realtime" task. To make it more interesting, that thread is native code called using pinvoke.

The problem:

When a lot of traffic is showing, the tasks are working very hard, and all the cores are maxing out. When that happens, the "realtime" thread (which runs on a 100% cpu core), starts to drop packets, because it doesn't get enough cpu time.

The question:

It is possible to somehow "reserve" one core for the "realtime" thread, and limit all the other threads (tasks) to other cores?

  • Of course, there are other processes running, consuming CPU time as well, but let's assume they consume little and constant resources.
  • This is a real problem that can be solved by "throw more cpu on it"... Not an option...

EDIT - Answering many useful comments:

  • We use WinPcap to capture all packets, which can be many (many).
  • The "realtime" thread is not really "realtime" (I think "realtime" is for processes - In .net ThreadPriority uses "Normal", "AboveNormal", etc..)
  • The "realtime" thread calls WinPcap serially, packet after packet. We suspect that since it is starved enough, it doesn't keep up, and WinPcap's buffer is overflowed.
seldary
  • 6,186
  • 4
  • 40
  • 55
  • 2
    Why not simple just raise the priority of the "task"? – Some programmer dude Apr 23 '14 at 07:17
  • 1
    Yeah - I don't understand. Listening for network packets is not typically CPU-intensive. Have you got a 100GB network? – Martin James Apr 23 '14 at 07:57
  • Are you using UDP or similar protocol where packets would be lost? If so, raising the priority of the network thread sounds like a better option, as suggested by @JoachimPileborg. – Martin James Apr 23 '14 at 08:00
  • 1
    @JoachimPileborg: If it **really** is a realtime thread (in Windows), then you can't raise the priority any higher. But then non-realtime threads won't steal the core anyway. So I suspect the bug is simply that the thread priority is in fact not realtime. – MSalters Apr 23 '14 at 09:21

2 Answers2

2

You can specify an affinity mask for processes and threads. This allows you to prevent the scheduler using specific processors. So you could set the thread affinity mask for one thread to a single processor, and set the thread affinity mask for all other threads to be all processors other than the reserved processor. Since .net insulates you from directly creating system threads it is actually not that easy to impose the necessary masks on all system threads.

Another way to persuade the scheduler to give your critical thread special treatment would be to boost its priority. That is something that you should only do with care and significant forethought. However, it's a much simpler way to make sure that your thread never has to wait for other threads in your process.

Perhaps it would be better for your app to use fewer threads than processors, but create a dedicated thread for the critical work. That way you could avoid any affinity mask or priority skullduggery and let the system's thread scheduler do the work for you

David Heffernan
  • 601,492
  • 42
  • 1,072
  • 1,490
  • 1
    I believe you've misinterpreted the exact nature of the problem, though your solution is correct. It's not that the packet-listener thread is running at 100%, it's that the packet-listener thread shares a core with other task threads and is being starved of CPU. So putting it on a core by itself and having all other threads run on all other cores would help reduce packet loss, as you say. – Eight-Bit Guru Apr 23 '14 at 07:27
  • @Eight-BitGuru - Exactly. – seldary Apr 23 '14 at 07:35
  • 1
    @Eight the answer describes doing just that – David Heffernan Apr 23 '14 at 07:43
  • @seldary presumably, would be necessary to tie the network interrupt handling to the dedicated core too. I'm not convinced that such affinity-twiddling would affect overall throughput in a positive way, since it reduces the number of cores available to the rest of the app by one. If you try this, and it actually does improve overall performance, I would be interested to hear about it. – Martin James Apr 23 '14 at 08:09
  • Threads that become ready on a signal, such as network threads when data becomes available, are often given a dynamic priority boost anyway and should be made running 'immediately'. I'm not convinced that you have identified the root cause of your packet-loss problem:( – Martin James Apr 23 '14 at 08:19
  • @DavidHeffernan yes I know, I acknowledged that; prior to your edit of your answer, however, your interpretation of the problem was askew. – Eight-Bit Guru Apr 23 '14 at 08:21
  • @Eight-BitGuru Actually I'm with Martin here. I think it quite plausible that the asker has misdiagnosed the problem. I think it quite plausible that lack of CPU resource for the critical thread is not actually the root of the problem. – David Heffernan Apr 23 '14 at 08:24
  • @MartinJames - We actually use WinPcap, so I'm not sure what you say applies here (?) – seldary Apr 23 '14 at 09:11
  • @DavidHeffernan Oh, I'm entirely willing to concede that the actual problem may differ from that which the symptoms indicate - all I was saying is that your original pre-edit answer contained a misinterpretation of the problem as described. – Eight-Bit Guru Apr 23 '14 at 10:08
1

Yes. That concept is called "processor affinity", and exists at both the process level and the thread level. It is a little awkward to invoke, but nothing scary: How Can I Set Processor Affinity in .NET?

Note that you would have to set the affinity for all the threads that you see. However, you are not the only creator of threads - things like the ThreadPool and GC also have some threads. I do not suggest it is a great idea to mess with those, but the ProcessThread class won't actively stop you from doing this.

Community
  • 1
  • 1
Marc Gravell
  • 1,026,079
  • 266
  • 2,566
  • 2,900
  • How would you go about making sure that all native threads in the managed process have the necessary mask? Is there notification that you can subscribe to whenever a system thread is created? – David Heffernan Apr 23 '14 at 08:03
  • @DavidHeffernan no, no such notification. Either the code has to do it when they are created, or you could poll periodically – Marc Gravell Apr 23 '14 at 08:10