Is Blocking code really expensive on modern systems?

Question

I'm trying to grasp a bit better the concepts of async programming (mostly for C#) and blocking/non blocking code.

In C#, if I call .Wait() on a Task , is it always considered "blocking" ? I understand that the current thread will be blocked. However the thread is put in a "waiting" state (AFAIK), and AFAIK it will never be scheduled by the OS until woken up when the Task completed (I assume the thread is woken up by kernel magic)

In that case, the CPU time taken by this blocking operation should be negligible during the waiting period. Is it indeed the case?

So where are the advantage of async programming coming from? Is it because it allows to go beyond 1000 or so threads that the OS wouldn't allow ? Is it because the memory overhead per async task is lower than the overhead of a thread? Keep in mind that the "event loop" that manages all the tasks in async context also has work to do to manage the scheduling of all async tasks, bookeeping etc. Is it really less work than what the kernerl has to do in the blocking case to manage threads?

Yes, `Wait()` is always blocking. Yes blocking is expensive. When a thread is blocked a CPU core is doing nothing but wait. Thread switching is *expensive* because a thread's entire stack has to be pulled back into memory and almost certainly the CPU cache. That's why blocks initially start with spinwaits, which peg a core at 100%. There are no event loops involved. Windows is natively asynchronous, with blocking operations *emulated* by the kernel. Even lower, IO is not performed by the CPU itself but devices with their own caches that notify the CPU when done — Panagiotis Kanavos, Aug 26 '22 at 21:39
With async operations, instead of spinwaiting or waiting for a thread to be switched back into memory, a CPU core could be performing useful work. Instead of waiting for a network or disk device to return data, the core could be processing some *other* request. In server and cloud environments such waste is counted in servers and electricity bills, not seconds per request — Panagiotis Kanavos, Aug 26 '22 at 21:42
"When a thread is blocked a CPU core is doing nothing but wait." What do you mean ? When a thread is blocked or active the OS can always swap it, right? Otherwise we would have no multi-threaded programs on single core machine — lezebulon, Aug 26 '22 at 21:43
@PanagiotisKanavos comments are intended for asking for more information, or for suggesting improvements for the question, [not for answering the question](https://prnt.sc/RfK7kkKxohGr "Use comments to ask for more information or suggest improvements. Avoid answering questions in comments."). Instead of answering the question in multiple successive comments, you should answer it by posting an answer. — Theodor Zoulias, Aug 26 '22 at 21:45
@TheodorZoulias I know, just like the other people you say the same thing. People with 5 or 6 digit rep know what is or what isn't a comment — Panagiotis Kanavos, Aug 26 '22 at 21:50
@PanagiotisKanavos having 5-6 digits rep is not an excuse for using this site incorrectly. If your comments above are not attempts to answer the question, then what are they? Are they side-notes that intend to inform the OP for something irrelevant to the question asked? — Theodor Zoulias, Aug 26 '22 at 21:52
@PanagiotisKanavos "When a thread is blocked a CPU core is doing nothing but wait." no it doesn't, the thread gets suspended which causes a context switch (which as you say is expensive). "spinwaits, which peg a core at 100%" not on modern CPUs, they just peg the CPU to *spin*, which is a low-power operation taking the *same time* as another instruction, see https://www.tabsoverspaces.com/233735-how-is-thread-spinwait-actually-implemented — Charlieface, Aug 27 '22 at 23:05
@Charlieface how is that lower power than any other operation and how does that not waste a core? Can the core do anything other while spinwaiting? Do you *not* have to add more servers to handle those lost due to spinwaiting? Have you seen what spinwaiting does to a web farm? — Panagiotis Kanavos, Aug 29 '22 at 06:32
@Charlieface I was unfortunate enough to have a tech lead that assumed blocking is cheap in a high-traffic 100 VM AWS web farm. His insistence and code caused requests to peg cores at 100%, at which point IIS would recycle the app pool. When we finally reduced blocking we went from 100 VMs to 50 VMs. — Panagiotis Kanavos, Aug 29 '22 at 07:01
@PanagiotisKanavos When a thread is blocked because of IO then it gets completely suspended ie taken off the CPU scheduler list. It cannot block a core, and if there are no other threads running then the CPU will power down. A spinwait is something different and does not happen when the thread is waiting on IO. It's used when doing a short wait for a lock, and while it does block the CPU preventing any other threads, it doesn't actually use a lot of power (electricity/heat), just a lot of time. — Charlieface, Aug 29 '22 at 08:40
@PanagiotisKanavos Yes, I'm not disagreeing with you on any of those points, of course it's a bad idea, due to the extra memory usage and the constant thread context switches. I'm just pointing out that blocking a thread on IO does not cause a spinwait, and that a spinwait block a core but does not use a lot of power. — Charlieface, Aug 29 '22 at 08:53

Cory Nelson · Accepted Answer · 2022-08-26T22:08:37.747

Wait() will block your thread the same as calling a non-async I/O.

Blocking is not inherently inefficient. In fact, it can be more performant if you have a process that will have very few threads. Windows' scheduler actually has some interesting special designs for I/O-blocked threads which you can read about in the Windows Internals books, such as boosting a thread to front of the line if it's been waiting on an I/O for a long time.

However, it doesn't scale. Every thread you create has overhead: memory for stack and register space, thread-local storage used by your app and inside of .NET, cache thrashing caused by all the extra memory needed, context switching, and so on. It's generally not going to be an efficient use of resources especially when each thread will spend a majority of its time blocked.

Async takes advantage of the fact that conceptually we don't really need everything a thread has to offer -- we only want concurrency, so we can make more domain-relevant optimizations in how we use our resources.

It rarely hurts a project to be async by default. If your app doesn't need to be hyper-optimized for scalability, it won't hurt or help you. If your app does, then it'll be a huge help. Things like async/await can just help you model your concurrency better, so regardless of your perf goals it can be useful.

Async I/O is moving towards an even cooler place: I/O APIs like Windows RIO and Linux's io_uring allow you to do I/O without even context switching. Currently .NET does not take advantage of these things, but PipeWriter and PipeReader were built with it in mind for the future.

Thank you, that makes sense. So, you confirm that indeed blocked thread are not really scheduled by the OS anyway ? — lezebulon, Aug 26 '22 at 22:07
Another thing I was wondering where I can't find any info on, is how exactly is C# "managing" all the tasks, which ones to schedule etc. This in itself has to also be complex, and causing some overhead — lezebulon, Aug 26 '22 at 22:08
Keep in mind `Task` is an abstraction, but the default thread pool will be fully managed as part of .NET 7 so you can go read the code for it if you want! — Cory Nelson, Aug 26 '22 at 22:14
@lezebulon you can find [here](https://stackoverflow.com/questions/17258428/thread-sleep-vs-task-delay/72258961#72258961 "Thread.Sleep vs Task.Delay?") an experimental demonstration of creating and awaiting concurrently 100,000 asynchronous operations. The overhead is so small that is hard to measure. You could try starting 100,000 concurrent operations using 100,000 blocked threads, and report your findings! — Theodor Zoulias, Aug 26 '22 at 22:47

Is Blocking code really expensive on modern systems?

1 Answers1