3

My question is not applicable only to C# and Asp.net but it is easier for me to ask more specifically.

When an Asp.net request awaits an async IO operation, the thread basically goes to the thread pool to be reused by other requests. Why is this more efficient than just sleeping the thread until the IO operation is finished? After all, when the thread is returned to the thread pool, its stack needs to be kept in memory to finish the original request. My assumption is that we cannot reuse the memory allocated to the thread unless we copy used stack memory somewhere else and copying data may introduce additional overhead that may not be justified.

Am I missing something? Are my assumtions wrong? Please explain.

EDIT: The answer pointed by jlew is missing one point. What happens to the stack memory used by the request when the thread is returned to the pool? If we cannot reuse the memory, then what is the point of reusing the thread? If we want to reuse the part of the stack that is not used, then we will have to move some memory around. If so, does moving memory around and reusing unused stack memory improve overall efficiency?

Vakhtang
  • 307
  • 5
  • 9
  • 1
    It is not more efficient if you are strictly looking at that one single operation. If you are looking at an application as a whole that uses many threads like a web app where threads are in demand especially with higher loads then it becomes efficient overall as the threads that were serving I/O bound requests can now be freed to serve other requests while the I/O completes. – Igor Dec 15 '16 at 21:05
  • 1
    Possible duplicate of [Why use async requests instead of using a larger threadpool?](http://stackoverflow.com/questions/9453560/why-use-async-requests-instead-of-using-a-larger-threadpool) – jlew Dec 15 '16 at 21:14
  • Thread.Sleep is a blocking call. Do not use it unless you are testing some lengthy operation. https://stackoverflow.com/questions/8815895/why-is-thread-sleep-so-harmful – William Xifaras Dec 15 '16 at 21:18
  • I know that threads that wait for I/O operations to complete can be freed and reused. I want to know what real benefits are. Where does the stack go? Is it copied or new stack is allocated for a subsequent request? Why not just increase thread count in a thread pool? Does thread introduce any significant overhead other than stack memory? – Vakhtang Dec 15 '16 at 21:19
  • @Vakho see the link from jlew. Should be enough to answer your question(s). – William Xifaras Dec 15 '16 at 21:23
  • Thank you for your responses. I read the other post that jlew suggested. It is missing one point though. What happens to the stack memory used by the request when the thread is returned to the pool? If we cannot reuse the memory, then what is the point of reusing the thread? If we want to reuse the part of the stack that is not used, then we will have to move some memory around. If so, does moving memory around and reusing unused stack memory improve overall efficiency? – Vakhtang Dec 15 '16 at 21:32
  • I have this article bookmarked and it is pretty detailed. It does not go into the cpu caching and whatnot, however, there is one line under the section "Why Not Increase the Thread Pool Size?" that states - "In contrast, the memory overhead for an asynchronous operation is much smaller. So, a request with an asynchronous operation has much less memory pressure than a request with a blocked thread" - I wish that would have been stated in more detail. - https://msdn.microsoft.com/en-us/magazine/dn802603.aspx – Ross Bush Dec 15 '16 at 21:51
  • Probably relevant: [There is no thread!](http://blog.stephencleary.com/2013/11/there-is-no-thread.html) – John Wu Dec 15 '16 at 22:17
  • You seem to be one of the people who question the "always use async IO" mentality. Good on you. Indeed, sync IO often *is* faster. It stops being viable at very high degrees of parallelism because threads consume memory and OS resources. These DOP levels are *very* high like 100s to 1000s of threads. Almost all ASP.NET IO should be synchronous with a few strategic places being async for a clear reason. If threads are not scarce, which often is the case, then saving them is pointless. – usr Dec 15 '16 at 22:23

1 Answers1

10

Why is this more efficient than just sleeping the thread until the IO operation is finished?

Think of threads as workers. Workers are expensive. Do you want to pay workers to sleep? No. You want to pay workers to do as much work as possible; if they're blocked, you get them working on something else rather than sleeping until the blockage is cleared up.

Threads are insanely expensive. If threads were cheap then sure, we could make lots. You only use pooling strategies for expensive resources.

Threads are expensive primarily in two ways: you mention the size of the stack, which is 1MB of reserved virtual memory per thread. But there are also significant costs at the OS level, which is not optimized for scenarios with thousands of threads. Determining which thread gets to run next, context switching to it, and switching away from it, all this has non-zero cost that increases as the number of threads increases. Ideally you want n independent threads in an n-processor machine, no more, no less.

After all, when the thread is returned to the thread pool, its stack needs to be kept in memory to finish the original request.

I can't make heads nor tails of this sentence. When the thread goes back to the pool its stack pointer is back at the bottom.

My assumption is that we cannot reuse the memory allocated to the thread unless we copy used stack memory somewhere else and copying data may introduce additional overhead that may not be justified.

I am beginning to make sense of it. Your mental model is:

  • stack is the reification of continuation (what are we doing next?) and activation (what are the values of locals in this method activation?)
  • awaited tasks that complete must continue at the point in the workflow where they left off, with the locals unchanged
  • therefore the continuation/activation information -- the stack -- must be copied somewhere, or something.

This mental model is plausible but wrong. Asynchronous workflows build state machines and bundle up the continuation as states in that machine, and then store references to the state machine in the task. Activation information is hoisted from stack slots into fields of a closure class. This gets the continuation/activation information off the stack and into the heap.

Now, keep in mind that the continuation of a task does not contain all the information that is on the stack; it's not a true continuation in the "call with current continuation" sense that it captures what happens when the current method completes. It captures what happens when the currently awaited task completes, which is sufficient.

Remember, stacks can only be used as the reification of continuation when the "what am I going to do next?" workflow is logically a stack -- where the thing you do next is the thing on the top of the stack. But asynchronous workflows form a tree of dependencies where the join points are awaits; it's not a stack in the first place, so representing the continuations as a stack is a non-starter.

What happens to the stack memory used by the request when the thread is returned to the pool?

The million bytes of virtual memory reserved for a thread stack stays reserved as long as the thread continues to live. This is important to realize! This is one reason why threads are so expensive in .NET. That 1MB of virtual memory is 100% in use as far as the OS memory manager is concerned, no matter how much of that stack has been pushed to. Most of that 1MB is garbage for most of its lifetime.

The pointer to the high water mark of that stack gets moved back to the beginning when the thread goes back to the pool, and everything beyond it is now garbage.

If we cannot reuse the memory, then what is the point of reusing the thread?

We can and do reuse the memory; the stack pointer is reset.

If so, does moving memory around and reusing unused stack memory improve overall efficiency?

I'm not following the thrust of this question; I suspect it is based on faulty premises. Can you rephrase it?

Eric Lippert
  • 647,829
  • 179
  • 1,238
  • 2,067
  • Thank you Eric. I guess I was close enough, because you hit the spot. The only thing I was missing from the puzzle was the fact that "Asynchronous workflows build state machines and bundle up the continuation as states in that machine, and then store references to the state machine in the task. This gets the continuation information off the stack and into the heap.". Thank you very much. No more clarification needed. – Vakhtang Dec 15 '16 at 22:06