Task.Run continues on the same thread causing deadlock

Question

Consider the following async method that I'm going to wait synchronously. Wait a second, I know. I know that it's considered bad practice and causes deadlocks, but I'm fully conscious of that and taking measures to prevent deadlocks via wrapping code with Task.Run.

    private async Task<string> BadAssAsync()
    {
        HttpClient client = new HttpClient();

        WriteInfo("BEFORE AWAIT");

        var response = await client.GetAsync("http://google.com");

        WriteInfo("AFTER AWAIT");

        string content = await response.Content.ReadAsStringAsync();

        WriteInfo("AFTER SECOND AWAIT");

        return content;
    }

This code will definitely deadlock (in environments with SyncronizationContext that schedules tasks on a single thread like ASP.NET) if called like that: BadAssAsync().Result.

The problem I face is that even with this "safe" wrapper it still occasionally deadlocks.

    private T Wait1<T>(Func<Task<T>> taskGen)
    {
        return Task.Run(() =>
        {
            WriteInfo("RUN");

            var task = taskGen();

            return task.Result;
        }).Result;
    }

These "WriteInfo" lines there in purpose. These debug lines allowed me to see that the reason why it occasionally happens is that the code within Task.Run, by some mystery, is executed by the very same thread that started serving request. It means that is has AspNetSynchronizationContext as SyncronizationContext and will definitely deadlock.

Here is debug output:

*** (worked fine)
START: TID: 17; SCTX: System.Web.AspNetSynchronizationContext; SCHEDULER: System.Threading.Tasks.ThreadPoolTaskScheduler
RUN: TID: 45; SCTX: &ltnull> SCHEDULER: System.Threading.Tasks.ThreadPoolTaskScheduler
BEFORE AWAIT: TID: 45; SCTX: &ltnull> SCHEDULER: System.Threading.Tasks.ThreadPoolTaskScheduler
AFTER AWAIT: TID: 37; SCTX: &ltnull> SCHEDULER: System.Threading.Tasks.ThreadPoolTaskScheduler
AFTER SECOND AWAIT: TID: 37; SCTX: &ltnull> SCHEDULER: System.Threading.Tasks.ThreadPoolTaskScheduler

*** (deadlocked)
START: TID: 48; SCTX: System.Web.AspNetSynchronizationContext; SCHEDULER: System.Threading.Tasks.ThreadPoolTaskScheduler
RUN: TID: 48; SCTX: System.Web.AspNetSynchronizationContext; SCHEDULER: System.Threading.Tasks.ThreadPoolTaskScheduler
BEFORE AWAIT: TID: 48; SCTX: System.Web.AspNetSynchronizationContext; SCHEDULER: System.Threading.Tasks.ThreadPoolTaskScheduler

Notice as code within Task.Run() continues on the very same thread with TID=48.

The question is why is this happening? Why Task.Run runs code on the very same thread allowing SyncronizationContext to still have an effect?

Here is the full sample code of WebAPI controller: https://pastebin.com/44RP34Ye and full sample code here.

UPDATE. Here is the shorter Console Application code sample that reproduces root cause of the issue -- scheduling Task.Run delegate on the calling thread that waits. How is that possible?

static void Main(string[] args)
{
    WriteInfo("\n***\nBASE");

    var t1 = Task.Run(() =>
    {
        WriteInfo("T1");

        Task t2 = Task.Run(() =>
        {
            WriteInfo("T2");
        });

        t2.Wait();
    });

    t1.Wait();
}

BASE: TID: 1; SCTX: &ltnull> SCHEDULER: System.Threading.Tasks.ThreadPoolTaskScheduler
T1: TID: 3; SCTX: &ltnull> SCHEDULER: System.Threading.Tasks.ThreadPoolTaskScheduler
T2: TID: 3; SCTX: &ltnull> SCHEDULER: System.Threading.Tasks.ThreadPoolTaskScheduler

Answer 1

We with a good friend of mine were able to figure this one out via inspecting stack traces and reading .net reference source. It's evident that the root cause of problem is that Task.Run's payload is being executed on the thread that calls Wait on the task. As it turned out this is a performance optimization made by TPL in order not to spin up extra threads and prevent precious thread from doing nothing.

Here is an article by Stephen Toub that describes the behavior: https://blogs.msdn.microsoft.com/pfxteam/2009/10/15/task-wait-and-inlining/.

Wait could simply block on some synchronization primitive until the target Task completed, and in some cases that’s exactly what it does. But blocking threads is an expensive venture, in that a thread ties up a good chunk of system resources, and a blocked thread is dead weight until it’s able to continue executing useful work. Instead, Wait prefers to execute useful work rather than blocking, and it has useful work at its finger tips: the Task being waited on. If the Task being Wait’d on has already started execution, Wait has to block. However, if it hasn’t started executing, Wait may be able to pull the target task out of the scheduler to which it was queued and execute it inline on the current thread.

Lesson: If you really need to synchronously wait asynchronous work the trick with Task.Run is not reliable. You have to zero out SyncronizationContext, wait, and then return SyncronizationContext back.

Answer 2

In the internals of IdentityServer I found another technique that works. It is very close to not reliable technique laid down in the the question. You will find the source code at the end or this answer.

Credits for the magic of this technique should go to Unwrap() method. Internally when called against Task<Task<T>> it creates a new "promise task" that completes as soon as both (the one we executing against and the nested one) tasks complete.

The reason why this works out and does not create probability of deadlock is simple -- promise tasks are no subjects for inlining and that makes sense as there is no "work" to inline. In turn that means that we blocking current thread and let default scheduler (ThreadPoolTaskScheduler) do the work in the new thread in absence of SynchronizationContext.

internal static class AsyncHelper
{
    private static readonly TaskFactory _myTaskFactory = new TaskFactory(CancellationToken.None, TaskCreationOptions.None, TaskContinuationOptions.None, TaskScheduler.Default);

    public static void RunSync(Func<Task> func)
    {
        _myTaskFactory.StartNew(func).Unwrap().GetAwaiter().GetResult();
    }

    public static TResult RunSync<TResult>(Func<Task<TResult>> func)
    {
        return _myTaskFactory.StartNew(func).Unwrap().GetAwaiter().GetResult();
    }
}

Moreover, there is a signature of Task.Run that does Unwrap implicitly which leads to the shortest safe implementation below.

T SyncWait<T>(Func<Task<T>> f) => Task.Run(f).Result;