Calling a PInvoke in an async doesn't return to the main thread after execution

Question

I'm working with an unmanaged library that mandates that all calls to its API is run on the same thread. We want to use the Reactive extensions's EventLoopScheduler to facilitate that since we'll be using Observable for other things.

I'm using a method similar to the Run method in the code sample below to execute code in the scheduler which will always run on the same thread. When I'm working with managed code this works as expected and all calls are run on the thread managed by the event loop and before / after the async call is the main thread.

But, when I call a P/Invoke (the one in the code sample is just an example, I'm not really calling this one in my code but the behavior is the same), the thread does run on the event loop thread, but so does everything after!

I've tried adding ConfigureAwait(true) (and false) but it doesn't change anything. I'm really confused by this behavior, why would calling a P/Invoke change the thread continuing after the await !!?

Here's the code to reproduce:

[DllImport("user32.dll", CharSet = CharSet.Unicode, SetLastError = true)]
private static extern int MessageBox(IntPtr hWnd, string lpText, string lpCaption, uint uType);

public static Task Run(Action action, IScheduler scheduler)
{
    return Observable.Start(action, scheduler).SingleAsync().ToTask();
}

public static string ThreadInfo() =>
    $"\"{Thread.CurrentThread.Name}\" ({Thread.CurrentThread.ManagedThreadId})";

private static async Task Main(string[] args)
{
    var scheduler = new EventLoopScheduler();

    Console.WriteLine($"Before managed call on thread {ThreadInfo()}");

    await Run(() => Console.WriteLine($"Managed call on thread {ThreadInfo()}"), scheduler);

    Console.WriteLine($"After managed call on thread {ThreadInfo()}");

    Console.WriteLine($"Before PInvoke on thread {ThreadInfo()}");

    await Run(() => MessageBox(IntPtr.Zero, $"Running on thread {ThreadInfo()}", "Attention", 0), scheduler);

    Console.WriteLine($"After PInvoke on thread {ThreadInfo()}");
}

The execution returns something like this:

Before managed call on thread "" (1)
Managed call on thread "Event Loop 1" (6)
After managed call on thread "" (1)
Before PInvoke on thread "" (1)
Message box displayed with text: Running on thread "Event Loop 1" (6)
After PInvoke on thread "Event Loop 1" (6)

Where I expected

Before managed call on thread "" (1)
Managed call on thread "Event Loop 1" (6)
After managed call on thread "" (1)
Before PInvoke on thread "" (1)
Message box displayed with text: Running on thread "Event Loop 1" (6)
After PInvoke on thread "" (1)

Answer 1

Tasks

A Task or a promise is a just an abstraction for callbacks. And async/await is just syntactic sugar for tasks.

Since it's a callback abstraction, await doesn't block the thread. Why does it look like it's blocking? That's because await rewrites your code into a state-machine which advances through its states when the Task being awaited completes.

It roughly gets rewritten like this:

switch (state)
{
    case 0:
        Console.WriteLine($"Before managed call on thread {ThreadInfo()}");
        Await(Run(() => Console.WriteLine($"Managed call on thread {ThreadInfo()}"), scheduler));
        return;
    case 1:

        Console.WriteLine($"After managed call on thread {ThreadInfo()}");
        Console.WriteLine($"Before PInvoke on thread {ThreadInfo()}");
        Await(Run(() => MessageBox(IntPtr.Zero, $"Running on thread {ThreadInfo()}", "Attention", 0), scheduler));
        return;
    case 2:
        Console.WriteLine($"After PInvoke on thread {ThreadInfo()}");
        return;
}

The actual rewrite uses goto rather than a switch, but the concept is the same. So when a task completes, it call this state-machine with state += 1 - in the same threading context. You only see task pool threads when you use a task scheduler.

Leaks in the abstraction

The explanation for why you see this particular behavior:

After managed call on thread "" (1)

is quite complicated. It has to do with whether a scheduled thunk completes immediately or not. If you add a Thread.Sleep in the first managed call, you'll notice that the continuation runs on the event loop thread.

This is due to scheduling optimizations preferring to only queue if something is currently running. When you call ToTask(), you're using the default scheduler, which is the current thread scheduler.

The current thread scheduler works like this:

Free? Run immediately.

Busy? Queue the work.

The run immediately behavior is why you see the log running on the main thread. If you just add a

var scheduler = new EventLoopScheduler();
scheduler.Schedule(() => Thread.Sleep(1000));

to the very beginning, you make the event loop busy, causing everything to go into queuing, so you then see everything logs in the event loop thread. So this has nothing to do with P/Invoke.

To be clear, this isn't about the schedulers being specified for observing, but subscribing. When you convert Observables into other abstractions like Tasks, Enumerables, Blocking Joins etc., some internal complexity may leak through.