Utilizing Async/Await in .NET Console applications breaks when calling Application.Run() or instantiating a WinForms UserControl object

Question

Background

Async/Await facilitates responsive applications in .NET by automatically creating a "state machine", allowing the primary thread of an application to remain responsive even while performing blocking work.

Windows Forms, WPF, and ASP.NET (to my knowledge) all incorporate a form of SynchronizationContext (although ASP.NET may have removed this recently; I'm not positive, as I don't work with it.)

I've recently needed to extend a Windows Forms application to also support accepting arguments from the Command Line, and in doing so, discovered Async/Await stopped working. After some number of (almost random) steps into my application, it would either hang or return to an incorrect point, effectively halting.

SynchronizationContext

After research, I discovered that under the covers, Async/Await relies on a SynchronizationContext to effectively handle routing machine state (as mentioned above.) What wasn't clear is what happened without a SynchronizationContext: Stephen Toub (on his blog post here) indicates that Async/Await will execute, but without thread affinity, and that without a SynchronizationContext, Async/Await can end up executing on random threads.

Stephen goes on to explain "AsyncPump.cs", his class for implementing a SynchronizationContext for console applications, and in testing AsyncPump, so far, it's been successful.

Questions

1. Stephen's post is from 2012; is there another solution? Perhaps his AsyncPump class has been integrated (and/or modified) into a more recent version of .NET? I would prefer to use an library-designated equivalent, if available, such so that if any changes occur to the under-the-covers implementation of Async/Await, it will automatically be updated as well, like the WindowsFormsSynchronizationContext would be.
2. Could I safely use the WindowsFormsSynchronizationContext? In Program.cs, I'm determining whether or not I want to instantiate and open a Form, using Application.Run() to do so, which automatically handles setting up a SynchronizationContext for me (as well as message pump, etc.) I tried instantiating a WindowsFormsSynchronizationContext and setting it on my main thread using SynchronizationContext.SetSynchronizationContext(), and although this compiles, I encountered the same problems as when I had no SynchronizationContext at all.

I'm looking for the best practice for supporting Async/Await in a console application, because (as far as I can tell) it definitely needs a SynchronizationContext in order to execute correctly.

---

Edit 1: Adding pseudocode to help illustrate the scenario

If my program has received more than one argument, I'm assuming that it's been invoked from the Command Prompt, and have created a custom "MyCustomConsole" class which uses P/Invoke to Win32 to call AttachConsole(-1). At this point, I can read/write from the CLI as my program was a Console application. If I haven't received any extra arguments, then I can launch a Windows Form GUI as expected ("Application.Run(new Form1());").

The problem is that the code I end up invoking to perform blocking operations ("RunBlockingOperationsAsync()") is Async/Await to remain responsive, and when invoked via the GUI (through "Application.Run()"), works fine. If I try to call "RunBlockingOperationsAsync" without "Application.Run()", the program deadlocks or jumps to unexpected areas whilst debugging, effectively crashing.

I tried implementing a WindowsFormsSynchronizationContext, but that fails in the same manner. However, utilizing Stephen Toub's "AsyncPump.cs" solution corrects the problem (see below.)

There must be a built-in .NET framework piece for this, right? I can't believe Async/Await could be so thoroughly implemented without a default implementation for Console applications. My current understanding is that Async/Await utilization within a Console application without Stephen's "AsyncPump.cs" class (or similar) would not execute properly; effectively, this makes using Async/Await in a Console application unusable as-is by default.

It seems like Console applications should have an equivalent version of "Application.Run()", which initializes an appropriate SynchronizationContext (and whatever else might be necessary—maybe nothing right now.)

using System;
using System.Linq;
using System.Threading.Tasks;
using System.Windows.Forms;
using System.Threading; // <-- Note that System.Threading is required for SynchronizationContext.

namespace WindowsFormsApp1
{
    static class Program
    {
        /// <summary>
        /// The main entry point for the application—NOTE this is the default WinForms implementation for 'Program.cs'.
        /// </summary>
        [STAThread]
        static void Main()
        {
            Application.EnableVisualStyles();
            Application.SetCompatibleTextRenderingDefault(false);

            MainAsync();
        }

        private static async Task MainAsync()
        {
            // If the application has received more than one argument, assume it's been invoked from the Command Prompt.
            if (Environment.GetCommandLineArgs().Count() > 1)
            {
                using (MyCustomConsole mcc = new MyCustomConsole())
                {
                    SynchronizationContext sctx = SynchronizationContext.Current;   // <-- Initializes sctx to NULL, as at this point in the program,
                                                                                    // there is no SynchronizationContext. It is initialized when
                                                                                    // "Application.Run()" is invoked.

                    // Doesn't work (no SynchronizationContext):
                    await mcc.Run();                                    // <-- If the MyCustomConsole class is invoked without using AsyncPump.cs,
                                                                        // it has no SynchronizationContext, and without it, Async/Await operations can
                                                                        // execute on any thread from the ThreadPool, which causes deadlocks and jumping
                                                                        // (almost at random?) to unexpected parts of my program, which I can only attribute
                                                                        // to the size of the program and including numerous nested Async/Await calls, depending
                                                                        // on what the program is trying to do.

                    // Perhaps instantiate a WindowsFormsSynchronizationContext and use it?
                    SynchronizationContext.SetSynchronizationContext = new WindowsFormsSynchronizationContext();
                    await mcc.Run();                                    // <-- Also fails in the same manner as above, despite having a SynchronizationContext.
                                                                        // I don't understand why.

                    AsyncPump.Run(async () => { await mcc.Run(); });    // <-- This works. AsyncPump.cs is the custom SynchronizationContext that
                                                                        // Stephen Toub provided in his blog. It not only handles SynchronizationContext,
                                                                        // but sets itself as the SynchronizationContext for the current thread, which
                                                                        // is required for Async/Await to operate with thread affinity.
                }
            }
            else // Otherwise, display the main form and operate with a GUI.
            {
                Application.Run(new Form1());   // <-- Application.Run() instantiates a WindowsFormsSynchronizationContext,
                                                // (amongst other things, like a message pump) and this is vital to a proper
                                                // Async/Await machine state that requires thread affinity.
            }
        }
    }
}

---

Resolution

The root of this problem is two-fold: First, a developer using Async/Await should understand that Async/Await's implementation can differ depending on SynchronizationContext; Stephen Toub does an excellent job explaining here. Understanding that a Console application does not have a specific SynchronizationContext by default, continuations are posted to the ThreadPool. If you debug a Console application, you would find that monitoring SynchronizationContext.Current is NULL.

Second, recognize that (for Windows Forms) Application.Run() sets up a Message Pump and a single-threaded SynchronizationContext. Monitoring SynchronizationContext.Current after Application.Run() would return a WindowsFormsSynchronizationContext object. Thanks to @noseratio, I've learned that instantiating a Windows Forms UserControl object will also instantiate and set SynchronizationContext.Current to use the new WindowsFormsSynchronizationContext, but only if it was NULL to begin with.

This explains my problem: The application I'm working on is a Windows Forms application, and when typically started, Application.Run() is used to invoke the Message Pump and also sets up a WindowsFormsSynchronizationContext. Async/Await works perfectly. However, when adding on support for CLI, I instantiated an object that derives from UserControl. As soon as I instantiate it, my formerly-NULL SynchronizationContext is now a WindowsFormsSynchronizationContext, and now Async/Await continuations are posted to it instead of the ThreadPool—what happens to continuations on the ThreadPool after a new SynchronizationContext is instantiated, I can't say. I experienced erratic program behavior, typically either "await Task.Delay()" calls hanging indefinitely, or control of my application (in the debugger) jumping around seemingly at random. Reportedly, setting (WindowsFormsSynchronizationContext.AutoInstall = false) should prevent automatically replacing a NULL SynchronizationContext with a WindowsFormsSynchronizationContext, but in my testing, it was still replaced (and Async/Await still broke.)

I did not test this with WPF, but I expect WPF would behave similarly (and/or developers would face a similar problem.)

There are multiple solutions:

1. The best solution, in my opinion, is to not instantiate a Windows Forms UserControl (or WPF equivalent) when you're executing in CLI mode, if you can help it. Abstract work into it's own classes and leave UserControls (and their equivalents) to View abstractions if possible. This allows Async/Await to run on whatever Synchronization Context your application needs: If Windows Forms, a WindowsFormsSynchronizationContext. If WPF, a Dispatcher (?) SynchronizationContext. If a Console application, it runs on the ThreadPool instead of a SynchronizationContext.

2. Explicitly set your own SynchronizationContext: @Stephen Toub's AsyncPump class; or @Stephen Cleary's AsyncContext class; or either of @TheodorZoulias's solutions worked (in my testing.) There may be good reason for using one of these solutions over #1, for example you may be working on a Console application, but have no choice but to instantiate a WinForms UserControl, or perhaps use a library that does so under-the-hood, unbeknownst to you. I would suggest monitoring SynchronizationContext.Current in various stages of an application if faced with this scenario.

Answer 1

Using Stephen Toub's AsyncPump seems sufficient. You could also try starting a standard message loop with Application.Run() (without a form), and run your code inside the Application.Idle event handler (handled only once). This way you can also interact with UI elements if it's needed for some reason (with a WebBrowser control for example).

if (Environment.GetCommandLineArgs().Count() > 1)
{
    EventHandler handler = null;
    handler = async (sender, e) =>
    {
        Application.Idle -= handler;
        using (MyCustomConsole mcc = new MyCustomConsole())
        {
            await mcc.Run();
        }
        Application.ExitThread();
    };
    Application.Idle += handler;
    Application.Run(); // Begins running a standard application message
                       // loop on the current thread, without a form.
}

---

Update: Another idea is to use a Dispatcher, the object used for thread synchronization in WPF applications. The Dispatcher creates automatically a DispatcherSynchronizationContext, so all awaited continuations that lack ConfigureAwait(false) will run in the same thread. A reference to the assembly WindowsBase.dll is needed.

using System.Windows.Threading;

if (Environment.GetCommandLineArgs().Count() > 1)
{
    var dispatcher = Dispatcher.CurrentDispatcher;
    var invokeTask = Task.Run(async () =>
    {
        try
        {
            await dispatcher.Invoke(async () =>
            {
                using (MyCustomConsole mcc = new MyCustomConsole())
                {
                    await mcc.Run();
                }
            });
        }
        finally
        {
            dispatcher.InvokeShutdown();
        }
    });
    Dispatcher.Run(); // blocking call
    await invokeTask; // await the task just to propagate exceptions
}

The Task.Run is needed so that the dispatcher.Invoke is called from a thread-pool thread, as well as the final shutdown of the dispatcher. Everything else happens in the main thread.

Answer 2

In the absence of synchronization context (or when the default SyncrhonizationContext is used), it's often possible for an await continuation to run synchronously, i.e., on the same thread where its antecedent task has ended. That can lead to obscure deadlocks, and it was one of the reasons TaskContinuationOptions.RunContinuationsAsynchronously was introduced in .NET Framework 4.6. For some more details and examples, check out this blog post: The danger of TaskCompletionSource class.

The fact that AsyncPump stops your code from hanging indicates you may have a similar situation somewhere inside mcc.Run(). As AsyncPump imposes true asynchrony for await continuations (albeit on the same thread), it reduces the chance for deadlocks.

That said, I'm not suggesting using AsyncPump or WindowsFormsSynchronizationContext as a workaround. Rather, you should try to find what exactly causes your code to hang (and where), and solve it locally, e.g. simply by wrapping the offending call with Task.Run.

One other issue I can spot in your code is that you don't wait or await the task returned by MainAsync. Because of that, at least for the console branch of your logic (especially without using AsyncPump), your program may be ending prematurely, depending on what's going in inside mcc.Run(), and you may be letting some exceptions go unobserved.

Answer 3

I'm looking for the best practice for supporting Async/Await in a console application, because (as far as I can tell) it definitely needs a SynchronizationContext in order to execute correctly.

async/await does not require a context. In the absence of a context, it will use the thread pool context. However, code that uses async/await can certainly make assumptions about threads. In your situation, it sounds as though your code is expecting to run in a single-threaded context. Since it was developed in a single-threaded context (WinForms), that's not surprising.

So the "best practice" for async/await in a console application is to just run it directly, without a context. But that's not possible in your case because the code you're trying to reuse assumes a single-threaded context.

Stephen's post is from 2012; is there another solution? Perhaps his AsyncPump class has been integrated (and/or modified) into a more recent version of .NET? I would prefer to use an library-designated equivalent, if available, such so that if any changes occur to the under-the-covers implementation of Async/Await, it will automatically be updated as well, like the WindowsFormsSynchronizationContext would be.

It has not been included in .NET.

There are a couple of options for including a message pump. One is to use a Windows Forms UI thread; another is a WPF UI thread. It's been a while since I've done either, but last time I checked the WPF approach was easier to get running, since WPF (unlike WinForms) was designed to allow multiple UI threads.

If you don't actually need a user interface (i.e., STA) thread with a message pump, you can also use a single-threaded context of your own. I wrote an AsyncContext type (docs) that I have used for this in the past. Unlike the UI contexts, it does not use a Windows message queue. As a single-threaded context, it does have a queue, but it is a queue of delegates.