Deadlock in WPF: How to not block the GUI thread in HwndHost.BuildWindowCore?

Question

We have a large code base for processing and visualizing images in C++ with Qt. Now a user wants to build on that, but their code base is .NET Core 3.1 with WPF. We use PInvoke to interface the native code, and can very successfully join the projects. The few native widgets are embedded with Win32-in-WPF-HwndHost wrappers. Everything works pretty good and we're quite happy about the mileage we're getting!

There is just one blocker problem: The HwndHost method BuildWindowCore can sometimes hang in a deadlock. We could identify the root cause of the deadlock:

When BuildWindowCore calls into Qt to re-parent the native widget into the managed widget's handle, we use a blocking call to ensure the re-parenting completed. However during re-parenting the widget, Qt sometimes calls DefWindowProc to pass unhandled window messages back to the parent WPF widget. Since the WPF thread is blocked with calling Qt, this is a circular blocking wait, ending in a deadlock.

While I can understand the problem, we are not knowledgeable enough about the WPF GUI thread to resolve the issue.

What we tried so far:

• Make the call to Qt in the background (with await) but BuildWindowCore can not be an async method.
• Move the re-parenting out of BuildWindowCore, and call it async later, but the HwndHost widget requires the re-parenting to take place in BuildWindowCore, or we get a WPF error that the native widget handle is not (yet) a child widget of the WPF widget.

I'm a bit at the end of my wit. We could make the call to native code non-blocking on C++ side, and poll for its completion in a loop in C#. But how would we give the control back to the WPF GUI thread while we poll for Qt to re-parent the widget?

The most closely related SO answer suggests using await Dispatcher.Yield(DispatcherPriority.ApplicationIdle), but this was in an async method.

In pseudo-code, I'm thinking about something like:

protected override HandleRef BuildWindowCore(HandleRef HWNDParent)
{
    NativeCode.beginReParenting(HWNDParent.Handle);

    while (!NativeCode.reParentingCompleted()) {
        // This method is async, is it clean to call it like this?
        Dispatcher.Yield(DispatcherPriority.ApplicationIdle);
    }

    return new HandleRef(this, NativeCode.getEmbeddedWidgetHandle());
}

My questions:

• Would Dispatcher.Yield() (or a similar concept) in a polling loop help to keep WPF responsive?
• How can we cleanly call the async Dispatcher.Yield() method in the GUI thread, when BuildWindowCore itself can not be async?
• Is this an "ok" solution, or are there better ways to not block the WPF GUI thread while calling HwndHost.BuildWindowCore()?

Answer 1

Your WPF pump (Dispatcher) is blocked. Async / await / Yield will not help you since you are not in an async function.

You need WPF to keep pumping while it is sat inside BuildWindowCore(). You can do this by calling your native code on a different thread, then have your WPF thread sit and pump messages until the worker thread is complete. You can do this using Dispatcher.PushFrame.

Something like this hopefully:

protected override HandleRef BuildWindowCore(HandleRef HWNDParent)
{
    // call into QT in worker thread
    var reParentTask = Task.Run(() => NativeCode.beginReParenting(HWNDParent.Handle));

    // pump messages while we wait for QT to do its stuff
    var frame = new DispatcherFrame();
    Dispatcher.CurrentDispatcher.BeginInvoke(DispatcherPriority.Background,
        new DispatcherOperationCallback(delegate (object f)
            {
                ((DispatcherFrame)f).Continue = !reParentTask.IsCompleted;
                return null;
            }),frame);

    Dispatcher.PushFrame(frame);

    return new HandleRef(this, NativeCode.getEmbeddedWidgetHandle());
}