When your Excel function runs there is no SynchronizationContext.Current
installed, so the async/await mechanism will runs the code after await
(including your catch handler) on a ThreadPool thread. That's not a context where you can directly show your WPF form.
Installing a DispatcherSynchronizationContext corresponding to a Dispatcher running on the main thread (or another thread) would work, but you have to do that for every UDF call. Somehow the native code path through Excel loses the .NET call context on the main thread, so the SynchronizationContext gets lost.
Better is probably to assume that the catch handler is running on a ThreadPool thread, and make a SynchronizationContext.Post
call from the catch handler to take you back to the main thread running your Dispatcher and WPF form.
You can look at how Excel-DNA implements the (WinForms) LogDisplay window. (https://github.com/Excel-DNA/ExcelDna/blob/master/Source/ExcelDna.Integration/LogDisplay.cs). You can call LogDisplay.WriteLine(...)
from any thread, and it will do a _syncContext.Post
to run the 'Show' on the main thread.
The C# async/await mechanism works less well with Excel since the native/managed transitions, and whatever Excel does internally, messes up the thread context that needs to flow between continuations. Even on the .NET side, it's not clear how thread context is managed between AppDomains (different Excel add-ins). So it's best not to rely on the .NET runtime being able to thread any kind of context through the managed/native transitions.