If you're hooking the AppDomain.UnhandledException event, then the issue isn't the fact that they might be calling back on a background thread, but rather than the third-party library is explicitly swallowing exceptions thrown by the handlers. This is a badly behaving library, but, since you're writing the event handler code, you can catch the exceptions in the event handler and force the application to shut down.
Since the third-party library can't be stopped by throwing an exception, you can force the thread to terminate by calling:
Thread.CurrentThread.Abort();
Thread.Abort()
is generally considered a bad practice, but it's slightly less dangerous here, as you are aborting yourself and therefore won't be injecting the exception into potentially nasty locations (you also know that the thread is not in an unmanaged context, so it will abort right away.) It's still a nasty hack, but the third-party application isn't giving you much choice. ThreadAbortException cannot be 'stopped', so the third-party code will terminate its thread as soon as it reaches the end of its exception handlers (even if they try to swallow it). If the third-party library is really nasty, it might invoke lots of code in its catch or finally blocks, even using dirty tricks to stay alive like launching other threads, but it would have to be pretty malicious to do this.
To get the friendly error message behavior, you can marshal the Exception to your UI thread using a SynchronizationContext or Dispatcher to call your exception-handler display (the easiest way may be to just re-throw the exception; I recommend embedding it as an InnerException so that you don't lose stack trace.)
If you really distrust the library (and don't want to give it the opportunity to stay alive even through its catch and finally blocks), you can explicitly marshal to the UI thread with a blocking invoke (to display your error message in a friendly way) and then call Environment.FailFast()
to kill your process. This is a particularly harsh way to terminate (it will not call any cleanup code and literally tears down the process as quickly as possible), but it minimizes the possibility for any potentially damaging side effects if application state has become very corrupted. One possible thing to watch for here is that the UI thread could be blocked by some other UI invocation tied to the third-party code. If this happens, the application will deadlock. Again, this would be a pretty malicious thing for it to do, so you can probably ignore it.