From a C# perspective, you don't catch the exception. Stack overflows in native code are pretty bad. The calling thread may have half an object on the stack, and there's no way you can unwind that.
Realistically speaking, your options are to suspend the offending thread, which means the C++ code won't return to C#. This isn't directly fatal - threads are suspended all the time by the OS, when no CPU core is available. But mutexes will remain locked etc.
While the offending thread is suspended, your C# code should perform an emergency save of all things that can be rescued. Next, schedule a restart (the Windows Task Scheduler can help here), and forcefully terminate the whole process.
This leaves out one "small" detail: catching the stack overflow and suspending that thread. This is probably best done with a Vectored Exception Handler, which is called directly by Windows. In this exception handler, check the exception object to see if it's a stack overflow. If so, you have a problem, since you're now running on a thread that is down to its last stack page (and that too might be almost full). So, you only do two things: call ResumeThread(helperThread)
and then SuspendThread(GetCurrentThread())
. The helper thread wakes up, calls back to C# to initiate the emergency save, and then calls TerminateProcess
.