Hijacking UnhandledException to keep Multi-Threaded Application Running

Question

Situation
Well, as you can see below, I have a main App, creating a thread, which creates a bunch of background workers. When the RunWorkerCompleted is fired on some BG worker, I sometimes end up dropping an Unhandled Exception, (for a reason I have clearly pinpointed, but this is not the matter).

architecture

So the Logs clearly show that my UnhandledException handler is entered and reports the exception. I know I am deliberately misusing the OnUnhandledException by stopping the App from exiting, using a Console.Read(). I did it on purpose because I want a human intervention/check before terminating the app.

    public static void OnUnhandledException(object sender, UnhandledExceptionEventArgs e)
    {
      Logger.log("UNHANDLED EXCEPTION : " + e.ExceptionObject.ToString());
      Mail.Sendmail("ADMIN ALERT : " + e.ExceptionObject.ToString());
      Console.Read(); // Yes, this is an ungraceful trick, I confess.
    }

However, what was supposed just to be a "pause" before manual exit turned out to keep the App alive, ie, the Thread is still generating workers, and running as if nothing happened. Even Weirder : The UnhandledException is still being dropped from time to time when it happens again. And it is logged each time, behaving just as if it was a plain ol' try catch. (This was the default behaviour before .Net 3.0)

Question
So, why is everything happening as if the UnhandledException was not thrown, and the thread and BG's keeping running as if nothing happened ? My guess is that as long as the OnUnhandledException handler is not done, and the App is still alive, all running threads that are still alive keep on living in a free world and doing their job.

This gives me a bad temptation, which is to keep this design, as a try/catch to unHandled Exception.
I know this is not a tidy idea, as you never know if the exception is serious or something that could be skipped. However, I would really prefer my program to keep running all day and watch the log report / mail alerts and decide for myself if it needs to be restarted or not. As this is a server application that needs to be running 24/7 I wanted to avoid ill-timed and repetitive interruptions due to minor still unhandled exceptions. The biggest advantage to that being that the server program keeps running while we are tracking down unhandled exceptions one by one and deliver patches to handle them as they occur.

Thank you for your patience, and please feel free to give feedback.

PS : No, I did not smoke pot.

A good question will not be downvoted, even if you're doing it all wrong ;-) — Steven, Jan 19 '12 at 10:17
I think your guess about why it is happening is correct. Is it possible to implement this application as a Windows service instead of a Console app? That way, you can have the service auto restart when it fails due to an un-handled exception. — Ɖiamond ǤeezeƦ, Jan 19 '12 at 10:27
@Ɖiamond ǤeezeƦ Nice Alt-0423 R you got there. So Geeky ! :) Yes, you are partly right and we are planning to refactor the app as a service. However, this would cause a stop/restart, and as it a binary file recorder we would prefer to keep it all in one block, rather that handling the stop/restart/rewrite and merge data. I dont know if I am being clear here. Anyway, I think you get the point. — Mehdi LAMRANI, Jan 19 '12 at 10:41

H H · Accepted Answer · 2012-01-19T11:30:33.780

3

My guess is that as long as the OnUnhandledException handler is not done, and the App is still alive, all running threads that are still alive keep on living in a free world and doing their job.

Correct. But your other threads are now doing their job in a potentially unstable process. There are good reasons for terminating the App when an unhandled exception happens.

I would really prefer my program to keep running

Besides the (potential) stability problems, you would accumulate the halted state of whatever is involved in the unhandled handling. My guess is that you would at least leak a Thread each time it happens. And you can't spare too many of them.

So it would be a very bad design, and only delay the crashing.

edited Jan 19 '12 at 11:30

answered Jan 19 '12 at 10:43

H H

263,252
30
330
514

This is the kind of considerations my Good-Developer Angel was telling me, but he got beaten by the Evil-Developer Devil. Until you came and gave him revenge :) – Mehdi LAMRANI Jan 19 '12 at 10:54
@Henk: If you've specifically designed your BackgroundWorker threads to die (and maybe then be rebooted) in the event of a unhandled exception, then I don't see a problem with letting them die whilst keeping the process running. A background thread is more likely to be brought down by a non-fatal (to the process) exception than something nasty like OutOfMemory. – HTTP 410 Jan 30 '12 at 18:14

Jaster · Answer 2 · 2012-01-19T10:48:39.617

2

First of all you need to subscribe the UnhadledThreadException handler. There you can not "Hijack" the exception, but you have access to it and also to the thread it belongs. What you can do now is putting the throwing Thread into suspetion. The Thread will be still alive!! This however is a bad, dirty hack! Do not do this!

Take a look at this article for some details.

I recommend switching your threading model to TPL (design remains). There you have access to those ugly cross-thread excepions and can handle/supress em gracefully.

edited Jan 19 '12 at 10:48

answered Jan 19 '12 at 10:34

Jaster

8,255
3
34
60

This seems interesting, however a bit unclear in terms of expression. Would you please mind giving more details, sources, and/or examples of that? Thank you – Mehdi LAMRANI Jan 19 '12 at 10:37
i have added a like how to catch an unhadled thread exception in winforms (can be done for WPF, etc aswell). – Jaster Jan 19 '12 at 10:50
http://stackoverflow.com/questions/2707295/how-to-handle-all-unhandled-exceptions-when-using-task-parallel-library for tpl handling – Jaster Jan 19 '12 at 10:51
Thank you I'll have a look at it – Mehdi LAMRANI Jan 19 '12 at 10:52
As of TPL, i'm still stuck with 3.5 Fwk for now :( – Mehdi LAMRANI Jan 19 '12 at 11:04
:/ without tpl its going to be quite a pain in the ****. Exception handling is the main reason why I switch existing threading models to tpl. – Jaster Jan 19 '12 at 11:11

HTTP 410 · Answer 3 · 2012-01-19T13:01:11.503

If you want to allow a BackgroundWorker thread to fail without killing the whole process, there's a cleaner way to do this.

First, intercept the RunWorkerCompleted event. This event is raised when the BackgroundWorker thread terminates, either normally or via an unhandled exception. Then in your event handler, check for the unhandled exception:

// Runs when the BackgroundWorker thread terminates.
private void ThreadFinished(object sender, RunWorkerCompletedEventArgs e)
{
    // Process any unhandled exception
    if (e.Error != null)
    {
        this.LogError(e.Error.Message, e.Error.StackTrace, blah, blah);

In this way, you can let the BackgroundWorker thread die and log any exception without terminating the whole process.

This is not about _unhandled_ exceptions. The OP didn't disclose the real code. The assumption is that an exception bypasses this and all other forms of catch. — H H, Jan 30 '12 at 18:58

Hijacking UnhandledException to keep Multi-Threaded Application Running

3 Answers3