How to handle and recover from exceptions within long running subscription thread

Question

I'm using ServiceStack.Redis within several ASP.NET MVC applications in order to facilitate basic messaging between those applications. Within one application I have a class which sets up a subscription and handles any messages that the application is interested in, for example:

public MessageBus(IRedisClientsManager redisClientsManager)
{
    Thread subscriptionThread = new Thread(() => {
        try
        {
            using (var redisClient = redisClientsManager.GetClient())
            using (var subscription = redisClient.CreateSubscription())
            {
                subscription.OnMessage = (channel, message) =>
                {
                    handleMessage(message);
                };
                subscription.SubscribeToChannels("MyChannel");
            }
        }
        catch (Exception ex)
        {
            ErrorLog.GetDefault(null).Log(new Error(ex));
        }
    });
    subscriptionThread.Start();
}

Since "SubscribeToChannels" is blocking, I have it running in a separate thread. I want this thread to stay alive the entire time the MVC application is running and I'm concerned that the thread will just die or the connection to Redis will stop if any sort of exception occurs.

My question is: are there any examples out there of how to recover from exceptions (connection failures, timeouts, etc) that may occur while the subscription is open?

score 0 · Accepted Answer · edited May 23 '17 at 12:11

Regarding exceptions killing a thread, use a long-running loop:

while(!ShutdownRequested) {
    try{...}
    catch(Exception e) {/*Log and probably do some rate-limiting in case of terminal issue*/}
}

Bear in mind that the catch will swallow everything, including OutOfMemory exceptions so you'll want some sanity checking like a fail count / delay so that you don't always retry immediately.

Don't forget that you can retain a reference to the background thread from the parent and check the ThreadState too.

As to hosting in ASP, this is a bad idea (See this answer). When the worker pool is recycled (will happen eventually) the thread will die and won't respawn until a new page is requested (at the earliest).

You should put this into a windows service so it runs when the server starts and if it has to communicate with your site, it should do so over WCF (2-way) or by hitting a Url (for push from service to site).

That way, it will only die when the service is stopped (hopefully only reboots).

Thank you for the answer. Your comments helped me decide on whether or not to continue trying to make the Redis messaging work or stick with my existing WCF service. For now I've decided to stick with our existing services. — jaggedaz, May 30 '14 at 16:50

How to handle and recover from exceptions within long running subscription thread

1 Answers1