IIS6 Application Pool Crash

Question

During recent stress and volume testing, we realized that after 30minutes, all uses gets disconnected from the website. After event logging, it came to our attention that the application pool crashes. Doing some google investigation, apparently in most causes this is due to unhandled exceptions.

SO when the application crashes, the following exception details are displayed:

An unhandled exception occurred and the process was terminated.

Application ID: DefaultDomain

Process ID: 7852

Exception: System.Runtime.Serialization.SerializationException

Message: Type 'FuseFarm.FrameworkException' in Assembly 'FuseFarm, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' is not marked as serializable.

StackTrace:    at System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo.InitSerialize(Object obj, ISurrogateSelector surrogateSelector, StreamingContext context, SerObjectInfoInit serObjectInfoInit, IFormatterConverter converter, ObjectWriter objectWriter)
   at System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo.Serialize(Object obj, ISurrogateSelector surrogateSelector, StreamingContext context, SerObjectInfoInit serObjectInfoInit, IFormatterConverter converter, ObjectWriter objectWriter)
   at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.Serialize(Object graph, Header[] inHeaders, __BinaryWriter serWriter, Boolean fCheck)
   at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize(Stream serializationStream, Object graph, Header[] headers, Boolean fCheck)
   at System.Runtime.Remoting.Channels.CrossAppDomainSerializer.SerializeObject(Object obj, MemoryStream stm)
   at System.AppDomain.Serialize(Object o)
   at System.AppDomain.MarshalObject(Object o)

I have no idea why it's trying to Serialize FrameworkException, and I can't see in the code where this is being done either. But I do see several parts of code where

new FrameworkException(exData, "ContractComposition.SubmitContract");

is being called, but not being handled. After checking the global.asax.cs, the following is happening:

protected void Application_Error(object sender, EventArgs e)
{
    ILog log = LogManager.GetLogger(typeof(Global));
    string environmentName = WebConfigurationManager.AppSettings["EnvironmentName"];
    if (!String.IsNullOrEmpty(environmentName) && (environmentName == "DEMO" || environmentName == "LIVE"))
    {
        Exception currentException = Server.GetLastError().GetBaseException();
        Session["errorMessage"] = currentException.Message;
        Session["errorSource"] = currentException.Source;
        Session["errorTrace"] = currentException.StackTrace;
        log.Error(currentException.Message, currentException);
        if (currentException != null)
        {
            Session["error"] = currentException.GetType().Name;
            switch (currentException.GetType().ToString())
            {
                case "System.ApplicationException":
                case "FuseFarm.FrameworkException":
                    break;
                default:
                    new FrameworkException(currentException.Message + "\n" + currentException.StackTrace, currentException.Source, currentException);
                    break;
            }

        }
        Server.Transfer("~/error.aspx");
    }
}

Throwing a new exception in Application_Error... This doesn't seem right? Who and what will handle this error if it's thrown at this point?

OKay I can see that it wont really raise a new frameworkException as it will break on the case "FuseFarm.FrameworkException"... But still not sure how to tackle this issue. — FaNIX, Sep 16 '11 at 01:31
Exceptions are serialized when they need to cross an AppDomain boundary. The runtime is trying to serialize it for you. All exceptions must property implement `ISerializable` and a serialization constructor for this reason. — vcsjones, Sep 16 '11 at 01:36
why do you already use members of the object even before the check: if (currentException != null) ? anyway? — Davide Piras, Sep 16 '11 at 01:41

score 3 · Accepted Answer · edited May 23 '17 at 11:58

It's Serializing FrameworkException because it is trying to go across an AppDomain boundary.

All objects that go across an AppDomain must be serialized, and an exception is no different.

You can consider it a bug when an Exception does not properly implement serialization.

I don't believe that your error handler is what is the source of the problem. It's hard to tell given that stack trace - a full memory dump would yield better information.

Your best bet is to just properly make the exception serializable.

This may not completely resolve your issue - boiled down, you will still be throwing exceptions. Hopefully once that is corrected you will see the real cause of the problem.

Okay, I made my exception serializable according to that the article you posted. Will do SvT again and report back :) — FaNIX, Sep 16 '11 at 04:23

score 0 · Answer 2 · answered Sep 16 '11 at 01:33

If an exception is thrown in Application_Error, your app pool will crash, just like you're seeing. Based on the code you've shown, I would conclude that the error logger referred to in the line:

log.Error(currentException.Message, currentException);

is attempting to serialize exceptions that are passed into it. When currentException ends up being of type FuseFarm.FrameworkException, the error handler crashes on that line and your app pool shuts down.

If this is the case, you'll need to either mark FuseFarm.FrameworkException as Serializable, change the logger to not attempt to serialize objects, or put a try/catch block in the Application_error handler and do something else with those exceptions if you wish the app pool to keep going.

I don't think that `log.Error` is the reason for the serialization, otherwise the stack trace would say so. I believe it is the runtime trying to cross an AppDomain boundary. (See http://stackoverflow.com/questions/1066701/why-should-i-always-make-my-exceptions-serializable-net) — vcsjones, Sep 16 '11 at 01:40
@vcsjones, I think you're right. You should make your comment above an answer. — Steve Danner, Sep 16 '11 at 01:46

IIS6 Application Pool Crash

2 Answers2