7

We are seeing, under load, session data becoming corrupted or missing, but the sessions themselves remain present.

Our website is hosted on IIS 7 running ASP.Net 4.0 and using InProc session state in a web-farm with a total of 4 servers behind a Cisco ACE Appliance load balancer.

At this time the issue is random, we cannot reproduce the issue at will. This web application has been functioning correctly for the last seven months.

We realize Microsoft does not recommend using InProc with web farms, even if sticky sessions are being used.

We do have a lab environment, reasonably identical to our production environment, but are unable to reproduce there under substantial load (we use WAPT).

In our production environment, we have tried isolating just one server behind the load balancer in order to eliminate "server hopping" caused by the load balancer itself. The problem persists, however, even when running on one server. We do not see any AppPool or IIS recycling happening at all in production. As a standard practice, we do recycle the production app pools daily at 3am EST, and have enjoyed months of uptime on the OS.

What's being stored in session are a wide variety of objects, from simple types (integers and strings), to our entire shopping cart (a complex object graph), and even a user control (.ascx) instance. Due to the inability to easily serialize many of these objects, we are unable to make the switch to out-of-process session storage in a reasonable period of time.

Someone suggested trying to capture an HTTP session using Fiddler. The issue with running Fiddler is that we can't intentionally reproduce the issue ourselves. So, this leaves us unable to capture an HTTP trace of a failure event as it occurs. The trace logs from WAPT in our lab would likely provide the same data as Fiddler, but as I said, we can't reproduce it there.

I'd greatly appreciate any insights anyone may have...

Mark Richman
  • 28,948
  • 25
  • 99
  • 159
  • Do you have any information about which session data is being corrupted? Is it the shopping carts? Is it random? How do you know if you're not able to reproduce it? And, I'm sure you've heard this before, so I might sound like a broken record, you might want to do something about the need to store an entire object graph of your users' shopping carts in session and the storage of a user control instance as well. – Sumo Aug 24 '11 at 02:21
  • We know even though we can't reproduce it because we receive emails with stack traces when it happens, coincidental with customer calls. It's often random values, but one in particular seems to be dropping, which comes from a shared readonly property which wraps access to this session variable. It is quite random, but definitely happens under load because my customer service folks don't start receiving complaints until after some time has passed after the app pool starts. Yes, we know we need to get that obscene object graph and user control out of session, but that fix is some weeks away. :/ – Mark Richman Aug 24 '11 at 02:27
  • Can you post more details about this shared variable? – Sumo Aug 24 '11 at 02:32
  • The variable itself is not shared, just the property - it's VB.NET, so in C# it would be a static. There is no backing field, just the Session("foo") in the getter. – Mark Richman Aug 24 '11 at 10:27
  • 1
    As long as you're not doing [this](http://stackoverflow.com/questions/2804256/is-it-safe-to-access-asp-net-session-variables-through-static-properties-of-a-sta), you should be OK with that. Another question, what do you consider "corrupt?" Can you actually post the stack trace in the error emails you receive? – Sumo Aug 24 '11 at 10:35
  • And, I know you said you have your app pool recycling at 3 AM, but do you also have the "Regular Time Interval" setting set to 0 instead of the default 1740? – Sumo Aug 24 '11 at 10:39
  • We have no unexpected app pool restarts. – Mark Richman Aug 24 '11 at 18:05
  • Code sample: http://pastebin.com/nXr3wTy1 – Mark Richman Aug 24 '11 at 18:28
  • Your comment in the piece of code you pasted seems to suggest that the issue is not "Session Corruption" but rather that the session is expiring somehow, 'unexpectedly' since the object you are looking for is no longer there. I think Sumo's comment is on the right track and what I would do, at least to mitigate the problem, is to check for null, recreate the UserManager object if null and add it again to the Session. – Icarus Aug 24 '11 at 22:30

1 Answers1

1

Based on all of the information gathered here so far, I am going to answer with an educated guess as to what the problem is.

The session is likely expiring somehow.

The app pool recycling is not the only thing that can cause a session expiry. And, as Hanselman will tell you, this is a common occurence when you combine InProc session management and high volume.

Edit: Take a look at an older blog post for IIS6 that details how to determine the cause of lost sessions like this, especially ones that affect specific users, not just all sessions, as that can happen, too. The section of interest is just after the Application_End code snippet talking about Web Gardens. I've looked for a newer piece of information similar to this and all I could really find that talked about all of the issues was an answer to another question here on SO.

Community
  • 1
  • 1
Sumo
  • 4,066
  • 23
  • 40