I am assuming that you are having the session stored in "inProc" mode. In this mode, ASP.NET applications' session, cache etc are stored in the web server's RAM (via aspnet_wp.exe process). And .NET does not get to use all of it. There is a setting in machine.config which tells the threshold limit (by default 60%). Once this threshold is reached, IIS will recycle the worker process, and all the session information is lost.
Note that, if your server hosts more than one asp.net application, the 60% of memory is to be shared by all apps. So if the cumulative memory usage reaches the threshold the worker process still gets recycled.
Alternative to this, besides optimizing your application to use session sparingly,is to set the application to use session in out of process mode (using a stateserver or sqlserver to store session information).
Out of process mode can bring down your system performance.
Refer to this article for more information on Session State Management.