Random w3wp.exe crashes in .net 4

Question

I have a website which has been up and running absolutely fine for about 8 months now. It's running .net 4 intergrated mode.

Recently i've started to get some "random" w3wp.exe crashes, and after 5 of them, IIS rapid fail protection kicks in and I have to manually login to the server and start the application pool again.

Here's what the event viewer says for the Error:

Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2
Faulting module name: nlssorting.dll, version: 4.0.30319.296, time stamp: 0x504835c7
Exception code: 0xc00000fd
Fault offset: 0x000000000000191f
Faulting process id: 0x1998
Faulting application start time: 0x01ce6e6b9b80c949
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\nlssorting.dll
Report Id: d9cf3164-da5e-11e2-8cc5-f46d0440f6d5

Straight after the crashes, i get an "Information" log in the event viewer which at the bottom gives me the location of a .wer file.

This is what the .wer files contains:

Version=1
EventType=APPCRASH
EventTime=130162847687759734
ReportType=2
Consent=1
ReportIdentifier=d7c5e520-da5e-11e2-8cc5-f46d0440f6d5
IntegratorReportIdentifier=d7c5e51f-da5e-11e2-8cc5-f46d0440f6d5
Response.type=4
Sig[0].Name=Application Name
Sig[0].Value=w3wp.exe
Sig[1].Name=Application Version
Sig[1].Value=7.5.7601.17514
Sig[2].Name=Application Timestamp
Sig[2].Value=4ce7afa2
Sig[3].Name=Fault Module Name
Sig[3].Value=nlssorting.dll
Sig[4].Name=Fault Module Version
Sig[4].Value=4.0.30319.296
Sig[5].Name=Fault Module Timestamp
Sig[5].Value=504835c7
Sig[6].Name=Exception Code
Sig[6].Value=c00000fd
Sig[7].Name=Exception Offset
Sig[7].Value=000000000000197d
DynamicSig[1].Name=OS Version
DynamicSig[1].Value=6.1.7601.2.1.0.1296.17
DynamicSig[2].Name=Locale ID
DynamicSig[2].Value=2057
DynamicSig[22].Name=Additional Information 1
DynamicSig[22].Value=6141
DynamicSig[23].Name=Additional Information 2
DynamicSig[23].Value=61419d6dee6cf74b8ac2b00b4c3b3373
DynamicSig[24].Name=Additional Information 3
DynamicSig[24].Value=c19b
DynamicSig[25].Name=Additional Information 4
DynamicSig[25].Value=c19b8acf029a3088171b1f5f3dd9dc77
UI[2]=c:\windows\system32\inetsrv\w3wp.exe
UI[5]=Check online for a solution (recommended)
UI[6]=Check for a solution later (recommended)
UI[7]=Close
UI[8]=IIS Worker Process stopped working and was closed
UI[9]=A problem caused the application to stop working correctly. Windows will notify you if a solution is available.
UI[10]=&Close
LoadedModule[0]=c:\windows\system32\inetsrv\w3wp.exe
LoadedModule[1]=C:\Windows\SYSTEM32\ntdll.dll
LoadedModule[2]=C:\Windows\system32\kernel32.dll
LoadedModule[3]=C:\Windows\system32\KERNELBASE.dll
LoadedModule[4]=C:\Windows\system32\ADVAPI32.dll
LoadedModule[5]=C:\Windows\system32\msvcrt.dll
LoadedModule[6]=C:\Windows\SYSTEM32\sechost.dll
LoadedModule[7]=C:\Windows\system32\RPCRT4.dll
LoadedModule[8]=C:\Windows\system32\pcwum.DLL
LoadedModule[9]=C:\Windows\system32\USER32.dll
LoadedModule[10]=C:\Windows\system32\GDI32.dll
LoadedModule[11]=C:\Windows\system32\LPK.dll
LoadedModule[12]=C:\Windows\system32\USP10.dll
LoadedModule[13]=C:\Windows\system32\ole32.dll
LoadedModule[14]=c:\windows\system32\inetsrv\IISUTIL.dll
LoadedModule[15]=C:\Windows\system32\IMM32.DLL
LoadedModule[16]=C:\Windows\system32\MSCTF.dll
LoadedModule[17]=C:\Windows\system32\CRYPTBASE.dll
LoadedModule[18]=C:\Windows\system32\ntmarta.dll
LoadedModule[19]=C:\Windows\system32\WLDAP32.dll
LoadedModule[20]=c:\windows\system32\inetsrv\w3wphost.dll
LoadedModule[21]=C:\Windows\system32\OLEAUT32.dll
LoadedModule[22]=c:\windows\system32\inetsrv\nativerd.dll
LoadedModule[23]=C:\Windows\system32\CRYPT32.dll
LoadedModule[24]=C:\Windows\system32\MSASN1.dll
LoadedModule[25]=C:\Windows\system32\XmlLite.dll
LoadedModule[26]=C:\Windows\system32\ktmw32.dll
LoadedModule[27]=c:\windows\system32\inetsrv\IISRES.DLL
LoadedModule[28]=C:\Windows\system32\CRYPTSP.dll
LoadedModule[29]=C:\Windows\system32\rsaenh.dll
LoadedModule[30]=C:\Windows\system32\mscoree.dll
LoadedModule[31]=C:\Windows\system32\CLBCatQ.DLL
LoadedModule[32]=C:\Windows\system32\mlang.dll
LoadedModule[33]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\webengine4.dll
LoadedModule[34]=C:\Windows\system32\MSVCR100_CLR0400.dll
LoadedModule[35]=C:\Windows\system32\USERENV.dll
LoadedModule[36]=C:\Windows\system32\profapi.dll
LoadedModule[37]=C:\Windows\system32\PSAPI.DLL
LoadedModule[38]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\mscoreei.dll
LoadedModule[39]=C:\Windows\system32\SHLWAPI.dll
LoadedModule[40]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
LoadedModule[41]=C:\Windows\system32\inetsrv\iiscore.dll
LoadedModule[42]=c:\windows\system32\inetsrv\W3TP.dll
LoadedModule[43]=c:\windows\system32\inetsrv\w3dt.dll
LoadedModule[44]=C:\Windows\system32\HTTPAPI.dll
LoadedModule[45]=C:\Windows\system32\slc.dll
LoadedModule[46]=C:\Windows\system32\WS2_32.dll
LoadedModule[47]=C:\Windows\system32\NSI.dll
LoadedModule[48]=C:\Windows\system32\Normaliz.dll
LoadedModule[49]=C:\Windows\system32\faultrep.dll
LoadedModule[50]=C:\Windows\system32\Secur32.dll
LoadedModule[51]=C:\Windows\system32\SSPICLI.DLL
LoadedModule[52]=C:\Windows\system32\NLAapi.dll
LoadedModule[53]=C:\Windows\system32\napinsp.dll
LoadedModule[54]=C:\Windows\System32\mswsock.dll
LoadedModule[55]=C:\Windows\system32\DNSAPI.dll
LoadedModule[56]=C:\Windows\System32\winrnr.dll
LoadedModule[57]=C:\Windows\System32\wshtcpip.dll
LoadedModule[58]=C:\Windows\System32\wship6.dll
LoadedModule[59]=C:\Windows\system32\IPHLPAPI.DLL
LoadedModule[60]=C:\Windows\system32\WINNSI.DLL
LoadedModule[61]=C:\Windows\system32\rasadhlp.dll
LoadedModule[62]=C:\Windows\System32\fwpuclnt.dll
LoadedModule[63]=C:\Windows\System32\inetsrv\cachuri.dll
LoadedModule[64]=C:\Windows\System32\inetsrv\cachfile.dll
LoadedModule[65]=C:\Windows\System32\inetsrv\cachtokn.dll
LoadedModule[66]=C:\Windows\System32\inetsrv\cachhttp.dll
LoadedModule[67]=C:\Windows\System32\inetsrv\compdyn.dll
LoadedModule[68]=C:\Windows\System32\inetsrv\compstat.dll
LoadedModule[69]=C:\Windows\System32\inetsrv\defdoc.dll
LoadedModule[70]=C:\Windows\System32\inetsrv\protsup.dll
LoadedModule[71]=C:\Windows\System32\inetsrv\redirect.dll
LoadedModule[72]=C:\Windows\System32\inetsrv\static.dll
LoadedModule[73]=C:\Windows\System32\inetsrv\authanon.dll
LoadedModule[74]=C:\Windows\System32\inetsrv\authbas.dll
LoadedModule[75]=C:\Windows\System32\inetsrv\authsspi.dll
LoadedModule[76]=C:\Windows\system32\NETAPI32.dll
LoadedModule[77]=C:\Windows\system32\netutils.dll
LoadedModule[78]=C:\Windows\system32\srvcli.dll
LoadedModule[79]=C:\Windows\system32\wkscli.dll
LoadedModule[80]=C:\Windows\System32\inetsrv\iprestr.dll
LoadedModule[81]=C:\Windows\System32\inetsrv\modrqflt.dll
LoadedModule[82]=C:\Windows\System32\inetsrv\logcust.dll
LoadedModule[83]=C:\Windows\System32\inetsrv\custerr.dll
LoadedModule[84]=C:\Windows\System32\inetsrv\loghttp.dll
LoadedModule[85]=C:\Windows\System32\inetsrv\isapi.dll
LoadedModule[86]=C:\Windows\System32\inetsrv\filter.dll
LoadedModule[87]=C:\Windows\System32\inetsrv\validcfg.dll
LoadedModule[88]=c:\Windows\Microsoft.NET\Framework64\v4.0.30319\aspnet_filter.dll
LoadedModule[89]=C:\Windows\system32\inetsrv\wbhst_pm.dll
LoadedModule[90]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\webengine.dll
LoadedModule[91]=C:\Windows\assembly\NativeImages_v4.0.30319_64\mscorlib\4f52500ab48877b85e71430f4f46670f\mscorlib.ni.dll
LoadedModule[92]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\nlssorting.dll
LoadedModule[93]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System\a91f32875cb3ba779f1b3ceff1690251\System.ni.dll
LoadedModule[94]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Core\0a8d99339ffe6b25debb8f8201c27664\System.Core.ni.dll
LoadedModule[95]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Web\5b905bd7b71f9fd6bea2d05cc1ae85f8\System.Web.ni.dll
LoadedModule[96]=C:\Windows\system32\sxs.dll
LoadedModule[97]=C:\Windows\system32\RpcRtRemote.dll
LoadedModule[98]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Configuration\fa65f89fd682c459fc5e7bcbd0418317\System.Configuration.ni.dll
LoadedModule[99]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Xml\f4afb233f160b8e55aad4660e45b374c\System.Xml.ni.dll
LoadedModule[100]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clrjit.dll
LoadedModule[101]=C:\Windows\assembly\NativeImages_v4.0.30319_64\Microsoft.Build.Uti#\14e16d61fae3cd1d9a1fa79b789f8438\Microsoft.Build.Utilities.v4.0.ni.dll
LoadedModule[102]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Runtime.Cach#\8fdbe304abab0631b8a4310b35f3d93a\System.Runtime.Caching.ni.dll
LoadedModule[103]=C:\Windows\system32\shfolder.dll
LoadedModule[104]=C:\Windows\system32\SHELL32.dll
LoadedModule[105]=C:\Windows\assembly\NativeImages_v4.0.30319_64\Microsoft.JScript\85204dde340780329b569b025e249c23\Microsoft.JScript.ni.dll
LoadedModule[106]=C:\Windows\system32\version.dll
LoadedModule[107]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\587f6661\a99d8ff8\App_Code.cgixlnxh.dll
LoadedModule[108]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Data.Linq\feaa494ad67542d2060b31b9eeb6458b\System.Data.Linq.ni.dll
LoadedModule[109]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Data\b928128fca867546a858a1a39240d85c\System.Data.ni.dll
LoadedModule[110]=C:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
LoadedModule[111]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\587f6661\a99d8ff8\assembly\dl3\595a888a\f26c0653_7f81cd01\HtmlAgilityPack.dll
LoadedModule[112]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Drawing\5ae853f556290da9399b15b3619f7e15\System.Drawing.ni.dll
LoadedModule[113]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\587f6661\a99d8ff8\assembly\dl3\85ba5013\f0c8f388_706bce01\TweetSharp.dll
LoadedModule[114]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Web.Extensio#\0180a2d993d2a9699cf07f7163524fff\System.Web.Extensions.ni.dll
LoadedModule[115]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Transactions\7b2099a1386e38ff198a51939304ce6e\System.Transactions.ni.dll
LoadedModule[116]=C:\Windows\Microsoft.Net\assembly\GAC_64\System.Transactions\v4.0_4.0.0.0__b77a5c561934e089\System.Transactions.dll
LoadedModule[117]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\587f6661\a99d8ff8\App_global.asax.yxdky-qn.dll
LoadedModule[118]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.ServiceModel#\7a5a5ff4a0b3bb4ba4bcc13166918e36\System.ServiceModel.Activation.ni.dll
LoadedModule[119]=C:\Windows\system32\bcrypt.dll
LoadedModule[120]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Runtime.Dura#\799274e49455d0fe4ca563f42143bef2\System.Runtime.DurableInstancing.ni.dll
LoadedModule[121]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Numerics\a66416296451fe6d2d8a5506ca41b23d\System.Numerics.ni.dll
LoadedModule[122]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.ServiceModel\15834d73d2846fc01ed54488ccfff5c8\System.ServiceModel.ni.dll
LoadedModule[123]=C:\Windows\assembly\NativeImages_v4.0.30319_64\SMDiagnostics\31f93b6be386908ff2727bcd825de0ca\SMDiagnostics.ni.dll
LoadedModule[124]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Xaml.Hosting\cf8401f4952deb5303e0d7fd459ce530\System.Xaml.Hosting.ni.dll
LoadedModule[125]=C:\Windows\system32\inetsrv\gzip.dll
LoadedModule[126]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\587f6661\a99d8ff8\assembly\dl3\3d63b311\fe7c9b8a_706bce01\Hammock.ClientProfile.dll
LoadedModule[127]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\587f6661\a99d8ff8\assembly\dl3\6a128bd2\c184e08a_706bce01\Newtonsoft.Json.dll
LoadedModule[128]=C:\Windows\system32\rasapi32.dll
LoadedModule[129]=C:\Windows\system32\rasman.dll
LoadedModule[130]=C:\Windows\system32\rtutils.dll
LoadedModule[131]=C:\Windows\system32\winhttp.dll
LoadedModule[132]=C:\Windows\system32\webio.dll
LoadedModule[133]=C:\Windows\system32\credssp.dll
LoadedModule[134]=C:\Windows\system32\dhcpcsvc6.DLL
LoadedModule[135]=C:\Windows\system32\dhcpcsvc.DLL
LoadedModule[136]=C:\Windows\system32\security.dll
LoadedModule[137]=C:\Windows\system32\schannel.DLL
LoadedModule[138]=C:\Windows\system32\ncrypt.dll
LoadedModule[139]=C:\Windows\system32\bcryptprimitives.dll
LoadedModule[140]=C:\Windows\system32\GPAPI.dll
FriendlyEventName=Stopped working
ConsentKey=APPCRASH
AppName=IIS Worker Process
AppPath=c:\windows\system32\inetsrv\w3wp.exe

That nlssorting.dll seems to crop up a lot but I can't seem to find anything online related. The only thing I can find which matches my error is here, but that doesn't really help me.

I'm completely stumped as to where to go from here to fix this. Here's what I've tried:

Loading up IIS log files and trying every request from about 30 minutes before a crash, and none of the pages cause any errors.
Searching my code for any recursion which might cause a stackoverflow, but there isn't any
trawling online for ANYTHING that might help

Has anyone else ever had any problems with nlssorting.dll ? Can i get some more information from the .wer file that might help me pin point where this is happening?

Thanks in advance for any help!

UPDATE

I was using a 3rd party DLL, which was causing a stack overflow exception (0xc00000fd)

After more investigation, it was only happening after a certain chain of events happened - hence the "random" in the title. Removing the DLL fixed the problem.

Look at that: http://stackoverflow.com/questions/3044752/how-do-i-crash-the-app-pool/3045137#3045137 — Aristos, Jun 24 '13 at 16:24
@Aristos I've looked through my code for any recursion and can't find any. the site has also been running absolutely fine for about 8 months, and these crashes have only just started happening. it's not a big site. do you have any knowledge on extracting more information from the .wer file? — royse41, Jun 25 '13 at 08:32
@JennyO'Reilly i did - it was a stack overflow exception, caused by a 3rd party dll which only happened in very, very particular circumstances (certain events had to happen in a certain order to cause the crash) - hence the "random crashes" in the title. i'll try and post an answer in a little while. — royse41, Jul 01 '14 at 11:05
I followed the answer from "Thomas C. G. de Vilhena" here: http://stackoverflow.com/questions/5053708/how-to-debug-w3wp-exe-process-was-terminated-due-to-a-stack-overflow-works-on and found the cause. It was an endless recursion occuring in a very special case. — Krisztián Balla, Sep 05 '14 at 08:49
@JennyO'Reilly same - the 0xc00000fd exception code was a give away that it was a stack overflow (which is caused by recursion 99% of the time in my experience), just had to find where. glad you got it fixed! — royse41, Sep 05 '14 at 09:14

score 2 · Answer 1 · answered Jun 30 '14 at 07:01

We had the same problem with one of our sites. Using SVN we tracked it down to a method that was scaning for images within a folder.

I modified the code as follows:

Checking array length of scan results to be > 0 instead of == 1
Adding CultureInfo.InvariantCulture to all Int32.ToString() calls

After this we no longer experienced the error. The exact reason is still unknown.

I believe that none of the above points should make a difference in our environment. I believe that the problem could have been, people modifying image files and folders while the image scanning method was called.

I hope this helps somebody.

score 2 · Answer 2 · answered Jan 22 '15 at 21:44

For anyone who's curious, this is a PITA to debug. Here are three reasons rumored for this to happen:

(1) Stack overflows, as in the original post.

(2) Too much CPU / memory usage, which becomes obvious and rapid fail protection closes the process.

(3) Unable to respond to pings / requests due to application hogging resources, but in a way that rapid fail protection deems appropriate to end the process, not explicitly because of either (1) or (2).

Our solution was to add manual log tracing in the production environment until we eventually found recursion which was leading the application to be stopped by reliability services (for inability to respond to pings, or process randomly crashing) rather than throwing an in-application exception.

score 1 · Answer 3 · answered Jun 03 '15 at 15:21

I had the issue where w3p would throw an unhandled error as soon I spun up the site\api URI from the web browser, then it would crash.

I was able to pinpoint what part of my code was causing it in my case it was in the Owin Startup class and I was reading some configuration records from a database but prior to that it gets the connection string from a configuration file outside of the web app directory.

I checked the ownership of the folder it showed my account but apparently the subfolders were not owned by me so I set ownership to me again, clicked OK to allow permissions to traverse the objects and voila bye bye w3p error and the API loaded.

So in my case it was an access denied error on the folder \ file that contained the connection string.

Random w3wp.exe crashes in .net 4

3 Answers3