4

I'm having an issue with a C# application which can, at random times, crash without warning or meaningful information. The same data can be put through the application, but it doesn't crash at the same time, it doesn't even crash sometimes.

I've used WinDbg to get some information about the exception. There is a NullReferenceException being thrown along with an Access Violation error.

Using SOSEX, I've got a stack using !mk. I've got a stack with the following at the top:

03:M 000000001aefd4f8 000007fef94cfa70 System.Runtime.Remoting.SoapServices.GetXmlTypeForInteropType(System.Type, System.String ByRef, System.String ByRef)(+0x0 IL,+0xd9e9a0 Native)
04:M 000000001aefd4f8 000007fef94cfa6f System.Runtime.Remoting.Metadata.RemotingTypeCachedData.get_QualifiedTypeName()(+0x0 IL,+0xd9ea5f Native)
05:U 000000001aefd500 000007fef991339b clr!CopyValueClassUnchecked+0xb8
06:U 000000001aefd550 000007fef9902491 clr!JIT_BoxFastMPIGT__PatchTLSLabel+0x51
[WARNING: Multiple managed methods at this address]07:M 000000001aefd580 000007fef87350ad System.Collections.Generic.List`1[[System.__Canon, mscorlib]].System.Collections.Generic.IEnumerable<T>.GetEnumerator()(+0x0 IL,+0xff47bced Native)
[WARNING: Multiple managed methods at this address]08:M 000000001aefd5d0 000007fef6d60e4f System.Linq.Enumerable.FirstOrDefault[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1<System.__Canon>, System.Func`2<System.__Canon,Boolean>)(+0x22 IL,+0x5f Native)
09:M 000000001aefd630 000007fe9a7885d8 Settings.VersionedSettings.GetSetting(System.DateTimeOffset, System.Guid)(+0x4e IL,+0x258 Native)

The lines starting with [WARNING: Multiple managed methods at this address] seem particularly worrying - what does this mean?

Looking bottom up through the stack, it all looks ok up until this point. Then a random method call to IEnumerable.FirstOrDefault throws it off. The VersionedSettings.GetSetting(...) method makes no such calls. This is why I'm left a little confused.

EDIT: I've run !verifyheap in WinDbg and the output was "No heap corruption detected.". My knowledge of WinDbg is limited so I'm not sure how accurate this command is but I thought it was worth adding.

Does anybody know what is going on?

pkunal7
  • 283
  • 3
  • 11
  • 1
    Does your application ever use `unsafe` code? It looks like you have some stack corruption. Also, I imagine "Multiple managed methods at this address" somehow means that more than one function got mapped to a single address; which might be why the wrong method is being called. If you have the source you should probably do a full rebuild against all your libraries and make sure that all the versions and whatnot are correct. – Alex Van Liew Aug 07 '15 at 17:39
  • With managed c++, the native functions with a native address that are marked as managed will also have a managed function address. Or something like that. This is called double thunking. But you shouldn't have that happen in C#. This is definitely some sort of stack corruption. Do you have a managed library as a dependency? Or p/invoke? Managed C++ projects get compiled with different CLR native support options... perhaps this was compiled for a similar native/managed CLR as opposed to a pure managed runtime? – Greg Aug 07 '15 at 19:21
  • 3
    The DAC provides a method enumerator that takes an instruction pointer as a parameter. I have never understood exactly what this means, though I can speculate. What the message means in sosex is that sosex enumerated the methods at the address and found more than one sharing the same code. I'll see if I can dig in a bit more this week and find out what causes this situation. The way sosex handles this situation is "last one wins". Perhaps it would be better to change it to "first one wins". If you can provide a repro, it would help me to diagnose. – Steve Johnson Aug 09 '15 at 20:18
  • Thanks for your responses. I've had a look and made sure that we have no unsafe code in the code base. We've performed numerous rebuilds and the version numbers all look correct. @ Greg what do you mean by managed libraries? We've got a lot of dependencies we developed in C#. I've not had a look at the compiled output for p/invoke but I'd be surprised if we had any. Unfortunately I can't really provide a repro; the problem happens randonly on the 18 servers (albeit regularly in that one will usually go down per run). Plus it only appears under high load. – pkunal7 Aug 10 '15 at 15:49

0 Answers0