Why can't I investigate a dumpfile, using Windbg "!DumpHeap -stat" command?

Question

First off, I'm a newbie on the different .NET platforms.

I'm trying to investigate a dump of a Managed .Net application (I don't know the version), using Windbg.

In order to do this, I'd like to launch !DumpHeap -stat command.
At first, this does not work because sos is not loaded:

0:000> !DumpHeap -stat
No export DumpHeap found
0:000> .load sos

However, there seems to be another problem:

0:000> !DumpHeap -stat
The garbage collector data structures are not in a valid state for traversal.
It is either in the "plan phase," where objects are being moved around, or
we are at the initialization or shutdown of the gc heap. Commands related to 
displaying, finding or traversing objects as well as gc heap segments may not 
work properly. !dumpheap and !verifyheap may incorrectly complain of heap 
consistency errors.
Object <exec cmd="!ListNearObj /d b331dbb0">b331dbb0</exec> has an invalid method table.

On the internet, I've found some posts, mentioning that this issue might be caused by a version mismatch, which looks to be confirmed by the .chain result:

0:000> .chain
Extension DLL search Path:
    => Hereby my entire %PATH% environment variable
Extension DLL chain:
    C:\ProgramData\dbg\sym\SOS_x86_x86_4.7.2563.00.dll\5A334E146eb000\SOS_x86_x86_4.7.2563.00.dll: image 4.7.2563.0, API 1.0.0, built Fri Dec 15 05:16:06 2017
        [path: C:\ProgramData\dbg\sym\SOS_x86_x86_4.7.2563.00.dll\5A334E146eb000\SOS_x86_x86_4.7.2563.00.dll]
    sos: image 4.6.1087.0, API 1.0.0, built Wed Nov 30 05:49:55 2016
        [path: C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\sos.dll]

As you can see, there indeed seems to be a version mismatch:

C:\...\SOS (ALL CAPS) seems to be of version 4.7.2653
sos (small letters)   seems to be of version 4.6.1087

Let's solve this:

0:000> .unload C:\ProgramData\dbg\sym\SOS_x86_x86_4.7.2563.00.dll\5A334E146eb000\SOS_x86_x86_4.7.2563.00.dll
0:000> .load C:\ProgramData\dbg\sym\SOS_x86_x86_4.6.1087.00.dll\583E5B8E6b1000\SOS_x86_x86_4.6.1087.00.dll
// I found this file, somewhere on my PC, I just hope it's correct :-)

Does this solve the issue? It seems not:

0:000> !DumpHeap -stat
The garbage collector data structures are not in a valid state for traversal.
It is either in the "plan phase," where objects are being moved around, or
we are at the initialization or shutdown of the gc heap. Commands related to 
displaying, finding or traversing objects as well as gc heap segments may not 
work properly. !dumpheap and !verifyheap may incorrectly complain of heap 
consistency errors.
Object <exec cmd="!ListNearObj /d b331dbb0">b331dbb0</exec> has an invalid method table.

Ok. So still no solution. Could it be that there are other versions which are wrong?

0:000> .cordll
CLR DLL status: Loaded DLL C:\ProgramData\dbg\sym\mscordacwks_x86_x86_4.7.2563.00.dll\5A334E146eb000\mscordacwks_x86_x86_4.7.2563.00.dll

Indeed, also CLR seems to be referring to that wrong version. Let's solve that one:

0:000> .cordll -u
CLR DLL status: No load attempts

(First unload the current one, then load a new one)

0:000> .cordll -lp C:\ProgramData\dbg\sym\mscordacwks_x86_x86_4.6.1087.00.dll\583E5B8E6b1000\mscordacwks_x86_x86_4.6.1087.00.dll
// Again a file I found somewhere on my PC, but it seems not to be working:
CLRDLL: Consider using ".cordll -lp <path>" command to specify .NET runtime directory.
CLR DLL status: ERROR: Unable to load DLL C:\ProgramData\dbg\sym\mscordacwks_x86_x86_4.6.1087.00.dll\583E5B8E6b1000\mscordacwks_x86_x86_4.6.1087.00.dll\mscordacwks_x86_x86_4.7.2563.00.dll, Win32 error 0n87

And now I'm out of options : while Googling Win32 error 0n87 I find information about wrong parameters, dump investigations going wrong, but not a way to get this CLR DLL loaded.

Can anybody help me here (maybe we can start by determining which version I need to chose, I just took a random pick :-) )?
Thanks in advance

Answer 1

0:000> .load sos

That command will load the SOS extension from the WinDbg plugin directory. Usually that's a version for .NET 1.x, which, at that time did not come along with the .NET framework installation.

Newer versions of the .NET framework come with a suitable version of the SOS DLL. It's installed in the .NET framework directory, not in the WinDbg directory.

Later we see that above command loaded version 4.6.1087.0. I can't really explain this, except that the file was exchanged manually.

The garbage collector data structures are not in a valid state for traversal.

If the Internet says this could be a version mismatch, that might be true - I can't judge about it.

Another option is, that the statement is simply true and there is currently a garbage collection in progress and thus the heap is inconsistent.

A third option is related to native code (C++ or similar) which has written to .NET memory and destroyed some heap information.

[...] which looks to be confirmed by the .chain result:

So, we see 2 versions of SOS loaded. One was obviously loaded already (e.g. by !analyze, did you run that before?), the other was loaded by the .load sos command.

Let's solve this: [...]

You're trying to solve the issue using .unload and .load. However, this will only unload one SOS DLL and then load a second one again. If you wanted a single clear version of SOS, you should have .unloaded 2 times (all of the SOS DLLs) and then loaded the correct version.

Does this solve the issue? It seems not

As said before, GC could be running right now. You might be interested in a related question: How to capture a process memory dump of a .NET process when .NET is not in the middle of a garbage collection (GC)

By looking at the native call stacks (~*k), it should be possible to figure out whether or not .NET is currently garbage collecting.

Could it be that there are other versions which are wrong?

Indeed, also CLR seems to be referring to that wrong version.

Well, you never determined (or at least did not tell us) the version of .NET which is loaded into the process. Doing a lm vm clr should give you the version which is loaded (an exception are long running processes and an update was installed in the meanwhile, then the version information might be wrong, because it's read from disk at the time capturing the crash dump).

The usual command to load the appropriate SOS is .loadby sos clr, which tells WinDbg to load the SOS from the exact same place where clr.dll is located. This will work if the crash dump is from your machine. It may be more complex if you got the crash dump from someone else.

.cordll -lp C:\ProgramData\dbg\sym\mscordacwks_x86_x86_4.6.1087.00.dll\583E5B8E6b1000\mscordacwks_x86_x86_4.6.1087.00.dll

-lp stands for load from path but you specified the file name. If you look at the error message, it says mscordacwks_x86_x86_4.6.1087.00.dll twice.

However, I doubt that correcting mscordacwks will help - it never complained that it was wrong. An error message would look like this:

CLRDLL: [...]\mscordacwks.dll:<loaded version> doesn't match desired version <expected version>

Summary

1. If you want only one version of SOS, unload all other versions
2. If the heap is in an invalid state, it might exactly be that and not a version mismatch. Another option is a heap corruption by some native code writing in .NET heap.
3. Capture a crash dump whan garbage collection is not in progress
4. Give us more information about the CLR version, native call stacks, etc. so that we can better help you (do some research before you post new questions)