2

One certain problem is only reproducible on customer side. We cannot reproduce it locally despite all our attempts.

But I know that TaskMgr in Windows 2008 R2 has a possibility to create dump file for a process. So, my question: is it possible to create dump on customer site for a certain process of our software and then investigate that dump file locally?

We already made a new build of our software (we saved a build sandbox and *.PDB files for all binaries). Then we installed that on site and now we are waiting when customer report that problem happens again so we will create a dump file for hanging process and then try to investigate it.

My question has 2 parts:

  1. Would such method work at all?
  2. If yes - how exactly to do that?

At the moment I have some doubt if that would work. Because I have tried to create a proof-test on my local Win 2008 R2 VM. I build all with .PDB files, then I run our software in a mode when it makes a long pause in the middle and I clicked "Create Dump File" in TaskMgr exactly when it does a pause (its simple call of Sleep(30000)). Then I tried to load that dump file in WinDbg and check what I could find there. First thing which makes me pessimistic about such way is a wrong stack trace. In particular - I cannot see a full stack trace in WinDbg. It shows me only stack trace for wow64.dll and ntdll.dll modules, I cannot see stack trace for our code. In particular I see only this:

wow64cpu!TurboDispatchJumpAddressEnd+0x6c0
wow64cpu!TurboDispatchJumpAddressEnd+0x56b
wow64!Wow64SystemServiceEx+0x1ce
wow64!Wow64LdrpInitialize+0x42a
ntdll!RtlUniform+0x6e6
ntdll!RtlCreateTagHeap+0xa7
ntdll!LdrInitializeThunk+0xe

But when I try to attach process with debugger I see a full call-stack, like this:

ntdll.dll! 7754fd910
[Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]
ntdll.dll!7754fd9l0
KernelBase.dll! 76ae3bd50
KernelBase.dll! 76ae44a 5Q
ScrVm.DLL!Profiler::DoSleep(intmilliseconds=30000) Line 205
ScrVm.DLL!Script::VmToolKit::iMethod_Sleep(unsigned char & han
ScrVm.DLL!CComponent::Invoke(const _SU::basic_string<char,std
ScrVm.DLL!Script::VirtualMachine::do_Invoke(Script::VmCommand 
ScrVm.DLL!Script::VirtualMachine::InnerLoop( Line 4471
ScrVm.DLL!Script::VirtualMachine::Execute(unsigned long hFunc=
ScrVm.DLL!ScriptProcessor::Run(const _SU::basic_string<char,st
ScrVm.DLL!ScriptProcessor::ProcessDocumentO Line 285 + 0x40 by
ScrVm.DLL!DocumentProcessor::Process(BinaryDOM::Document * pDo
ScrVm.DLL!CFuncExecScript::ExecuteO Line 219
ScrVm.DLL!SrvManager::ExecuteO Line 586 +0xldbytes
ScrVm.DLL!SrvManager::Run(tag_TReqHdr "pRequestBuf=0x00187
ScrVm.DLL!SrvManager::HandleRequest(tag_TReqHdr " pRequest
ScrVm.DLL!SrvProcessRequest(tag_TReqHdr * pRequesffiuf=0x0
ScrVm.DLL!ProcessRequest(char "pRequesffiuf=0x001873b6, char "
ScrVm.DLL!ProcessRequest_DLL(char " achMsg=0x001873b6, char "a
siteExec212.exe!00409b2d0
siteExec212.exe!0040a4cfO

As you can see WinDbg seems only showing last 7 items in stack which are useless for me. Question - is it possible to discover the full stack trace from dump file created in TaskMgr in Windows 7/2008? Or at least - I need more items in stack trace, to see from what place in our code this call was made.

Note: compiler MS VisualStudio 2008, WinDbg 6.12 x64.

Matt Johnson-Pint
  • 230,703
  • 74
  • 448
  • 575
dmitry_bond
  • 378
  • 1
  • 3
  • 15
  • Sorry are you complaining about the MS call stacks being incomplete? try fixing the symbols before displaying the call stack `.symfix;.reload;kb` – EdChum Mar 16 '15 at 12:01
  • To answer your other question, yes you can open a dump file and debug it, you need to have matching pdbs to get sensible call stacks, MS do this all the time as you can imagine the millions of different versions of their dlls that exist on customer systems. – EdChum Mar 16 '15 at 12:02
  • Could you please clarify a bit? For example, I just run following commands: 1) x ScrVm!* I can see - it was listed all symbols in ScrMv.dll (that is part of our S/W). So, symbols are already loaded for it. 2) lm o I can see in a list of modules following record: 00000000`03050000 00000000`03253000 SCRVM C (private pdb symbols) D:\[...]\SCRVM.pdb 3) kv But it is still showing me a short stack trace - only 7 items. As you can see symbols for ScrVm.dll is loaded and recognized but it is still unable to show a full stack trace. :-( Any ideas? Hints?... – dmitry_bond Mar 16 '15 at 12:35
  • I'm talking about symbols for Windows dlls, for your dlls so long as the symbols match it should be able to resolve them, you can confirm this by running `!sym noisy` and checking if the debugger is happy with the pdbs, secondly I don't quite understand what you are referring to, it maybe that you are not seeing a full stack trace, this is because the default output is 20 lines, you can change this by adding a length after display call stack: `kb 40` this displays 64 calls, or changing it using `.kframes 0x40` again set to depth of 64 – EdChum Mar 16 '15 at 13:10
  • Hm... I just discovered interesting difference. When create dump file with TaskMgr -> Create Dump File then WinDbg cannot find stack trace for my code in such dump file! But when I create dump file with SysInterlans ProcDump.exe then WinDbg perfectly see full stack trace for it. Looks very strange... – dmitry_bond Mar 16 '15 at 14:39
  • Are you using the 32-bit task manager? – EdChum Mar 16 '15 at 14:40
  • No. I'm using default TaskMgr. I even not suspected that Windows have 32-bit version of TaskMgr... – dmitry_bond Mar 16 '15 at 15:28
  • possible duplicate of [How do I take a good crash dump for .NET?](http://stackoverflow.com/questions/24874027/how-do-i-take-a-good-crash-dump-for-net) – Thomas Weller Mar 17 '15 at 07:20

1 Answers1

6

Since your process is 32 bit you must use the 32 bit version of Task Manager to create the dump. Default installs have it in C:\Windows\SysWow64\taskmgr.exe

Also, make sure to use the 32 bit version of windbg.

Marc Sherman
  • 2,303
  • 14
  • 22
  • Waw! Fantastic. It works now. So, 32-bit version of TaskMgr was really the key to make a dump working. But as I can see 32-bit version of WinDbg is not required. I'm using 64-bit version of WinDbg and it works fine for 32-bit dumps. Thank you. *PS.* As I already mentioned using ProcDump.exe from SysInternals also works - it is creating correct dump format where I can see a full stack trace. – dmitry_bond Mar 16 '15 at 15:17
  • Yes, unlike Task Manager, Procdump always does the right thing with respect to bitness. BTW, marking this as the answer would be great since this is my birthday ;) – Marc Sherman Mar 16 '15 at 17:41
  • @dmitry_bond: The 64 bit version of WinDbg will only work as long as you don't need to load any extensions. E.g. if you have a .NET dump and want to load SOS, you need the 32 bit version of SOS which cannot be loaded in a 64 bit environment. – Thomas Weller Mar 17 '15 at 07:18