Crash dump retrieved from a different OS points to message loop. How to get to the fault location from there? (TeamViewer crashes my process.)

Question

One of the customers of our software complained that it was crashing on his Windows XP machine. I asked him to download ProcDump and run our process through it as such:

procdump -e -ma -w myproc.exe

For which the user emailed me collected crash dump .dmp file.

I then placed that .dmp file into the same folder as the original .exe image file for the process, as well as the .pdb file, and ran the .dmp file through Visual Studio 2010:

Then when I clicked "Debug with native only" VS showed the source code for the MFC's message loop function with the pointer location after DispatchMessage call (see green arrow):

The call stack didn't help much either:

and when I jumped to the reported fault address, I got nothing:

The app was compiled in VS 2008 as a dialog-based C++/MFC application.

So I'm checking the size of the .dmp file and it is 26.6 MB. There's data in it. The question I have is how can I get more useful info out of it to help pinpoint the location of the fault in the source code?

PS. As an afterthought, I see that the crash happened on Windows XP while I'm running this crash dump .dmp file on Windows 8.1. Is that the reason why I'm getting garbage in the stack trace?

EDIT: Here's what I get in disassembler if I try to see parameters in MSG struct that DispatchMessage was called with:

Also when I try to watch values in pState I get symbol not found.

EDIT 2: Concerning modules and the module loaded into the fault EIP address. The memory slot for 0x1963AB50 appears not be occupied by any module:

EDIT 3: Sorry, it's getting too long. I eventually opened up the crash dump in WinDbg and did !analyze -v command on it. Here's the output. It seems like the DLL that was missing/unloaded is tv_w32.dll:

Microsoft (R) Windows Debugger Version 10.0.10586.567 X86
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\Users\User\Desktop\Myproc\Release\Myproc.exe_160708_141237.dmp]
User Mini Dump File with Full Memory: Only application data is available

Comment: '
*** procdump -e -ma -w Myproc.exe
*** Unhandled exception: C0000005.ACCESS_VIOLATION'

************* Symbol Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       srv*c:\MyServerSymbols*https://msdl.microsoft.com/download/symbols
Symbol search path is: srv*c:\MyServerSymbols*https://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows XP Version 2600 (Service Pack 3) MP (2 procs) Free x86 compatible
Product: WinNt, suite: SingleUserTS
Machine Name:
Debug session time: Fri Jul  8 05:12:39.000 2016 (UTC - 7:00)
System Uptime: 9 days 6:05:50.960
Process Uptime: 3 days 5:08:52.000
.....................................
Loading unloaded module list
..
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(ba4.4a4): Access violation - code c0000005 (first/second chance not available)
eax=00000001 ebx=00000002 ecx=7c92741c edx=7c98e174 esi=1963ab50 edi=1963ab50
eip=19608536 esp=0012ee3c ebp=0012ee5c iopl=0         nv up ei pl nz na po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000202
<Unloaded_tv_w32.dll>+0x8536:
19608536 ??              ???
0:000> .bugcheck
               ^ Syntax error in '.bugcheck'
0:000> !analyze -v
*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************


DUMP_CLASS: 2

DUMP_QUALIFIER: 400

CONTEXT:  (.ecxr)
eax=00000001 ebx=00000002 ecx=7c92741c edx=7c98e174 esi=1963ab50 edi=1963ab50
eip=19608536 esp=0012ee3c ebp=0012ee5c iopl=0         nv up ei pl nz na po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000202
<Unloaded_tv_w32.dll>+0x8536:
19608536 ??              ???
Resetting default scope

FAULTING_IP: 
tv_w32!unloaded+8536
19608536 ??              ???

EXCEPTION_RECORD:  (.exr -1)
ExceptionAddress: 19608536 (<Unloaded_tv_w32.dll>+0x00008536)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 00000000
   Parameter[1]: 19608536
Attempt to read from address 19608536

DEFAULT_BUCKET_ID:  BAD_INSTRUCTION_PTR

PROCESS_NAME:  Myproc.exe

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.

EXCEPTION_CODE_STR:  c0000005

EXCEPTION_PARAMETER1:  00000000

EXCEPTION_PARAMETER2:  19608536

READ_ADDRESS:  19608536 

FOLLOWUP_IP: 
tv_w32!unloaded+8536
19608536 ??              ???

FAILED_INSTRUCTION_ADDRESS: 
tv_w32!unloaded+8536
19608536 ??              ???

WATSON_BKT_PROCSTAMP:  576c0d97

WATSON_BKT_PROCVER:  1.5.16.0

PROCESS_VER_PRODUCT:  Myproc Name

WATSON_BKT_MODULE:  unknown

WATSON_BKT_MODVER:  0.0.0.0

WATSON_BKT_MODOFFSET:  19608536

MODLIST_WITH_TSCHKSUM_HASH:  8b7920bcb3af2eb1b97366fd0663a3798ab285ea

MODLIST_SHA1_HASH:  b5acbaed45de37ad3c73ee4478c8f6678e94eaa6

NTGLOBALFLAG:  0

APPLICATION_VERIFIER_FLAGS:  0

PRODUCT_TYPE:  1

SUITE_MASK:  272

DUMP_FLAGS:  3

DUMP_TYPE:  0

APP:  Myproc.exe

ANALYSIS_SESSION_HOST:  DESKTOP-ABC

ANALYSIS_SESSION_TIME:  07-08-2016 22:18:58.0444

ANALYSIS_VERSION: 10.0.10586.567 x86fre

IP_MODULE_UNLOADED: 
tv_w32!unloaded+8536
19608536 ??              ???

THREAD_ATTRIBUTES: 
OS_LOCALE:  DEU

PROBLEM_CLASSES: 



BAD_INSTRUCTION_PTR
    Tid    [0x4a4]
    Frame  [0x00]: tv_w32!unloaded



INVALID_POINTER_READ
    Tid    [0x4a4]
    Frame  [0x00]: tv_w32!unloaded


BUGCHECK_STR:  BAD_INSTRUCTION_PTR_INVALID_POINTER_READ

LAST_CONTROL_TRANSFER:  from 00000000 to 19608536

STACK_TEXT:  
0012ee38 00000000 19608648 ffff071b 00000000 <Unloaded_tv_w32.dll>+0x8536


THREAD_SHA1_HASH_MOD_FUNC:  0959aaead905d5402017a9de12facac3929f71c3

THREAD_SHA1_HASH_MOD_FUNC_OFFSET:  52f8c4ddebdd2e855f90e79f96f6fc29a470ad1d

THREAD_SHA1_HASH_MOD:  2e69fc2d39893b6d7d9991dcc4967479dbafb5c5

SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  tv_w32!unloaded+8536

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: tv_w32

IMAGE_NAME:  tv_w32.dll

DEBUG_FLR_IMAGE_TIMESTAMP:  573435d6

STACK_COMMAND:  .ecxr ; kb

FAILURE_BUCKET_ID:  BAD_INSTRUCTION_PTR_c0000005_tv_w32.dll!unloaded

BUCKET_ID:  BAD_INSTRUCTION_PTR_INVALID_POINTER_READ_UNLOADED_IP_tv_w32!unloaded+8536

PRIMARY_PROBLEM_CLASS:  BAD_INSTRUCTION_PTR_INVALID_POINTER_READ_UNLOADED_IP_tv_w32!unloaded+8536

BUCKET_ID_OFFSET:  8536

BUCKET_ID_MODULE_STR:  tv_w32

BUCKET_ID_MODTIMEDATESTAMP:  573435d6

BUCKET_ID_MODCHECKSUM:  45ff9

BUCKET_ID_MODVER_STR:  0.0.0.0

BUCKET_ID_PREFIX_STR:  BAD_INSTRUCTION_PTR_INVALID_POINTER_READ_UNLOADED_IP_

FAILURE_PROBLEM_CLASS:  BAD_INSTRUCTION_PTR

FAILURE_EXCEPTION_CODE:  c0000005

FAILURE_IMAGE_NAME:  tv_w32.dll

FAILURE_FUNCTION_NAME:  unloaded

BUCKET_ID_FUNCTION_STR:  unloaded

FAILURE_SYMBOL_NAME:  tv_w32.dll!unloaded

TARGET_TIME:  2016-07-08T12:12:39.000Z

OSBUILD:  2600

OSSERVICEPACK:  6532

SERVICEPACK_NUMBER: 6

OS_REVISION: 0

OSPLATFORM_TYPE:  x86

OSNAME:  Windows XP

OSEDITION:  Windows XP WinNt (Service Pack 3) SingleUserTS

USER_LCID:  0

OSBUILD_TIMESTAMP:  2014-03-12 03:48:40

ANALYSIS_SESSION_ELAPSED_TIME: e0a

ANALYSIS_SOURCE:  UM

FAILURE_ID_HASH_STRING:  um:bad_instruction_ptr_c0000005_tv_w32.dll!unloaded

FAILURE_ID_HASH:  {7c7c5a5b-462a-0301-1674-660f1336c576}

Followup:     MachineOwner
---------

In the `AfxInternalPumpMessage` function, can you see what the last message handled was? Do you have a handler for that message? What does that handler do? Does the handler function contain any local arrays, and you write out of bounds of it? Because these kind of errors, with corrupted stack, is mostly because of out-of-bounds writing of local variables in one or more of your own functions. If the code isn't to big it might be a good idea to have someone review it. Or at the very least try to add some debug logging and tracing. — Some programmer dude, Jul 09 '16 at 03:03
@JoachimPileborg: Thanks. I updated my original question. (See edit.) — c00000fd, Jul 09 '16 at 03:14
You're getting the source for the message handler because that's the first frame on the stack that the debugger found that has source. The contents of the EDI register are whatever they are at the site of the crash, which is at 0x1963AB50. Look in the `Modules` tab to see what DLL is loaded at that address. Does your release build include the debug symbols? — 1201ProgramAlarm, Jul 09 '16 at 04:46
@1201ProgramAlarm: I updated my post. (See edit 2.) To answer your questions: yes, as you see symbols are loaded into my exe. As for the module at that fault address, then it appears to be none. Although I'm not sure what is that `xpsp2res.dll` that is missing? This is not something that my process has a dependency on. — c00000fd, Jul 09 '16 at 04:57
`xpsp2res.dll` has resources (?) that were added to Windows as part of XP Service Pack 2, and is automatically loaded. It's possible that your program got lost (from a corrupted stack or a clobbered function pointer) and wandered into someplace where the is no code (and the stack dump is garbage), or there was code there that has since been unloaded (during exception handling; or it could be a virtual function in a DLL that was manually unloaded before deleting the object that uses it; or managed or other run time generated code). — 1201ProgramAlarm, Jul 09 '16 at 05:06
@1201ProgramAlarm: Thanks. That's what I was afraid of. I'm curious at this point. Is it worth trying to install `WinDbg` to analyze this crash dump through it? — c00000fd, Jul 09 '16 at 05:11
I don't know. I'm not really familiar with `WinDbg`. I rarely have any success debugging from a dump file beyond the basic "here's where it crashed and these are the CPU registers" (then figuring out what went wrong to end up with those values). It could also be a 3rd party DLL that loads into every program. Without at least the DLL name and the access violation address (to see if it is bad/missing code, or a bad pointer) you can't do much with it. — 1201ProgramAlarm, Jul 09 '16 at 05:24
@1201ProgramAlarm: I ran it through `WinDbg`, see edit 3 above. (It's Microsoft's debugging tool that they use to analyze crash dumps that are submitted to them) and it gave me the module name `tv_w32.dll` that was unloaded where the crash happened. It's not on my dependency list either. Hmm? — c00000fd, Jul 09 '16 at 05:31
@1201ProgramAlarm: That's what I've been doing.... is it Team Viewer? How would it be associated with my process? It has nothing to do with it. — c00000fd, Jul 09 '16 at 05:36
I don't know what it is. It could be set in the registry to load automatically into every process, or some other program could inject it into running processes. — 1201ProgramAlarm, Jul 09 '16 at 05:38
@1201ProgramAlarm: I think [I got it](http://stackoverflow.com/questions/11996801/how-is-teamviewers-quickconnect-button-accomplished). It's f'ing TeamViewer that's crashing my process! Or something called QuickConnect in it. It installs system wide hooks to [change the title bar](https://www.teamviewer.com/images/press/quickconnect_en.jpg). Man, that's annoying! And users blame our software for this. — c00000fd, Jul 09 '16 at 05:49

score 1 · Answer 1 · edited May 23 '17 at 12:02

1

Just want to post it here for whoever else runs into this issue. After several hours of digging through the crash dump file, eventually WinDbg was able to pinpoint the culprit (read my postings in the question itself.)

The bottom line is that our customer was using TeamViewer with their Quick Connect option that basically injects third-party UI elements into running programs. This in combination with apparent memory leak (see my OP above and discussion of the use of the xpsp2res.dll module) caused the crash.

The actual crash happened in tv_w32.dll that was unloaded before the dump was generated. And since I don't have source files for TeamViewer, nor that it is in any way related to our software, that's pretty much all I can do at this point.

PS. As a bottom line, having analyzed the dump, I would strongly advise people against using TeamViewer as a go-to remote desktop solution. It seems to be very buggy and causes memory leaks.

edited May 23 '17 at 12:02

Community

1
1

answered Jul 09 '16 at 06:29

c00000fd

20,994
29
177
400

Your analysis is great. I too hit a crash similar to this today (tv_x64.dll, exact version is 12.0.83369) - and resolved it by closing Teamviewer. So there *is* a bug here. -- However, I have to say that "seems to be very buggy" and "strongly advise against it" are *so* over the top. We've been using TV for years, and this is the first confirmed problem I have with it. It's an excellent tool that has saved us tons of time. So Bug: yes. But still a good tool. – Martin Ba Sep 21 '17 at 08:59
@MartinBa: [Ref1](https://www.scmagazineuk.com/teamviewer-has-potential-security-flaw-reddit-community-in-upheaval/article/531004/), [Ref2](https://arstechnica.com/information-technology/2016/06/teamviewer-users-are-being-hacked-in-bulk-and-we-still-dont-know-how/), [Ref3](https://arstechnica.com/information-technology/2016/06/teamviewer-says-theres-no-evidence-of-2fa-bypass-in-mass-account-hack/), [ref4](https://www.theregister.co.uk/2016/06/06/teamviewer_sorry/). Having dug in a little bit into the TeamViewer code w/debugger, I totally stand by my "buggy" & `insecure` statement. #SloppyCode – c00000fd Sep 21 '17 at 17:12
Oh my. A Software has a vulnerability. Shocking. (Or maybe not. Whatcha trying to prove by 1 year old mass media reports of all the same incident?) – Martin Ba Sep 21 '17 at 18:04

Crash dump retrieved from a different OS points to message loop. How to get to the fault location from there? (TeamViewer crashes my process.)

1 Answers1

Linked