1

The problem exists at 017D0B5F call eax :

017D0B56  mov         esi,esp 
017D0B58  mov         edx,dword ptr [ebp-20h] 
017D0B5B  push        edx  
017D0B5C  mov         eax,dword ptr [ecx+8] 
017D0B5F  call        eax  
017D0B61  cmp         esi,esp 
017D0B63  call        @ILT+2525(__RTC_CheckEsp) (17C49E2h) 
017D0B68  cmp         dword ptr [ebp-2Ch],0 
017D0B6C  je          CSourceStream::DoBufferProcessingLoop+10Ah (17D0B8Ah) 
017D0B6E  mov         eax,dword ptr [ebp-2Ch] 
017D0B71  push        eax  
017D0B72  push        offset string "Deliver() returned %08x; stoppin"... (17F7278h)

Here's the corresponding source:

 // Virtual function user will override.
 hr = FillBuffer(pSample);

 if (hr == S_OK) {
 hr = Deliver(pSample);
            pSample->Release();

            // downstream filter returns S_FALSE if it wants us to
            // stop or an error if it's reporting an error.
            if(hr != S_OK)
            {
              DbgLog((LOG_TRACE, 2, TEXT("Deliver() returned %08x; stopping"), hr));
              return S_OK;
            }

Is it possible to infer which line in source has the problem according to disassembly?

UPDATE

What does __RTC_CheckEsp mean ?

UPDATE2

Reproducing in debugger

alt text

UPDATE3

alt text

Daniel Trebbien
  • 38,421
  • 18
  • 121
  • 193
ollydbg
  • 3,475
  • 7
  • 28
  • 29

2 Answers2

2

Looks like it is the pSample->Release() call - what error do you get?

017D0B56  mov         esi,esp 
017D0B58  mov         edx,dword ptr [ebp-20h]     // get the pSample this pointer
017D0B5B  push        edx                         // push it
017D0B5C  mov         eax,dword ptr [ecx+8]       // move pSample to eax
017D0B5F  call        eax                         // call it
017D0B61  cmp         esi,esp                     // maybe a stack/heap check?
017D0B63  call        @ILT+2525(__RTC_CheckEsp) (17C49E2h) 
017D0B68  cmp         dword ptr [ebp-2Ch],0       // if hr!=S_OK
017D0B6C  je          CSourceStream::DoBufferProcessingLoop+10Ah (17D0B8Ah) 
017D0B6E  mov         eax,dword ptr [ebp-2Ch] 
017D0B71  push        eax                         // get ready to call DbgLog
017D0B72  push        offset string "Deliver() returned %08x; stoppin"... (17F7278h)
Jeff
  • 1,969
  • 3
  • 21
  • 35
  • Here's the error:`Unhandled exception at 0x0469b000 in FlashPlayer.exe: 0xC0000005: Access violation writing location 0x65bb27f6.`. Are you sure it's `pSample->Release() ` and not `Deliver(pSample);` ? – ollydbg Oct 10 '10 at 04:29
  • @Jeff, your statement for the call is wrong. You say 'call it'. What does that mean? `Release` is probably a virtual function here and the `mov eax,dword ptr [ecx+8]` is most likely computing that function address and that is what it is calling. – linuxuser27 Oct 10 '10 at 04:39
  • Yes, it is the Release() call. The 3rd function in the v-table for a COM object (ecx+8). Sounds like heap corruption to me. – Hans Passant Oct 10 '10 at 06:09
  • @Hans Passant ,Is there a way to fix this kind of corruption? – ollydbg Oct 10 '10 at 09:23
  • It does sound like heap corruption, and this can be difficult to find, since you don't know when the heap corruption occurred. If you can track the v-table for pSample, and find out when it is valid, and when it is corrupt, that's the way to go. Also you should look at the memory addresses around the v-table - see if it's only the one entry, or if a whole block of memory is bad. You might be able to recognize the corrupted bytes, which gives you an idea of where they came from. – Jeff Oct 10 '10 at 14:49
  • I've provided more detail about the crash above.Does it mean the problem is at `Deliver()` or `Release()` ? – ollydbg Oct 10 '10 at 15:39
  • It still looks like the problem is in Release(). Here's two possibilities: 1. The vtable for pSample got corrupted *sometime* before this code. So you are only seeing the effect, not the cause of the problem. So you have to trace the lifetime of pSample and discover when the vtable was corrupted. 2. pSample->Release() is fine, but something in the release of the COM object is corrupt. In that case, it will be much more difficult to find (as if #1 was not hard enough) – Jeff Oct 10 '10 at 16:29
  • Do you know what's `@ILT+2525(__RTC_CheckEsp)` ? – ollydbg Oct 11 '10 at 15:06
  • as linuxuser27 says:"__RTC_CheckEsp is a call that verifies the correctness of the esp, stack, register. It is called to ensure that the value of the esp was saved across a function call. It is something that the compiler inserts for you." – Jeff Oct 11 '10 at 18:13
0

You can use the DIA SDK to query what line of source corresponds to an RVA. Note that DIA requires the symbols (i.e. PDB files). Look at this SO question on RVAs.

After you have determined the RVA for the disassembly in question you can load the PDB for that binary. Create a session and then look at the findLinesByRVA() function on the IDiaSession interface. This will return you an enumeration of lines that correspond to that RVA. Query the resulting IDiaLineNumber instances for what file the line number corresponds to.

Responding to your update, __RTC_CheckEsp is a call that verifies the correctness of the esp, stack, register. It is called to ensure that the value of the esp was saved across a function call. It is something that the compiler inserts for you.

Community
  • 1
  • 1
linuxuser27
  • 7,183
  • 1
  • 26
  • 22
  • Is there a binary utility so that I don't need to build it myself? – ollydbg Oct 10 '10 at 03:52
  • I am not aware of a utility that does that for you other than Visual Studio, Windbg, or some other debugger. Come to think of it though, that would be a nice utility. Maybe a project? :) – linuxuser27 Oct 10 '10 at 03:55
  • Just by looking at it normally no. But what I can tell you is that many times when the compiler generates a `call` to a register, `eax` in this case, that is very often a call to a virtual function because the virtual function is getting computed. Therefore I would assume that issue is with line `pSample->Release();` if `Release()` is virtual. If not, `Release()` could have been inlined and the error is in that function. Why is a debugger not an option here? – linuxuser27 Oct 10 '10 at 04:30