64-bit VMWare detection code triggers breakpoint

Question

I'm doing VMWare detection in a .NET 2.0 C# console project. The detection code is implemented in C as an exported function in a DLL that's called from the C# code using P/Invoke. The C# code is compiled as x86 and as x64 into two separate executables. The C DLL is also compiled for both platforms. Here's the exported function:

PUBLIC Int32 __declspec(nothrow) WINAPI GetVMType ()
{
    Int32 nVMWareType = 0;

    try
    {
        if ( !IsInVMWare ( nVMWareType ) )
        {
            ... // allocate memory, write data, etc.
        }
    }
    catch ( ... )
    {
        nVMWareType = -1;
    }

    return ( nVMWareType );
} // <-- breakpoint happens here

I have the following code using inline assembly for VMWare detection:

PRIVATE Bool IsInVMWare ( Int32& nType )
{
    Bool bResult = false;
    Int32 nVersion = -1;
    nType = -1;

    __try
    {
        #ifndef _WIN64 // 32-bit detection
        __asm
        {
            push    edx
            push    ecx
            push    ebx

            mov     eax, 'VMXh'
            mov     ebx, 0          // anything but 'VMXh'
            mov     ecx, 10         // get VMWare version
            mov     edx, 'VX'       // port number
            in      eax, dx         // read port
            cmp     ebx, 'VMXh'     // is it a reply from VMWare?
            je      lblInVMWare

            xor     ecx, ecx        // not in VMWare - clear return value

        lblInVMWare:
            mov     [nVersion], ecx // vmware product type
            pop     ebx
            pop     ecx
            pop     edx
        }
        #else
        nVersion = GetVMWareVersion (); // 64-bit detection
        #endif

        nType = nVersion;

        if ( nType > 0 )
            bResult = true;
    }
    __except ( EXCEPTION_EXECUTE_HANDLER )
    {
        bResult = false;
    }

    return ( bResult );
}

The 64-bit detection code is implemented as GetVMWareVersion() in assembly:

PUBLIC GetVMWareVersion
    .CODE
    ALIGN   8

GetVMWareVersion PROC

    push    rbp
    mov     rbp, rsp

    push    rdx
    push    rcx
    push    rbx

    mov     rax, 'VMXh'
    mov     rbx, 0       ; anything but 'VMXh'
    mov     rcx, 10      ; get VMWare version
    mov     rdx, 'VX'    ; port number
    in      rax, dx      ; read port
    cmp     ebx, 'VMXh'  ; is it a reply from VMWare?
    je      $0@GetVMWareVersion

    xor     rax, rax     ; not in VMWare - clear return value
    jmp     $1@GetVMWareVersion

$0@GetVMWareVersion:
    mov     rax, rcx     ; VMWare product type

$1@GetVMWareVersion:
    pop     rbx
    pop     rcx
    pop     rdx

    mov     rsp, rbp
    pop     rbp
    ret

GetVMWareVersion ENDP

END

The 32-bit detection code runs fine on a non-VM Windows 7. When the 64-bit version runs (same environment), it triggers a DebugBreak() call with the following call stack:

KernelBase.dll!DebugBreak() + 0x2 bytes
[Frames below may be incorrect and/or missing, no symbols loaded for KernelBase.dll]
mscorwks.dll!PreBindAssembly() + 0x9ce69 bytes
mscorwks.dll!PreBindAssembly() + 0x9d28e bytes
mscorwks.dll!CreateApplicationContext() + 0x769d bytes
mscorwks.dll!StrongNameTokenFromPublicKey() + 0x64f8 bytes
mscorwks.dll!StrongNameTokenFromPublicKey() + 0x66ff bytes
mscorwks.dll!CreateApplicationContext() + 0x7f62 bytes
ntdll.dll!vsprintf_s() + 0x12b bytes
ntdll.dll!RtlUnwindEx() + 0x852 bytes
ntdll.dll!KiUserExceptionDispatcher() + 0x2e bytes
mscorwks.dll!IEE() + 0xd285 bytes
cccccccccccccccc()
0000000000d78180()
cccccccccccccccc()

I also have this in the Output window when I debug the C DLL in Visual Studio:

First-chance exception at 0x000007feedbdfad1 (mscorwks.dll) in MyApp.exe:
    0xC0000005: Access violation reading location 0xffffffffffffffff.

Every once in a while (not always) I also get this in the Event Log:

.NET Runtime version 2.0.50727.5456 - Fatal Execution Engine Error
    (000007FEEDB27916) (80131506)

I'm at a total loss as to why this happens. I did a lot of searching but I couldn't find anything that would seem to apply to this problem. Some sites suggest to switch to .NET 4.0 but that's not an option.

I analyzed all the code in GetVMType () - it does some memory allocation and writes data into dynamically allocated arrays but that code is correct: all memory is released and no memory is overwritten incorrectly.

If I modify the 64-bit assembly code to skip the in instruction, the breakpoint is not triggered. This is not an issue when running as 32-bit code.

The calling C# program has a top-level exception handler and an event handler for UnhandledException event but when this problem happens, the application just quits - none of the handlers are called.

Does anyone have any idea what could be wrong with this setup? I spent hours of debugging and trying to see what's happening but it seems something inside .NET breaks when the P/Invoke call returns after that in instruction executes.

Answer 1

My guess is that your function corrups registers.

Running on real hardware (non-VM) should probably trigger exception at "in rax, dx". If this happens then control is passed to your exception handler, which sets result, but does not restore registers. This behaviour will be fully unexpected by caller. For example, it can save something into EBX/RBX register, then call your asm code, your asm code does "mov RBX, 0", it executes, catches exception, sets result, returns - and then caller suddently realizes that his saved data isn't in EBX/RBX anymore! If there was some pointer stored in EBX/RBX - you're going to crash hard. Anything can happen.

Surely, your asm code saves/restores registers, but this happens only when no exception is raised. I.e. if your code is running on VM. Then your code does its normal execution path, no exceptions are raised, registers will be restored normally. But if there is the exception - your POPs will be skipped, because execution will be passed to exception handler.

If this is true, then your 32-bit code doesn't work either. If it's working - then it happens only by coinsidence. I.e. by pure chance. And that chance is that in case of 32-bit code which have less available registers to use, someone already saves/restores registers before calling into your asm code.

The correct code should probably do PUSH/POPs outside of try/except block, not inside.