Hooked NtOpenFile fails when std::wstring is defined on the stack

Question

I have written a hooking dll using the mhook library. In a spezial case the NtOpenFile() fails when a std::wstring is defined as stack var. Defining it on the heap the code is working.

The code is working without problems except when a certain win32 application (lets call it nuisance.exe) tries to open an existing testfile (like c:\temp\anyfile.log) the access fails. Mostly STATUS_INVALID_ACL (0xC0000077) is returned then.

I have reduced my code line by line and finally found that the error happens when in a called function a std::wstring is defined (this example below). The error happens every time an on different OS's

NTSTATUS NtOpenFileApiHook::NtOpenFileHook(PHANDLE              FileHandle,
                                  ACCESS_MASK           DesiredAccess,
                                  POBJECT_ATTRIBUTES    ObjectAttributes,
                                  PIO_STATUS_BLOCK      IoStatusBlock,
                                  ULONG                 ShareAccess,
                                  ULONG                 OpenOptions 
                                  )
{
    NTSTATUS Status = STATUS_SUCCESS;

    // using this function the call will fail
    AfterThis_NtOpenFile_WillFail();

    // using this function INSTEAD the call will work
    AfterThis_NtOpenFile_WillWork();

    // calling the real NtOpenFile using a pointer
    // nothing was changed hier, the original parameters are passed
    Status = RealNtOpenFile(FileHandle, ...);

    return Status;
}

int AfterThis_NtOpenFile_WillFail()
{
    std::wstring String = L"";

    return 0;
}

int AfterThis_NtOpenFile_WillWork()
{
    std::wstring * pString = new std::wstring();
    pString->assign(L"");
    delete pString;

    return 0;
}

I have fixed it this way for this call. But I'm afraid that other functions in other circumstainces could fail so I'm looking for the reason and (probably) for a solution.

Nuisance.exe is a C# application with default stacksize callling a win32 dll about which I know nothing.

[Dangling pointer bugs](http://en.wikipedia.org/wiki/Dangling_pointer) are common in C or C++ code. Undiagnosable until somebody overwrites the stack location. Like your hook will do. — Hans Passant, Jun 25 '14 at 17:16
I do not understand why this bug is in each case (VM, physical, different OS's) happening exactly at this point. Shouldn't such bug (dangling pointer) happen without this regularity? (+1 for that hint - looking at your profile: you have a lot on the ball) — marsh-wiggle, Jun 26 '14 at 06:36
Undefined behavior is undefined. This include sometimes working, always failing in the same place, failing at random places, failing only when the big boss is in the room, and more. — chwarr, Jun 26 '14 at 07:14
@c45207 I agree, that could be so. But specially in this hooking code I produced some errors like buffer overflows and bad pointers. When errors were such deterministic, even after major system changes, the reasons were no memory or pointer corruptions. So I'm just asking if there is a possible other reason. — marsh-wiggle, Jun 26 '14 at 14:02

score 2 · Accepted Answer · answered Jun 15 '21 at 11:14

If Nuisance.exe was a C++ application, I would imagine that it calls NtOpenFile in a way similar to this, allocating one of pointer parameters on overwritten stack:

   POBJECT_ATTRIBUTES    MakeObjectAttributes()
   {
      POBJECT_ATTRIBUTES oa = {...};
      return &oa; // Pointer to stack variable - UB
   }
   ...
   NtOpenFile(..., MakeObjectAttributes(), ...)

STATUS_INVALID_ACL (0xC0000077) error might suggest that SecurityDescriptor within OBJECT_ATTRIBUTES is allocated this way.

Then it matters how much stack is used by AfterThis_NtOpenFile_WillFail, and it is more than AfterThis_NtOpenFile_WillWork, since std::wstring would be larger than just a couple of pointers due to small string optimization.

If the call chain is always the same, the corruption may be deterministic.

I don't know if code equivalent of returning address of temporary is possible in C#. But the DLL may be in C/C++ or similar language that allows dandling pointers.

To prove/disprove the role of stack, try to allocate other data on stack that has std::wstring size. More precise proof could be checking passed pointer to see if they point to stack area that is about to be overwritten.

Thanks for your explanations, this sounds very reasonable. Unfortunately the project is finished and I can't test it any more. — marsh-wiggle, Jun 15 '21 at 11:23

Hooked NtOpenFile fails when std::wstring is defined on the stack

1 Answers1