Currently we're working on the subject of manually loading DLL in lower 4Gbs of process' virtual address space on x64 platform. It is needed because DLL is written with explicit usage of 32-bit types everywhere and cannot be rewritten. So we are to use this peculiarity in the project and we are to make sure DLL uses only lower 4Gbs to stay workable.
There are some links across the web with implementation of manual DLL loading, we've taken this one as a base: link
This variant works. There are only some issues detected so far:
- No debug with sources is possible now, OS just doesn't see this module, it is a merely memory region for it and nothing else, so no PDB is loaded.
- Our project is implemented in such a way that DLL calls function from external framework where some payload is located. Then exception is raised in the framework (on purpose, not occasionally) and here is where problem occurs. This exception remains unhandled but handler is present in the code of the framework.
When DLL is loaded via LoadLibrary on x86 or x64 (and we're lucky it loads in lower 4Gbs region) everything works just fine. We can see the whole SEH chain (in WinDbg, for example) and exception is handled fine.
When loading DLL manually WinDbg shows something like this:
>!exchain
Frame 0x01: MSVCR120D!__ExceptionPtr::_RethrowException+0x1e1 (000007fe`d9cf4281)
ehandler MSVCR120D!__GSHandlerCheck (000007fe`d9e11eb0)
Frame 0x0b: error getting module for 000000000214daa1
Frame 0x0c: error getting module for 0000000000000003
Frame 0x0d: error getting module for 0000000100000000
Frame 0x0e: error getting module for 0000000002ffa420
Frame 0x0f: error getting module for 0000000100000000
Frame 0x10: error getting module for 0000000000000004
We've tried turning /SafeSEH
option off but with the same result. We've done it because one guess was that OS can refuse to process exception handlers which are not in protected modules.
Current guess on why it happens is that OS needs so to say internally visible module (with some kernel objects created in the process of DLL loading via legal system function LoadLibrary) where exception chain can go through.
What do you think about possible solutions to this issue?
Edit: answered below.