5

Assuming a scenario where DX12 is being hooked for overlay rendering it looks like the best function to hook is the IDXGISwapChain::Present the same way it was done for DX11. Having this function hooked the swap chain is available and from that the device can be retrieved to create resources. Given those resources it’s possible to record rendering commands too. The problem arises when we are trying to execute the rendering commands as there is no option to retrieve the associated command queue from the swap chain so there is nothing like this:

CComPtr<ID3D12Device> pD3D12Device;
if (pSwapChain->GetDevice(__uuidof(ID3D12Device), (void**)(&pD3D12Device)) == S_OK)
{
    pD3D12Device->GetCommandQueueForSwapChain( swapChain )->ExecuteCommandLists(…);
}

The other option would be creating a new command queue to execute on, like this:

CComPtr<ID3D12Device> pD3D12Device;
if (pSwapChain->GetDevice(__uuidof(ID3D12Device), (void**)(&pD3D12Device)) == S_OK)
{
    D3D12_COMMAND_QUEUE_DESC queue_desc = {};
    queue_desc.Flags = D3D12_COMMAND_QUEUE_FLAG_NONE;
    queue_desc.Type = D3D12_COMMAND_LIST_TYPE_DIRECT;
    HRESULT commandQueueRes = _device->CreateCommandQueue( &queue_desc, IID_PPV_ARGS( &_commandQueue ) );

    _commandQueue->ExecuteCommandLists( ... );
}

This results in an error and subsequent device removal. See the error message below.

D3D12 ERROR: ID3D12CommandQueue::ExecuteCommandLists: A command list, which writes to a swap chain back buffer, may only be executed on the command queue associated with that buffer. [ STATE_SETTING ERROR #907: EXECUTECOMMANDLISTS_WRONGSWAPCHAINBUFFERREFERENCE]

The problem is not resolved even if the ID3D12CommandQueue::ExecuteCommandLists is hooked as well because there is no way to retrieve the associated swap chain from the command queue either.

So my question is what is the recommended way to deal with this problem in a scenario where the swap chain creation happens before the hooking could possibly happen?

moradin
  • 305
  • 2
  • 6
  • I do not have a solution for you if you cannot hook the creation, your best shot is to hook the factory swapchain create sadly to insert a `SetPrivateData` in order to retrieve the value later. You also need to hook ResizeBuffer1 because it is possible to change the command queue, and even possible to associate a command list per buffer in the swapchain. – galop1n Mar 29 '16 at 16:41
  • As I mentioned hooking the swapchain creation is not an option because it can happen before the hooking. Thanks for mentioning ResizeBuffer1, I knew about it but it's good to have it here in case someone reads this topic in the future. – moradin Mar 31 '16 at 08:04

1 Answers1

1

In case anyone is looking for the answer here's what I found out. There is no official way to do this, for overlay rendering the recommended way is to use DirectComposition but this has performance consequences which is not very nice for game overlays.

Investigating the memory a bit there is a possible solution to get the CommandQueue from the swap chain with something like this:

#ifdef _M_X64
    size_t* pOffset = (size_t*)((BYTE*)swapChain + 216);
#else
    size_t* pOffset = (size_t*)((BYTE*)swapChain + 132);
#endif
    *(&_commandQueue) = reinterpret_cast<ID3D12CommandQueue*>(*pOffset);

Obviously this solution is not recommended but it might be useful if someone just want's to do some debugging.

My final solution is to hook into a function that uses the CommandQueue (I use ExecuteCommandLists) and get the pointer there and use it later to render the overlay. It's not completely satisfying but it works as long as there are no multiple swap chains.

moradin
  • 305
  • 2
  • 6