Realtime video processing for the complete Windows desktop

Question

I want to add dynamic video effects to the complete Windows desktop.

I want to be able to turn the screen grey, bevel the edges and add some scan lines to make it look like an old CRT screen, or make the screen glitch, the way they show hacked systems in movies that don't understand tech, etc.

The effects themselves are out of scope of this question. This question is how to apply them, i.e. how do I get the desktop image produced by Windows, apply my effects, and present the result on the same display.

I know several ways which might work.

1. Hook into WinAPI calls which draw stuff.
2. Create a fake secondary monitor making it the primary display device, grab video stream from it, apply my effects, and present on the real monitor
3. Create a custom display driver which applies the effects.

All of them have downsides: complexity, driver signing requirement, complicated setup. Any better ways to achieve what I want?

Answer 1

1 is very hard, too many different APIs.

2 The hard part is making a fake monitor. If you’ll instead buy a $10 device called “HDMI dummy plug” it will become relatively simple, with 100% documented API. Use desktop duplication API to get texture of monitor 1, apply whatever effect you want and show that on monitor 2. If you want good performance, you better implement the processing completely on GPU, e.g. render a quad with a custom pixel shader.

3 will work but very hard to do.

There’s another way. It’s tricky to implement and uses undocumented APIs, but quite reliable in my experience, at least it was so on Windows 7. Write a DLL that injects itself into dwm.exe. That’s a windows process “desktop windows manager”, it composes whatever is visible on desktop. After DLL inject, create a new thread, in that thread call D3D11CreateDeviceAndSwapChain, then use e.g. MinHook to intercept Present, and ideally also ResizeBuffers methods of IDXGISwapChain interface. If succeeded, dwm.exe will call functions from your DLL every time it presents a frame, or when desktop resolution changes. Then, in the present functions you can do your stuff, e.g. add another render pass implementing your effect, then call original implementations to actually present the result to desktop.

This is easy in theory but quite tricky to implement in practice, however. E.g. it’s hard to debug dwm.exe, you’ll have to rely on logging, or maybe use a virtual machine with remote debugger. Also this is not portable across windows versions. Another limitation, it won’t work for full-screen apps like videogames, they bypass dwm.exe. With videogames, will only work for “borderless full-screen window” in-game setting.

Update: another approach, much simpler. Create a topmost full screen window with per-pixel transparency. The OS supports them for decades, set WS_EX_LAYERED and WS_EX_TRANSPARENT extended style bits. You won’t be able to do grayscale because you can only overlay your stuff but not read what's underneath, but edges, scanlines, and glitches are totally doable. For best performance, use some GPU-centric API to render that window, e.g. Direct2D or D3D in C++. It’s also much easier to debug, either use 2 monitors, or position the transparent window so it occupies a rectangle in the corner leaving you enough screen space for the IDE. Here's an example (not mine).

Answer 2

For capturing and redrawing the desktop on a window, you can't go past the handy windows magnification tool.
What you will want to create, is a full screen version of it, with custom processing to make it look how you want.

Luckily, there is a GitHub containing working source for it

You'll notice there is a function in the fullscreen file:

//
// FUNCTION: SetColorGrayscaleState()
//
// PURPOSE: Either apply grayscale to all colors on the screen, or restore the original colors.
//
void SetColorGrayscaleState(_In_ BOOL fGrayscaleOn)
{
    // Apply the color matrix required to either invert the screen colors or to show the regular colors.
    PMAGCOLOREFFECT pEffect = (fGrayscaleOn ? &g_MagEffectGrayscale : &g_MagEffectIdentity);

    MagSetFullscreenColorEffect(pEffect);
}

Which gives you a pretty solid basis for working out how to affect the pixels being drawn back to the window.

To make this window click through-able, you need to create it using something like the following line of code:

HWND hWnd = CreateWindowEx(WS_EX_LAYERED|WS_EX_TRANSPARENT, cName, wTitle, NULL, 0, 0, 640, 480, NULL, 0, GetModuleHandle(NULL), 0);

The important part here is the WS_EX_LAYERED|WS_EX_TRANSPARENT flag combination.

From the microsoft docs:

NOT IMPORTANT. However, if the layered window has the WS_EX_TRANSPARENT extended window style, the shape of the layered window will be ignored and the mouse events will be passed to other windows underneath the layered window.

With this base, and the knowledge of how to build a clickthrough window, you should be able to make a program that changes the colours / adds some artifacts to a full screen window in front of the desktop.

All programs I have made this way have had lower FPS than your native desktop, but should still be fine for a cool looking program.