How to prevent color key flickering visibly when enabling LWA_COLORKEY

Question

I have a window that should sometimes have a transparent hole in it, and sometimes not. Ideally, we would use SetWindowRgn, but that disables visual styles, which not only looks ugly but doesn't draw correctly with per-monitor DPI-awareness, so I am trying to use a layered window with a color key.

When enabling the color key, I first call SetLayeredWindowAttributes(hWnd, colorkey, 0, LWA_COLORKEY), then invalidate the window so that it is redrawn. At this point the window should not contain the key color. Then the window receives WM_PAINT at some point later, and the key color is painted, but at this point the window should have LWA_COLORKEY set, so again, I expect the key color not to be visible.

When disabling the color key, I first repaint the window (synchronously) so that it does not contain the key color, and then disable WS_EX_LAYERED, so again, I never expect to see the key color.

However, a window with the following window procedure constantly flickers between green, transparent and the background color as the mouse moves a across it.

It seems that perhaps SetLayeredWindowAttributes does not take effect immediately (and not even before the next WM_PAINT). How can I make sure that this attribute has taken effect before repainting, or otherwise prevent the key color being visible?

LRESULT CALLBACK WndProc(HWND hWnd, UINT message, WPARAM wParam, LPARAM lParam)
{
    static auto const colorkey = RGB(0,255,0);
    static auto const hbrush = CreateSolidBrush(colorkey);
    static auto transparent = false;
    switch (message)
    {
    case WM_PAINT:
        {
            PAINTSTRUCT ps;
            HDC hdc = BeginPaint(hWnd, &ps);
            if (transparent) {
                RECT rect{30,30,500,500};
                FillRect(hdc, &rect, hbrush);
            }
            EndPaint(hWnd, &ps);
        }
        break;
    case WM_MOUSEMOVE:
        if (transparent) {
            transparent = false;
            RedrawWindow(hWnd, nullptr /* lprcUpdate */, nullptr /* hrgnUpdate */, RDW_INVALIDATE | RDW_ERASE | RDW_UPDATENOW | RDW_ALLCHILDREN);
            SetWindowLongPtr(hWnd, GWL_EXSTYLE, GetWindowLongPtr(hWnd, GWL_EXSTYLE) & ~WS_EX_LAYERED);
        } else {
            SetWindowLongPtr(hWnd, GWL_EXSTYLE, GetWindowLongPtr(hWnd, GWL_EXSTYLE) | WS_EX_LAYERED);
            SetLayeredWindowAttributes(hWnd, colorkey, 0, LWA_COLORKEY);
            transparent = true;
            InvalidateRect(hWnd, nullptr, TRUE);
        }
        break;
    case WM_DESTROY:
        PostQuitMessage(0);
        break;
    default:
        return DefWindowProc(hWnd, message, wParam, lParam);
    }
    return 0;
}

@Anders: calling `SetWindowPos(hWnd, nullptr, 0, 0, 0, 0, SWP_SHOWWINDOW | SWP_NOSIZE | SWP_NOMOVE | SWP_NOACTIVATE | SWP_NOZORDER)` after SetWindowLongPtr doesn't make a difference, if that's what you're referring to. Otherwise a reference to the MSDN page you have in mind would be helpful. — Max, Jul 02 '19 at 13:18
@Anders: gotta love the way MSDN says you have to do something, but doesn't just come out and tell you what you have to do. But in any case, no, that doesn't fix it. We've managed to find a workaround to leave the transparency enabled, but I'll leave the question in case somebody else has the same issue. — Max, Jul 03 '19 at 09:39
@Max SetLayeredWindowAttributes with color key make that "all pixels **painted by the window in this color** will be transparent." but it seems you are preventing from the windows painted by your specified color (green), right? — Rita Han, Jul 03 '19 at 09:41
@RitaHan-MSFT I'm not sure what you're asking. The problem is that the pixels painted by the window in the specified color do not immediately become transparent. — Max, Jul 05 '19 at 07:56
@Max I can reproduce your issue. What you want, looks like for me, is disable the green color to display, if not, how do you define the "immediately"? — Rita Han, Jul 05 '19 at 08:24
@RitaHan-MSFT: since the window has LWA_COLORKEY set, painting green should cause the window to be transparent: the green color should *never* be seen. — Max, Jul 05 '19 at 08:40
@Max I reproduced this issue. But it works when debugging step by step. I'll consult related engineer for this issue to see if it is expected behavior. — Rita Han, Jul 09 '19 at 07:04

score 0 · Answer 1 · answered Jul 05 '19 at 17:42

I don't think layered windows are designed to be turned on and off multiple times per second (Windows will allocate/destroy a 32 BPP image etc. each time you toggle).

SWP_FRAMECHANGED and an extra erase does make it much better for me at least:

case WM_MOUSEMOVE:
    if (transparent) {
        transparent = false;
        RedrawWindow(hWnd, nullptr /* lprcUpdate */, nullptr /* hrgnUpdate */, RDW_INTERNALPAINT|RDW_INVALIDATE|RDW_ERASE|RDW_ERASENOW|RDW_UPDATENOW|RDW_ALLCHILDREN);
        SetWindowLongPtr(hWnd, GWL_EXSTYLE, GetWindowLongPtr(hWnd, GWL_EXSTYLE) & ~WS_EX_LAYERED);
        SetWindowPos(hWnd, 0, 0, 0, 0, 0, SWP_NOMOVE|SWP_NOSIZE|SWP_NOZORDER|SWP_FRAMECHANGED|SWP_NOACTIVATE);
        InvalidateRect(hWnd, nullptr, true);
    } else {
        SetWindowLongPtr(hWnd, GWL_EXSTYLE, GetWindowLongPtr(hWnd, GWL_EXSTYLE) | WS_EX_LAYERED);
        SetLayeredWindowAttributes(hWnd, colorkey, 0, LWA_COLORKEY);
        SetWindowPos(hWnd, 0, 0, 0, 0, 0, SWP_NOMOVE|SWP_NOSIZE|SWP_NOZORDER|SWP_FRAMECHANGED|SWP_NOACTIVATE);
        transparent = true;
        InvalidateRect(hWnd, nullptr, true);
    }
    TCHAR b[99];wsprintf(b,TEXT("%d tick=%d"), transparent, GetTickCount()), SetWindowText(hWnd, b);
    break;

The problem was that the window could contained user-defined content so no single color was guaranteed to be available. We worked around this by dynamically choosing the key color. — Max, Jul 08 '19 at 08:51

score 0 · Answer 2 · answered Jul 29 '19 at 09:27

It seems those layered window changes take some time to be reflected in the rendering of the window. Add a sleep can make the green doesn't show.

case WM_MOUSEMOVE:
    if (transparent) {
        transparent = false;
        RedrawWindow(hWnd, nullptr /* lprcUpdate */, nullptr /* hrgnUpdate */, RDW_INVALIDATE | RDW_ERASE | RDW_UPDATENOW | RDW_ALLCHILDREN);
        SetWindowLongPtr(hWnd, GWL_EXSTYLE, GetWindowLongPtr(hWnd, GWL_EXSTYLE) & ~WS_EX_LAYERED);
    }
    else {
        SetWindowLongPtr(hWnd, GWL_EXSTYLE, GetWindowLongPtr(hWnd, GWL_EXSTYLE) | WS_EX_LAYERED);
        SetLayeredWindowAttributes(hWnd, colorkey, 0, LWA_COLORKEY);
        Sleep(1); // Add sleep
        transparent = true;
        InvalidateRect(hWnd, nullptr, TRUE);
    }
    break;

How to prevent color key flickering visibly when enabling LWA_COLORKEY

2 Answers2