1

I have been trying to set BlueStacks window transparent:

DWORD MakeWindowTransparent(HWND hWnd, unsigned char factor)
{
    /* First, see if we can get the API call we need. If we've tried
     * once, we don't need to try again. */
    if (!initialized)
    {
        HMODULE hDLL = LoadLibrary(L"user32");

        pSetLayeredWindowAttributes =
            (PSLWA)GetProcAddress(hDLL, "SetLayeredWindowAttributes");

        initialized = TRUE;
    }

    if (pSetLayeredWindowAttributes == NULL)
        return FALSE;

    /* Windows need to be layered to be made transparent. This is done
     * by modifying the extended style bits to contain WS_EX_LAYARED. */
    SetLastError(0);

    auto winlong = SetWindowLong(hWnd,
    GWL_EXSTYLE,
    GetWindowLong(hWnd, GWL_EXSTYLE) | WS_EX_LAYERED);

    if ((winlong == 0) && (GetLastError() != 0)) { 
        auto error = GetLastErrorAsString(); 
        return FALSE;
    } 

    if (!pSetLayeredWindowAttributes(hWnd,RGB(255, 255, 255),factor, LWA_COLORKEY | LWA_ALPHA))
    { 
        auto error = GetLastErrorAsString(); 
        return FALSE; 
    } 

    return TRUE;
}


int main() {
  HWND hWnd = FindWindowA(NULL, L"BlueStacks");
  MakeWindowTransparent(hWnd, 0);
}

BlueStacks can run in opengl and directx, I have tested the code above, using both modes.

MakeWindowTransparent is returning 0

pSetLayeredWindowAttributes auto error = GetLastErrorAsString(); error returned is: wrong parameter.

I have tested the code with other OpenGL windows, and it did not pause in any of the errors, also the window got transparent correctly.

Some information I have collected about the window:

enter image description here

Appreciate any help.

  • What is `MakeWindowTransparent` returning? As per docs, call `GetLastError` to see what the system might be telling you: https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-setlayeredwindowattributes – edtheprogrammerguy Nov 23 '20 at 20:30
  • @edtheprogrammerguy `return pSetLayeredWindowAttributes` returns 0 –  Nov 23 '20 at 20:54
  • 1
    Where is the actual failure? `MakeWindowTransparent()` returns false under 3 different conditions - 1) if `SetLayeredWindowAttributes` is not available in `user32.dll`; 2) if `SetWindowLong()` fails; 3) if `SetLayeredWindowAttributes()` fails. So, which failure is actually happening? And what does `GetLastError()` actually return when that failure occurs? Also FYI, calling `GetLastError()` after `SetWindowLong()` is only valid if `SetWindowLong()` returns 0, which you are not checking for. If `SetWindowLong()` returns non-zero, the value of `GetLastError()` is *indeterminate* – Remy Lebeau Nov 23 '20 at 20:55
  • @RemyLebeau GetLastError() are not being triggered `pSetLayeredWindowAttributes` = (user32.dll!0x00007ffff3a33170) `SetWindowLong` returns (262144) –  Nov 23 '20 at 20:59
  • 1
    @Caio that is not what I asked. WHICH of the API functions is failing exactly - `LoadLibrary()`, `GetProcAddress()`, `SetWindowLong()`, or `SetLayeredWindowAttribute()`? And what does `GetLastError()` return exactly for THAT specific failed function? You are checking `GetLastError()` in only 1 of the 4 possible failure points of this code. You need to write better error handling logic if you want to discover the root cause of the failure. – Remy Lebeau Nov 23 '20 at 20:59
  • @RemyLebeau sorry i'm still learning, does this help? https://imgur.com/a/P0Gc4Kl `winlong = SetWindowLong()` and `mwt = makewindowtransparent()` –  Nov 23 '20 at 21:09
  • 1
    @Caio That shows `SetWindowLong()` returning non-zero, which means the subsequent `GetLastError()` needs to be skipped. We can't see whether or not `GetLastError()` after `SetWindowLong()` is returning non-zero, making `MakeWindowTransparent()` exit prematurely before it is able to call `SetLayeredWindowAttributes()`. DO NOT check `GetLastError()` after `SetWindowLong()` UNLESS AND ONLY IF `SetWindowLong()` returns 0, eg: `LONG winlong = SetWindowLong(...); if ((winlong == 0) && (GetLastError() != 0)) return FALSE;` That will then allow `SetLayeredWindowAttributes()` to be called *properly*. – Remy Lebeau Nov 23 '20 at 21:32
  • 1
    @Caio If, after that point, you find `SetLayeredWindowAttributes()` is actually being called, and it is returning 0, then you need to check `GetLastError()` again to find out WHY it is returning 0. – Remy Lebeau Nov 23 '20 at 21:37
  • @RemyLebeau I confirmed the window is not getting WS_EX_LAYERED any idea what else I could try? –  Nov 24 '20 at 23:05
  • @Caio confirmed HOW exactly? – Remy Lebeau Nov 24 '20 at 23:05
  • I have download AutoHotkey and did a loop: `WinSet,ExStyle,+0x02000000,BlueStacks` and checked the window properties using Winspector, on StylesEx it did not get `WS_EX_LAYERED`, I also tested the loop in other windows and it did got the parameter correctly. I too checked the window properties just right after `SetWindowLong(hWnd,GWL_EXSTYLE, GetWindowLong(hWnd, GWL_EXSTYLE) | WS_EX_LAYERED);` –  Nov 24 '20 at 23:14

1 Answers1

1

Why are you using both flags LWA_COLORKEY | LWA_ALPHA? If you are specifying a color RGB(255, 255, 255), just use LWA_COLORKEY.

Also, why are you messing with GetProcAddress? SetLayeredWindowAttributes is available starting with Windows 2000; are you targeting platforms older than that?

I am not familiar with BlueStacks, but SetLayeredWindowAttributes to make a window transparent is only working part of the time suggests that SetLayeredWindowAttributes doesn't work with Direct3D; do you know if BlueStacks uses Direct3D?

For OpenGL see this: How to make an OpenGL rendering context with transparent background?

UPDATE I was playing with that BlueStacks, and figured that you can re-parent it to another window (that's not recommended, as you will need to handle some edge cases). Anyway, I used an existing Notepad window and was able to set alpha on it, and that affected the child BlueStacks window:

int main() {
  HWND hWnd = FindWindow(NULL, L"BlueStacks");
  HWND hWndN = FindWindow(NULL, L"Untitled - Notepad");
  ::SetParent(hWnd, hWndN);

  auto winlong1 = SetWindowLongPtr(hWnd, GWL_EXSTYLE, 0);
  auto winlong2 = SetWindowLongPtr(hWnd, GWL_STYLE, WS_CHILD | WS_VISIBLE);
  ::SetWindowPos(hWnd, 0, 0, 0, 0, 0, SWP_FRAMECHANGED | SWP_NOSIZE | SWP_NOZORDER | SWP_NOREDRAW | SWP_NOACTIVATE);

  MakeWindowTransparent(hWndN, 200);
}

I've also cleaned up your function to get rid of GetProcAddress:

DWORD MakeWindowTransparent(HWND hWnd, unsigned char factor)
{
  auto winlong = SetWindowLong(hWnd,
    GWL_EXSTYLE,
    GetWindowLong(hWnd, GWL_EXSTYLE) | WS_EX_LAYERED);

  if ((winlong == 0) && (GetLastError() != 0)) {
    return FALSE;
  }

  if (!SetLayeredWindowAttributes(hWnd, 0, factor, LWA_ALPHA))
  {
    return FALSE;
  }

  return TRUE;
}

enter image description here

Vlad Feinstein
  • 10,960
  • 1
  • 12
  • 27
  • Windows10, about the colors in calling the function `MakeWindowTransparent(hWnd, 0);` Im currently using opengl @Vlad Feinstein –  Nov 24 '20 at 00:05
  • i tried to follow this example: https://stackoverflow.com/a/4055059/12733526 but my code did not compiles after I add it. I have add #include and the code, and got this error when compiling: `LNK2019 unresolved external symbol __imp_DwmEnableBlurBehindWindow referenced in function "private: enum ResultType __cdecl Line::Launcher(wchar_t *,wchar_t *,bool)"` –  Nov 24 '20 at 11:08
  • I have updated the topic with new information, also the error returned by `GetLastError()` –  Nov 24 '20 at 14:30
  • @Caio (1) could you please remove `LWA_ALPHA` and just pass `LWA_COLORKEY`? (2) Could you please make sure that `hWnd` is not `NULL`? – Vlad Feinstein Nov 24 '20 at 17:51
  • Vlad, thank you for your time, (1) I have removed `LWA_ALPHA` and pass only `LWA_COLORKEY` but it still did not get transparent, (2) I confirmed the handle is not NULL also collect some information about the window, I hope this can help: https://imgur.com/a/o3K313M –  Nov 24 '20 at 18:55
  • @Caio - Re: `but it still did not get transparent` - but did it fail? I still think you have to follow up on `DwmEnableBlurBehindWindow`. Did you link `Dwmapi.lib`? – Vlad Feinstein Nov 24 '20 at 20:11
  • It did failed on `!pSetLayeredWindowAttributes` im gonna try again, i did not had linked `Dwmapi.lib`, Edit: with the included lib it compiled, `auto dwm = DwmEnableBlurBehindWindow(hWnd, &bb)` dwm returned `S_OK`, but window still visible: https://imgur.com/a/YjftlsF –  Nov 24 '20 at 20:27
  • So, the mystery is partially solved: you get `ERROR_INVALID_PARAMETER` because the window is still not `WS_EX_LAYERED`. That's another problem - a call to `SetWindowLong` didn't fail, but it didn't set that style either. For the `DwmEnableBlurBehindWindow`, it appears that you need to do something in the OpenGL app (see that link in my answer). – Vlad Feinstein Nov 24 '20 at 21:08
  • I confirmed the window is not getting `WS_EX_LAYERED` any idea what else I could try? –  Nov 24 '20 at 22:59
  • Vlad, Wow so amazing!! Can't thank you enough :) I'm testing the code right now, and return to give a feedback –  Nov 24 '20 at 23:54
  • Vlad, using SetParent when i try to capture the BlueStack window using `GdipBitmapFromHandle` the bitmap captured is all black, happens the same if I try to capture the re-parent window, here: https://youtu.be/HVrUyvbQqck (black screenshot is an attempt to capture Bluestack hwnd, white screenshot re-parent window) –  Nov 25 '20 at 04:37
  • @Caio - sorry, I don't know how to capture that thing. As this is a different issue, I suggest you post a new question about that. – Vlad Feinstein Nov 25 '20 at 17:22