How can I send mouse clicks and drags to another program that is in the background?

Question

I am trying to write a program that takes in the visual output from another program (we'll call it P2) and displays it on a screen in a 3d world.

The flow of the image capture is as follows:

P2 -> BitBlt -> OpenGL Texture -> 3d World

I want the user to be able to look around and click on elements in P2 using a crosshair which would be stationary in the middle of the viewport of my program. However, in my implementation it seems that I would need to have 2 cursors - one to control the camera in my 3d world and one to control P2 - which I don't think is trivial.

Additionally, since a window only updates what part of it is visible and I want to have P2 constantly updating in as high resolution as possible, it makes sense to have P2 on a second virtual desktop. This makes the issue of interaction more difficult.

Some solutions I can think of with their downsides:

Make a Compositing Window Manager
- Sounds like a lot of work and I haven't been able to find any documentation on how this might be done.
Have the actual cursor over P2 but get the change in cursor position and use this to move the camera in the 3d world
- This wouldn't be a linear transformation from planar to spherical coordinates. Additionally, I'm not sure if the cursor could be on a different Desktop to the current Desktop.

I am open to suggestions of alternative capture methods if they would help. One example would be hooking the DirectX or OpenGL output of P2 and, if necessary, tricking it into rendering while not being active. This might allow P2 to be minimized but would not solve the issue of input.

Maybe it would be possible to hook the input functions of P2? Would this even be advised?

Here are some images to illustrate my program.

Update:

I have implemented SendNotifyMessage() and found that when I sent a WM_LBUTTONDOWN message to the application it became the foreground window. I set WS_EX_NOACTIVATE on P2 to stop this behaviour however, it still steals focus. I then need to Sleep(), presumably until P2 processes the messages, and then use SetForegroundWindow(). Note that SetFocus() and SetActiveWindow() do not result in my program regaining focus (so maybe focus is the wrong word). Are there any methods to remove this delay while keeping the SendNotifyMessage() asynchronous?

Additionally, in trying to use PostMessage() I found that the coordinate transformation was not correct. However, it worked perfectly in SendNotifyMessage(). What could be causing this behaviour?

Relevant Code

After getting the handle to the window I want to capture I run this:

prevWndStyle = GetWindowLong(hwnd, GWL_EXSTYLE);
SetWindowLong(hwnd, GWL_EXSTYLE, prevWndStyle | WS_EX_NOACTIVATE);

After receiving a WM_INPUT message in my WndProc and processing the input I run this code:

if (raw->data.mouse.usButtonFlags & RI_MOUSE_LEFT_BUTTON_DOWN) {
    int hitX, hitY;
    if (gls->Click(vec3(0.0f, 0.0f, 1.0f), 0, hitX, hitY)) {
        LPARAM lParam = MAKELPARAM(hitX, hitY);
        SendNotifyMessage(gls->ic.GetHWND(), WM_MOUSEMOVE, 0, lParam);
        SendNotifyMessage(gls->ic.GetHWND(), WM_LBUTTONDOWN, MK_LBUTTON, lParam);
        SendNotifyMessage(gls->ic.GetHWND(), WM_LBUTTONUP, 0, lParam);

        // I want to get rid of these
        Sleep(100);
        SetForegroundWindow(gls->aw.GetHWND());
    }
}

Update 2

I have found that WS_EX_NOACTIVATE stops the switch to the second desktop however when P2 is on the same desktop, P2 is brought to the foreground.

+1 because it seems like interesting questions overall. but i don´t really get what the key issue is. you pretty much made it clear the built-in cursor handling in desktops won´t cut it so why not "just do it" based on what you want? — Andreas, Oct 18 '17 at 19:25
1) `FindWindow()` 2) Continuously capture screenshots of the window 3) Transform the point to be relative to the window 4) `SendMessage()` a mouse click to the window. *I'll give a more elaborate answer later when I have time, if nobody else has done it by then.* — vallentin, Oct 18 '17 at 19:57
@Andreas I'm not quite sure what you mean by "just do it". The problem is that I don't know how to do it. Could you elaborate please? — user2471379, Oct 19 '17 at 04:46
@Vallentin Thank you for the suggestion. I implemented `SendNotifyMessage()` and found that when I sent a WM_LBUTTONDOWN message to the application it became the foreground window. I set `WS_EX_NOACTIVATE` on P2 to stop this behaviour however, it still steals focus. I then need to `Sleep()`, presumably until P2 processes the messages, and then use `SetForegroundWindow()`. Note that `SetFocus` and `SetActiveWindow()` do not result in my program regaining focus (so maybe focus is the wrong word). Are there any methods to remove this delay while keeping the `SendMessage()` asynchronous? — user2471379, Oct 19 '17 at 05:33
I´ll try elaborate. You have P1 and P2. Both have "cursors". When P1 is in focus the P1 cursor controls P1 and P2 and vice versa. When PX is out of focus its position is retrieved from PY. Is this a problem? — Andreas, Oct 20 '17 at 16:22
Regarding updating rendered image in out-of-focus P this should be possible by configuring your window through the compositor. Which compositor are you using for P1 and P2? Are you developing both P1 and P2? — Andreas, Oct 20 '17 at 16:27
@Andreas `SendNotifyMessage()` seems like a simpler solution than having two cursors. Although I still have the issue of having to `SetForegroundWindow()` after each click I imagine that would still be present if using two cursors. — user2471379, Oct 20 '17 at 16:36
@Andreas Currently I am moving (dragging with the mouse currently) P2 to a separate desktop. Here it still updates. I tried to research compositing on Windows but couldn't find anything applicable to me. How would I configure my window through the compositor? — user2471379, Oct 20 '17 at 16:36
One solution might be to set P2 as a child process of P1 (https://stackoverflow.com/questions/3459874) and use WS_EX_COMPOSITED to slave the update frequency of P2 to P1. Can´t say i´ve done those shananigans myself but might just work... — Andreas, Oct 21 '17 at 13:15
@Andreas It seems that setting P1 as the parent of P2 brings P2 into the foreground. I have tried hiding P2 using `ShowWindow()` with `SW_HIDE` but the image displayed inside P1 is then a blank window. — user2471379, Oct 22 '17 at 13:50

score 0 · Answer 1 · answered Oct 19 '17 at 05:26

0

I think you can use the WIN32API to simulate clicks and also to capture window contents, I've seen it done before in different 3d "window" environments and also I've seen it done in things like Termina Server.

So maybe when "they click you, you pass the click forward" (while doing coordinate transformation)?

Some links for your perusal:

https://msdn.microsoft.com/en-us/library/windows/desktop/gg153548%28v=vs.85%29.aspx?f=255&MSPPError=-2147217396

http://www.cplusplus.com/forum/windows/24162/

answered Oct 19 '17 at 05:26

Felipe Valdes

1,998
15
26

Thank you for your answer. I have implemented `SendNotifyMessage()` but I am having issues with the focus of the windows. I have updated my question to reflect this. – user2471379 Oct 19 '17 at 05:38
maybe creating a virtual desktop perhaps? – Felipe Valdes Oct 19 '17 at 05:45
P2 is already on another virtual desktop. Before I set `WS_EX_NOACTIVATE` on P2 it would switch to that desktop. Now I have to `Sleep()` and then `SetForegroundWindow()` before my program has keyboard and mouse focus. – user2471379 Oct 19 '17 at 05:49

How can I send mouse clicks and drags to another program that is in the background?

Update:

Relevant Code

Update 2

1 Answers1