4

Note: I am open to different solutions which achieve the desired capability

I am working on a project with many instances of the same game.

Therefore, I am sending keyboard and mouse instructions to each of theses processes, in parallel.

I am currently using win32ui as follows:

After finding the processes hwnd (windows handle) values from Get HWND of each Window?, so a hwnds_list with all the processes with a given name e.g. [788133, 723724, ...]

I am sending instructions to each of the processes, by creating a PyCWnd object:

PyCWnd = win32ui.CreateWindowFromHandle(hwnd)

Then, say I want to press the return key, I used:

def press_return(pycwnd):
    pycwnd.SendMessage(win32con.WM_KEYDOWN, win32con.VK_RETURN, 0)
    pycwnd.SendMessage(win32con.WM_KEYUP, win32con.VK_RETURN, 0)

Then I run this in parallel with:

def press_return_par(hwnds):
    # Get the Window from handle
    pycwnd = make_pycwnd(hwnds)
    time.sleep(0.1)
    press_return(pycwnd)

num_workers = len(hwnds_list)
with Pool(num_workers) as p:
    p.map(press_return_par, hwnds_list)

So, I have a good way of sending keyboard commands, and even scrolling with a mouse, but can't work out how to do this with mouse movements.

Ideally, I'd like to say, "Move to (x, y) coordinates over n time". This 'ideal' method needs to not effect the current cursor (or allow a locked cursor for each process/game), as I will want to do this across ~8 instances of the game.

I've looked through the official pywin32 docs http://timgolden.me.uk/pywin32-docs/contents.html, other answers that look bang on https://stackoverflow.com/a/3721198/11181287 but use win32api.mouse_event, so I don't know how to convert this to work with the multiple pycwnd objects.

  • https://stackoverflow.com/a/3721053/11181287 looks close, but doesn't seem to move the mouse, it just does the right click, although I have made some guesses for the MAKELPARAM function which is not listed.
  • In addition, https://github.com/oblitum/Interception could be helpful but haven't found good docs for how to apply this here.
  • As the game is an FPS game, running multiple instances through nucleus-coop, using a VM etc... won't be fast enough (from my current research).
  • PyAutoGUI is exactly the functionality I want, with the speed, but (as expected) I haven't been able to set it up to work for multiple mice/processes
  • There could be something in sending DirectX inputs into the game (black ops 2)?

(I'm running windows 10, Python 3.7.11, and only know Python)

GooJ
  • 257
  • 3
  • 13
  • 2
    [You can't simulate keyboard input with PostMessage](https://devblogs.microsoft.com/oldnewthing/20050530-11/?p=35513). The same issues apply for mouse input. If you want to automate a UI, use [UI Automation](https://learn.microsoft.com/en-us/windows/win32/winauto/entry-uiauto-win32) using, e.g. pywinauto with the `uia` backend. This may not work when targeting programs that do not have a standard UI, such as games. – IInspectable Feb 21 '22 at 08:14
  • @IInspectable Don't quite understand the can't simulate keyboard inputs point, I have done with the above code (even with the mouse wheel) to multiple processes that don't even have to be active using win32ui? My question is rather how do I extend this to the mouse movement inputs. I may be misunderstanding but I don't think my above approach is using PostMessage. Thank you for the suggestion on pywinauto, I will have a look at if this can target specific window handles later today. – GooJ Feb 21 '22 at 12:07
  • Using `SendMessage` doesn't make a difference. The input is fake, and it's down to the target application whether it being fake makes a difference. As for giving more than one application running on a single desktop the impression of having more than one mouse position, that clearly cannot work. [`GetCursorPos`](https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-getcursorpos) doesn't allow a client to spell out, which mouse cursor to identify. There is only one. If you need multiple mouse positions, you'll need to create multiple desktops. – IInspectable Feb 21 '22 at 13:06
  • The "fake" input is fine as it is working for the process with a keyboard - just can't seem to get this working with a mouse, and I don't have enough experience to make sense of the documenation (pretty much all in C++) – GooJ Feb 26 '22 at 20:27
  • Take a look at [this question](https://stackoverflow.com/q/2964051/16775594); it might help you. Unfortunately, I can't test this, since I'm not using Windows, and I don't have ready access to a Windows computer, but you should find some good information there. – Sylvester Kruin Mar 05 '22 at 18:51
  • @syl This is useless. Like anything in the intersection of Python, input faking, and Windows, it's complete garbage. Written by someone that's happy with code that doesn't instantly crash. The *obvious* issue here is, that it's using `mouse_event` (which, by itself, is [wrong](https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-mouse_event) already), so there's no way to control where the fake input goes. It'll get picked up by whichever thread happens to be the foreground thread at the time the code gets interpreted. – IInspectable Mar 13 '22 at 17:12
  • @IInspectable Okay! I'll try not to get involved in questions regarding the intersection of Python, input faking, and Windows; I (quite honestly) hate Windows-specific code anyway. I guess I'll leave it to the experts from now on. Have a good day! – Sylvester Kruin Mar 13 '22 at 17:17

1 Answers1

-1

I have two possible solutions to your mice issue.

  1. What if you used only one mouse to control all of the windows? With pyautogui you could tab into each window when necessary and control the mouse for that window. I'm not sure how efficient this would be and how fast the mouse control for each window would be, but it's still sort of a solution.

OR

  1. You could control the mouse with the keyboard. See this article https://www.windowscentral.com/how-control-mouse-using-keyboard-windows-10

I apologize for not just commenting, unfortunately I don't have enough reputation.

Kasra Panahi
  • 13
  • 2