2

I searched a lot for simulate mouse clicks and movement for directx games. I found a good sources about keypressing but nothing for mouse. Actually there is good stackoverflow topic about key press with direct input (Simulate Python keypresses for controlling a game). But i don't have enough experience to make it work for mouse clicks on certain location.

I tried a lot of python module like pyautogui, win32 etc. they are not working. Even i tried to click it over autohotkey with sending arguments to '.ahk' file but it's not stable and not a good way. I will appreciate for every comment. I'm working on just this click for 2 days and i'm totally lost. Thanks!

I found this code from reddit for clicks in game but it's not working either.

import ctypes

PUL = ctypes.POINTER(ctypes.c_ulong)

class KeyBdInput(ctypes.Structure):
    _fields_ = [("wVk", ctypes.c_ushort),
                ("wScan", ctypes.c_ushort),
                ("dwFlags", ctypes.c_ulong),
                ("time", ctypes.c_ulong),
                ("dwExtraInfo", PUL)]


class HardwareInput(ctypes.Structure):
    _fields_ = [("uMsg", ctypes.c_ulong),
                ("wParamL", ctypes.c_short),
                ("wParamH", ctypes.c_ushort)]


class MouseInput(ctypes.Structure):
    _fields_ = [("dx", ctypes.c_long),
                ("dy", ctypes.c_long),
                ("mouseData", ctypes.c_ulong),
                ("dwFlags", ctypes.c_ulong),
                ("time", ctypes.c_ulong),
                ("dwExtraInfo", PUL)]


class Input_I(ctypes.Union):
    _fields_ = [("ki", KeyBdInput),
                ("mi", MouseInput),
                ("hi", HardwareInput)]

class Input(ctypes.Structure):
    _fields_ = [("type", ctypes.c_ulong),
                ("ii", Input_I)]

def set_pos(x, y):
    x = 1 + int(x * 65536./1920.)
    y = 1 + int(y * 65536./1080.)
    extra = ctypes.c_ulong(0)
    ii_ = Input_I()
    ii_.mi = MouseInput(x, y, 0, (0x0001 | 0x8000), 0, ctypes.pointer(extra))
    command = Input(ctypes.c_ulong(0), ii_)
    ctypes.windll.user32.SendInput(1, ctypes.pointer(command), ctypes.sizeof(command))

def left_click():
    extra = ctypes.c_ulong(0)
    ii_ = Input_I()
    ii_.mi = MouseInput(0, 0, 0, 0x0002, 0, ctypes.pointer(extra))
    x = Input(ctypes.c_ulong(0), ii_)
    ctypes.windll.user32.SendInput(1, ctypes.pointer(x), ctypes.sizeof(x))

    extra = ctypes.c_ulong(0)
    ii_ = Input_I()
    ii_.mi = MouseInput(0, 0, 0, 0x0004, 0, ctypes.pointer(extra))
    x = Input(ctypes.c_ulong(0), ii_)
    ctypes.windll.user32.SendInput(1, ctypes.pointer(x), ctypes.sizeof(x))

left_click() function is working but click works with all modules what i need is set_pos() to work but unfortunately it's not.

sword1st
  • 525
  • 3
  • 7
  • 21

3 Answers3

4

I did a lot of research and found pyautoit module. It's really great and easy to use. I think it's only works with 32 bit python i couldn't install it to 64 bit version. You should install autoitx3 too but i'm not sure maybe it can work with just one dll. After install autoitx3 and pyautoit module you can use this sample code : 

import autoit
import time

time.sleep(2)
#"left" stand for left click, 1243 and 1035 is x and y coordinates and 1 is number of clicks.
autoit.mouse_click("left", 1243, 1035, 1)
sword1st
  • 525
  • 3
  • 7
  • 21
1

For Directx Games you must Self test it out, if the Direct Input does Works.

But with pywinauto Package you can simulate Mouse Clicks and Mouse Movements.

Look at pywinauto.mouse

To Install pywinauto Package for python 27 you can use this bat file.

install-Pywinauto.bat

C:\Python27\scripts\pip.exe install pywinauto
pause

Now you are ready.

Left-mouse-click.py 

import pywinauto
pywinauto.mouse.click(button='left', coords=(0, 0))

Double-Left-Mouse-Click.py

import pywinauto
pywinauto.mouse.double_click(button='left', coords=(0, 0))

and if you use it together with AutoPytonLauncher

you can Click a 3d Button Image on your Desktop to send any Keyboard Shortcuts Macro's or MouseClicks or MouseMovements to Windows Applications or Games. (without Loosing Focus the Active Windows.)

stevecody
  • 658
  • 1
  • 6
  • 18
  • Thank you for your comment but it's not working. It's just work without game. It can't click given coordinate when on game. – sword1st Mar 02 '19 at 21:52
  • ok, can you say in what Game it does not work, and did you try it together AutoPythonLauncher (it can send/execute without focus the Windows). - note: if it still does not work than the Game Maker did prevent to use the Mouse Coordinate Movements by commands, it is then taken By the Game Self. (maybe you can try to make a step by step Macro Movements, Example: getmousePosition + then true send a Keyboard Shortcuts macro's Send 10xUp(key)+10xLeft(key) – stevecody Mar 03 '19 at 09:11
  • And then If the Mouse Pointer is arrived on that Coordinate, maybe you can try to do a Simulate MouseClick with a Keyboard Shortcut Example:send 1xEnter(key) – stevecody Mar 03 '19 at 09:24
1
def move(x=None, y=None, duration=0.25, absolute=True, interpolate=False, **kwargs):

    if (interpolate):
        print("mouse move {}".format(interpolate))
        current_pixel_coordinates = win32api.GetCursorPos()
        if interpolate:
            current_pixel_coordinates = win32api.GetCursorPos()
            start_coordinates = _to_windows_coordinates(*current_pixel_coordinates)

            end_coordinates = _to_windows_coordinates(x, y)
            print("In interpolate")
            coordinates = _interpolate_mouse_movement(
                start_windows_coordinates=start_coordinates,
                end_windows_coordinates=end_coordinates
            )
            print(coordinates)
        else:
            coordinates = [end_coordinates]

        for x, y in coordinates:
            extra = ctypes.c_ulong(0)
            ii_ = Input_I()
            ii_.mi = MouseInput(x, y, 0, (0x0001 | 0x8000), 0, ctypes.pointer(extra))
            x = Input(ctypes.c_ulong(0), ii_)
            ctypes.windll.user32.SendInput(1, ctypes.pointer(x), ctypes.sizeof(x))

            time.sleep(duration / len(coordinates))
    else:
        x = int(x)
        y = int(y)

        coordinates = _interpolate_mouse_movement(
            start_windows_coordinates=(0, 0),
            end_windows_coordinates=(x, y)
        )

        for x, y in coordinates:
            extra = ctypes.c_ulong(0)
            ii_ = Input_I()
            ii_.mi = MouseInput(x, y, 0, 0x0001, 0, ctypes.pointer(extra))
            x = Input(ctypes.c_ulong(0), ii_)
            ctypes.windll.user32.SendInput(1, ctypes.pointer(x), ctypes.sizeof(x))

            time.sleep(duration / len(coordinates))


def _to_windows_coordinates(x=0, y=0):
    display_width = win32api.GetSystemMetrics(0)
    display_height = win32api.GetSystemMetrics(1)

    windows_x = (x * 65535) // display_width
    windows_y = (y * 65535) // display_height

    return windows_x, windows_y

def _interpolate_mouse_movement(start_windows_coordinates, end_windows_coordinates, steps=20):
    x_coordinates = [start_windows_coordinates[0], end_windows_coordinates[0]]
    y_coordinates = [start_windows_coordinates[1], end_windows_coordinates[1]]

    if x_coordinates[0] == x_coordinates[1]:
        x_coordinates[1] += 1

    if y_coordinates[0] == y_coordinates[1]:
        y_coordinates[1] += 1

    interpolation_func = scipy.interpolate.interp1d(x_coordinates, y_coordinates)

    intermediate_x_coordinates = np.linspace(start_windows_coordinates[0], end_windows_coordinates[0], steps + 1)[1:]
    coordinates = list(map(lambda x: (int(round(x)), int(interpolation_func(x))), intermediate_x_coordinates))

Attach this code to yours and call it using move(x,y). This is the link to the full code https://github.com/SerpentAI/SerpentAI/blob/dev/serpent/input_controllers/native_win32_input_controller.py Also it might help to run python with Administrative privileges

Bandoh
  • 31
  • 3