Simulate Python keypresses for controlling a game

Question

I'm trying to control a game (my two test games are Half Life 2 and Minecraft) using my Kinect and Python. Everything works except for one thing. The game will respond to simulated mouse events and simulated mouse movement (mouse events are done via ctypes and mouse movement is done using pywin32). The problem however is that the games ignore simulated keypresses. Both of them will pick up the simulated keypresses in either the chat window (Minecraft) or the developer console (Half Life 2) but not while playing the actual game.

I've tried several ways of sending the keypresses:

import win32com.client as client
wsh = client.Dispatch('WScript.Shell')
wsh.AppActivate(gameName)
wsh.SendKeys(key)

and:

import win32api
win32api.keybd_event(keyHexCode, 0, 0)

and:

import ctypes
import time

SendInput = ctypes.windll.user32.SendInput

# C struct redefinitions 
PUL = ctypes.POINTER(ctypes.c_ulong)
class KeyBdInput(ctypes.Structure):
    _fields_ = [("wVk", ctypes.c_ushort),
                ("wScan", ctypes.c_ushort),
                ("dwFlags", ctypes.c_ulong),
                ("time", ctypes.c_ulong),
                ("dwExtraInfo", PUL)]

class HardwareInput(ctypes.Structure):
    _fields_ = [("uMsg", ctypes.c_ulong),
                ("wParamL", ctypes.c_short),
                ("wParamH", ctypes.c_ushort)]

class MouseInput(ctypes.Structure):
    _fields_ = [("dx", ctypes.c_long),
                ("dy", ctypes.c_long),
                ("mouseData", ctypes.c_ulong),
                ("dwFlags", ctypes.c_ulong),
                ("time",ctypes.c_ulong),
                ("dwExtraInfo", PUL)]

class Input_I(ctypes.Union):
    _fields_ = [("ki", KeyBdInput),
                 ("mi", MouseInput),
                 ("hi", HardwareInput)]

class Input(ctypes.Structure):
    _fields_ = [("type", ctypes.c_ulong),
                ("ii", Input_I)]

# Actuals Functions

def PressKey(hexKeyCode):

    extra = ctypes.c_ulong(0)
    ii_ = Input_I()
    ii_.ki = KeyBdInput( hexKeyCode, 0x48, 0, 0, ctypes.pointer(extra) )
    x = Input( ctypes.c_ulong(1), ii_ )
    ctypes.windll.user32.SendInput(1, ctypes.pointer(x), ctypes.sizeof(x))

def ReleaseKey(hexKeyCode):

    extra = ctypes.c_ulong(0)
    ii_ = Input_I()
    ii_.ki = KeyBdInput( hexKeyCode, 0x48, 0x0002, 0, ctypes.pointer(extra) )
    x = Input( ctypes.c_ulong(1), ii_ )
    ctypes.windll.user32.SendInput(1, ctypes.pointer(x), ctypes.sizeof(x))

I should point out the code in the last one isn't mine, it's off another question here on Stack Overflow.

Does anyone know why none of these work and what the correct way to do this is?

In lieu of a proper response, might I suggest checking the source of Minecraft to see if there is a difference in how keyboard events are gathered and handled between the chat and gameplay? I'm going to take a look at it myself when I get home. — Logan, Jan 23 '13 at 21:35
I'm going to have to retract my earlier statement about it not working on Minecraft actually, with the ctypes system it does. I'll double check Half Life 2 now... — user573949, Jan 23 '13 at 21:51
More experimentation has revealed that the following behaviour: Minecraft: mouse movement, keypresses, no mouse clicks. Blacklight Retribution: all. Tribes Ascend: all. Half Life 2: mouse movement, mouse clicks, no keypresses. Does anyone have any idea of the reason for these inconsistencies or how to fix them? — user573949, Jan 27 '13 at 23:43
Blacklight and Tribes are both Unreal Engine, so I would assume you would get full compatibility across all Unreal games. I can't find any information about how Source handles keyboard events and I haven't worked with it. Minecraft is "open source", you just need to deobfuscate the .jar and take a look and see how they handle mouse clicks. — Logan, Jan 28 '13 at 19:20
I tried doing that once in the past but the code appeared to have been stripped of all variable names rendering it near unreadable? — user573949, Jan 28 '13 at 22:31
I'll try it now, I had resigned myself to the fate of never solving this and had stopped checking this post you see. — user573949, Feb 24 '13 at 12:14
I have now same issue, but problem is, directinput didnt work for me as well (sendinput). I am using C#, but it would be very similar to python. I am thinking if the game doesnt have higher priority then operation system has and the game communicate with processor on its own, cause it seems like the operation system doesnt even know about some keyboard input :/.. — tomdelahaba, Apr 23 '14 at 13:13

score 49 · Answer 1 · answered May 05 '14 at 08:38

I just had the same problem trying to simulate keypresses in Half-Life 2. As Robin said, the solution is to use ScanCodes instead of VKs.

I edited your last code example such that it uses ScanCodes. I tried it with Half-Life 2 and it works just fine:

import ctypes
import time

SendInput = ctypes.windll.user32.SendInput

# C struct redefinitions 
PUL = ctypes.POINTER(ctypes.c_ulong)
class KeyBdInput(ctypes.Structure):
    _fields_ = [("wVk", ctypes.c_ushort),
                ("wScan", ctypes.c_ushort),
                ("dwFlags", ctypes.c_ulong),
                ("time", ctypes.c_ulong),
                ("dwExtraInfo", PUL)]

class HardwareInput(ctypes.Structure):
    _fields_ = [("uMsg", ctypes.c_ulong),
                ("wParamL", ctypes.c_short),
                ("wParamH", ctypes.c_ushort)]

class MouseInput(ctypes.Structure):
    _fields_ = [("dx", ctypes.c_long),
                ("dy", ctypes.c_long),
                ("mouseData", ctypes.c_ulong),
                ("dwFlags", ctypes.c_ulong),
                ("time",ctypes.c_ulong),
                ("dwExtraInfo", PUL)]

class Input_I(ctypes.Union):
    _fields_ = [("ki", KeyBdInput),
                 ("mi", MouseInput),
                 ("hi", HardwareInput)]

class Input(ctypes.Structure):
    _fields_ = [("type", ctypes.c_ulong),
                ("ii", Input_I)]

# Actuals Functions

def PressKey(hexKeyCode):
    extra = ctypes.c_ulong(0)
    ii_ = Input_I()
    ii_.ki = KeyBdInput( 0, hexKeyCode, 0x0008, 0, ctypes.pointer(extra) )
    x = Input( ctypes.c_ulong(1), ii_ )
    ctypes.windll.user32.SendInput(1, ctypes.pointer(x), ctypes.sizeof(x))

def ReleaseKey(hexKeyCode):
    extra = ctypes.c_ulong(0)
    ii_ = Input_I()
    ii_.ki = KeyBdInput( 0, hexKeyCode, 0x0008 | 0x0002, 0, ctypes.pointer(extra) )
    x = Input( ctypes.c_ulong(1), ii_ )
    ctypes.windll.user32.SendInput(1, ctypes.pointer(x), ctypes.sizeof(x))

# directx scan codes http://www.gamespp.com/directx/directInputKeyboardScanCodes.html
while (True):
    PressKey(0x11)
    time.sleep(1)
    ReleaseKey(0x11)
    time.sleep(1)

I know it's reallly old answer but it's worth to try. First than you soo much for this clear answer but i couldn't figure out how to customize it for mouse movement and click certain location for game. I'm appreciated for any help — sword1st, Mar 02 '19 at 00:19
I would also like to see an example with MouseInput if possible — jmoreno, Apr 28 '21 at 09:13

score 7 · Answer 2 · answered Feb 13 '13 at 00:00

It's likely that the game is using DirectInput devices.

So, the game is expecting DirectInput key presses. According to the last post of this forum thread, DirectInput responds to ScanCodes, not VKs. You can try sending DirectInput key presses using this tool. The dev also supplies the source and a detailed explanation.

If this works, you could just try sending appropriate ScanCodes instead of VKs (list of scancodes).

There's also an older project called DirectPython that allows you to interface with DirectX/DirectInput.

The download link for that tool is dead, do you know how to actually send DirectInput keypresses with DirectPython? It only appears to be set up to listen for them, not dispatch them. — user573949, Feb 24 '13 at 13:59

Simulate Python keypresses for controlling a game

2 Answers2

Linked