Key echo in Python in separate thread doesn't display first key stroke

Question

I would try to post a minimal working example, but unfortunately this problem just requires a lot of pieces so I have stripped it down best I can.

First of all, I'm using a simple script that simulates pressing keys through a function call. This is tweaked from here.

import ctypes

SendInput = ctypes.windll.user32.SendInput

PUL = ctypes.POINTER(ctypes.c_ulong)

class KeyBdInput(ctypes.Structure):
    _fields_ = [("wVk", ctypes.c_ushort),
                ("wScan", ctypes.c_ushort),
                ("dwFlags", ctypes.c_ulong),
                ("time", ctypes.c_ulong),
                ("dwExtraInfo", PUL)]

class HardwareInput(ctypes.Structure):
    _fields_ = [("uMsg", ctypes.c_ulong),
                ("wParamL", ctypes.c_short),
                ("wParamH", ctypes.c_ushort)]

class MouseInput(ctypes.Structure):
    _fields_ = [("dx", ctypes.c_long),
                ("dy", ctypes.c_long),
                ("mouseData", ctypes.c_ulong),
                ("dwFlags", ctypes.c_ulong),
                ("time",ctypes.c_ulong),
                ("dwExtraInfo", PUL)]

class Input_I(ctypes.Union):
    _fields_ = [("ki", KeyBdInput),
                 ("mi", MouseInput),
                 ("hi", HardwareInput)]

class Input(ctypes.Structure):
    _fields_ = [("type", ctypes.c_ulong),
                ("ii", Input_I)]

def getKeyCode(unicodeKey):
    k = unicodeKey
    curKeyCode = 0
    if k == "up": curKeyCode = 0x26
    elif k == "down": curKeyCode = 0x28
    elif k == "left": curKeyCode = 0x25
    elif k == "right": curKeyCode = 0x27
    elif k == "home": curKeyCode = 0x24
    elif k == "end": curKeyCode = 0x23
    elif k == "insert": curKeyCode = 0x2D
    elif k == "pgup": curKeyCode = 0x21
    elif k == "pgdn": curKeyCode = 0x22
    elif k == "delete": curKeyCode = 0x2E
    elif k == "\n": curKeyCode = 0x0D

    if curKeyCode == 0:
        return 0, int(unicodeKey.encode("hex"), 16), 0x0004
    else:
        return curKeyCode, 0, 0

def PressKey(unicodeKey):
    key, unikey, uniflag = getKeyCode(unicodeKey)

    extra = ctypes.c_ulong(0)
    ii_ = Input_I()
    ii_.ki = KeyBdInput( key, unikey, uniflag, 0, ctypes.pointer(extra) )
    x = Input( ctypes.c_ulong(1), ii_ )
    ctypes.windll.user32.SendInput(1, ctypes.pointer(x), ctypes.sizeof(x))

def ReleaseKey(unicodeKey):
    key, unikey, uniflag = getKeyCode(unicodeKey)
    extra = ctypes.c_ulong(0)
    ii_ = Input_I()
    ii_.ki = KeyBdInput( key, unikey, uniflag + 0x0002, 0, ctypes.pointer(extra) )
    x = Input( ctypes.c_ulong(1), ii_ )
    ctypes.windll.user32.SendInput(1, ctypes.pointer(x), ctypes.sizeof(x))

I stored this in a file named keyPress.py.

Using this, I wanted to make a simple program that could detect what the user was typing while they were typing it in the python shell. The idea was that I would use msvcrt.getch() to get the key pressed, then the script above to make it seem like it was still pressed (and "echo" the key press in a sense")

Here is the code:

import keyPress
import msvcrt        
import threading

def getKey():

    k = msvcrt.getch()

    # Escaped Key: 224 is on the keyboard, 0 is on the numpad
    if int(k.encode("hex"), 16) == 224 or int(k.encode("hex"), 16) == 0:
        k = msvcrt.getch()
        if k == "H": k = "up"
        elif k == "P": k = "down"
        elif k == "K": k = "left"
        elif k == "M": k = "right"
        elif k == "G": k = "home"
        elif k == "O": k = "end"
        elif k == "R": k = "insert"
        elif k == "I": k = "pgup"
        elif k == "Q": k = "pgdn"
        elif k == "S": k = "delete"

    # Fix weird linebreak
    if k == "\r":
        k = "\n"

    return k


def actualGetKeys():
    while True:
        k = getKey()
        keyPress.PressKey(k)
        keyPress.ReleaseKey(k)

def getKeys():
    p = threading.Thread(target=actualGetKeys)
    p.daemon = True
    p.start()

I stored this in a file named keyGet.py.

This is all working very well, except that whenever the user presses enter, the first key isn't displayed on the screen. The console still knows that you typed it, it just doesn't show up there. Something like this:

What is happening? I've tried many many things and I can't seem to get this behavior to change.

I am now able to get this essentially working, as in it can capture key input asynchronously while a script is running, and execute with the text of each command you type into a command prompt (so you could, say, store these to an array). The only problem I am running into is something like this:

I know this is due to essentially having to have a robot retype their input after they type it, I'm just wondering if there is a way to do this that prevents that input from actually being displayed when the robot types it, so it acts just like the user would expect.

The CRT's console I/O functions such as `getch` don't use the same lock as stdio functions such as `fgets`. So when `getch` sets the console to raw mode it interferes with Python's `fgets` read. — Eryk Sun, Feb 08 '16 at 12:00
Even if you use a lock to keep them from interfering with each other, `getch` is only called for the first character of the line. The `fgets` cooked read (i.e. with command-line editing, F7 history menu, etc) consumes the rest of the line. I did the lock test using ctypes, with a combination of `PyOS_InputHook` (one-time setup) and hooking `PyOS_ReadlineFunctionPointer`. I can post that if you want, but the end result isn't useful for anything. These two ways of reading from the console just don't play well together. — Eryk Sun, Feb 08 '16 at 12:05
If you can elaborate on what you're trying to do, there may be another approach. For example, hooking `PyOS_ReadlineFunctionPointer` gives complete control over `input`, `raw_input`, and interactive shell input. Or if you just want to monitor input, your worker thread can use ctypes to call [`PeekConsoleInput`](https://msdn.microsoft.com/en-us/library/ms684344) to read from the input buffer without consuming it. — Eryk Sun, Feb 08 '16 at 22:08
All I would really like to do is have a thread running in the background that monitors what input is occurring and does things based on that, while still letting the user input and use the shell as normal. It sounds like PeekConsoleInput is probably the way to go then, thanks. I implemented a thing that calls PeekConsoleInput, but if it's in a separate thread it doesn't seem to be capturing anything. What is the recommended way to do this? — Phylliida, Feb 09 '16 at 02:35
Actually seems it does detect console input! It just only seems to do so if the console is locked up doing something else (say, running a very large for loop and then spamming pressing keys while it's going). This is a huge start though, thank you so much! Is there a way to retrieve these as the user types them in as well? (so not when the console is busy doing other things) — Phylliida, Feb 09 '16 at 02:44
Actually, it probably shouldn't use another thread, because then you're in a race with the shell's `fgets` (implemented via `ReadFile` in the CRT) to peek the input key events before `ReadFile` consumes them. I'd try assigning a ctypes callback to `PyOS_InputHook`, which gets called before `fgets`. Then peek the input events up to a carriage return before returning. Then `fgets` can read the line that you peeked. Ctrl+C might be tricky. I'll look into it. — Eryk Sun, Feb 09 '16 at 03:52
You're right! I was able to attach a callback to PyOS_InputHook and pull the key data per line typed, and then since it was blocking I simply outputted to the screen while they were typing so it looked identical to them actually typing things, and then once they pressed enter I moved the cursor back and let the events actually go through (I was just polling them) and then for them it was as if everything was as normal =) These were really the two things I wanted, so yes, thank you again! I'd like to clean up the code a little more then post it tomorrow. Let me know about the ctrl-c thing. — Phylliida, Feb 09 '16 at 05:46
The problem, as you noticed, is that it lacks the console's built-in command-line editing (character echoing, cursor positioning, backspace, del, history, including the F7 menu, or input aliases). In hindsight I think it isn't worth it to reproduce all of those features in code unless you really need the keys as they're typed (see pyreadline for example). It will probably be simpler to call `ReadConsoleW` in the hook function, to get a line of input the normal way. Then call `WriteConsoleInputW` to write the line back to the input buffer. — Eryk Sun, Feb 09 '16 at 08:28
Okay, that makes sense. I was trying that, and I got it working great, except that whenever you type something and press enter, it displays it again on the line below and then prints out. This works but is kinda annoying. Unfortunately [ANSI Terminal Control Escape Sequences](http://www.termsys.demon.co.uk/vtansi.htm) don't work in command prompt either, is there a way to address this? Specifically what I mean is you get something like [inserted in the question above] — Phylliida, Feb 12 '16 at 04:10
It's probably echoing the input to the screen. You can disable the echo, and enable it just when you're getting input. Get the current mode and set no echo using `kernel32.GetConsoleMode(hin, ctypes.byref(mode));` `noecho = mode.value & ~ENABLE_ECHO_INPUT;` `kernel32.SetConsoleMode(hin, noecho)`. — Eryk Sun, Feb 12 '16 at 23:32
As to ANSI terminal controls, there's ANSICON and ConEmu, both of which hook the client-side console API to implement ANSI graphics and/or xterm emulation. — Eryk Sun, Feb 12 '16 at 23:36
So this is slightly off-topic but how do you know all of this? I searched a ton and would have never been able to find most of this without your help. — Phylliida, Feb 13 '16 at 05:30
The console API hasn't changed much since it was released with NT 3.1 in '93. The implementation and hosting have changed. A console window used to be hosted on a thread in the Windows server process, csrss.exe, but in Windows 7 it was moved to instances of conhost.exe. It used to communicate with client processes (e.g. python.exe) using NT's local procedure calls, but in Windows 8 it switched to using the condrv device driver instead, so console handles are now kernel File handles, and the API was ported to I/O services (e.g. `NtWriteFile`, `NtDeviceIoControlFile`) instead of LPC. — Eryk Sun, Feb 13 '16 at 09:49

Phylliida · Accepted Answer · 2016-02-13T05:38:58.110

Here is the resulting code, basically written by eryksun's comments because somehow he knows all.

This is called readcmd.py

# Some if this is from http://nullege.com/codes/show/src@e@i@einstein-HEAD@Python25Einstein@Lib@subprocess.py/380/win32api.GetStdHandle
# and
# http://nullege.com/codes/show/src@v@i@VistA-HEAD@Python@Pexpect@winpexpect.py/901/win32console.GetStdHandle.PeekConsoleInput

from ctypes import *
import time
import threading

from win32api import STD_INPUT_HANDLE, STD_OUTPUT_HANDLE

from win32console import GetStdHandle, KEY_EVENT, ENABLE_WINDOW_INPUT, ENABLE_MOUSE_INPUT, ENABLE_ECHO_INPUT, ENABLE_LINE_INPUT, ENABLE_PROCESSED_INPUT

import keyPress


class CaptureLines():
    def __init__(self):
        self.stopLock = threading.Lock()

        self.isCapturingInputLines = False

        self.inputLinesHookCallback = CFUNCTYPE(c_int)(self.inputLinesHook)
        self.pyosInputHookPointer = c_void_p.in_dll(pythonapi, "PyOS_InputHook")
        self.originalPyOsInputHookPointerValue = self.pyosInputHookPointer.value

        self.readHandle = GetStdHandle(STD_INPUT_HANDLE)
        self.readHandle.SetConsoleMode(ENABLE_LINE_INPUT|ENABLE_ECHO_INPUT|ENABLE_PROCESSED_INPUT)

    def inputLinesHook(self):

        self.readHandle.SetConsoleMode(ENABLE_LINE_INPUT|ENABLE_ECHO_INPUT|ENABLE_PROCESSED_INPUT)
        inputChars = self.readHandle.ReadConsole(10000000)
        self.readHandle.SetConsoleMode(ENABLE_LINE_INPUT|ENABLE_PROCESSED_INPUT)

        if inputChars == "\r\n":
            keyPress.KeyPress("\n")
            return 0

        inputChars = inputChars[:-2]

        inputChars += "\n"

        for c in inputChars:
            keyPress.KeyPress(c)

        self.inputCallback(inputChars)

        return 0


    def startCapture(self, inputCallback):
        self.stopLock.acquire()

        try:
            if self.isCapturingInputLines:
                raise Exception("Already capturing keystrokes")

            self.isCapturingInputLines = True
            self.inputCallback = inputCallback

            self.pyosInputHookPointer.value = cast(self.inputLinesHookCallback, c_void_p).value
        except Exception as e:
            self.stopLock.release()
            raise

        self.stopLock.release()

    def stopCapture(self):
        self.stopLock.acquire()

        try:
            if not self.isCapturingInputLines:
                raise Exception("Keystrokes already aren't being captured")

            self.readHandle.SetConsoleMode(ENABLE_LINE_INPUT|ENABLE_ECHO_INPUT|ENABLE_PROCESSED_INPUT)

            self.isCapturingInputLines = False
            self.pyosInputHookPointer.value = self.originalPyOsInputHookPointerValue

        except Exception as e:
            self.stopLock.release()
            raise

        self.stopLock.release()

And here is keyPress.py

# Modified from http://stackoverflow.com/a/13615802/2924421

import ctypes
from ctypes import wintypes
import time

user32 = ctypes.WinDLL('user32', use_last_error=True)

INPUT_MOUSE    = 0
INPUT_KEYBOARD = 1
INPUT_HARDWARE = 2

KEYEVENTF_EXTENDEDKEY = 0x0001
KEYEVENTF_KEYUP       = 0x0002
KEYEVENTF_UNICODE     = 0x0004
KEYEVENTF_SCANCODE    = 0x0008

MAPVK_VK_TO_VSC = 0

# C struct definitions
wintypes.ULONG_PTR = wintypes.WPARAM

SendInput = ctypes.windll.user32.SendInput

PUL = ctypes.POINTER(ctypes.c_ulong)

class KEYBDINPUT(ctypes.Structure):
    _fields_ = (("wVk",         wintypes.WORD),
                ("wScan",       wintypes.WORD),
                ("dwFlags",     wintypes.DWORD),
                ("time",        wintypes.DWORD),
                ("dwExtraInfo", wintypes.ULONG_PTR))

class MOUSEINPUT(ctypes.Structure):
    _fields_ = (("dx",          wintypes.LONG),
                ("dy",          wintypes.LONG),
                ("mouseData",   wintypes.DWORD),
                ("dwFlags",     wintypes.DWORD),
                ("time",        wintypes.DWORD),
                ("dwExtraInfo", wintypes.ULONG_PTR))

class HARDWAREINPUT(ctypes.Structure):
    _fields_ = (("uMsg",    wintypes.DWORD),
                ("wParamL", wintypes.WORD),
                ("wParamH", wintypes.WORD))

class INPUT(ctypes.Structure):
    class _INPUT(ctypes.Union):
        _fields_ = (("ki", KEYBDINPUT),
                    ("mi", MOUSEINPUT),
                    ("hi", HARDWAREINPUT))
    _anonymous_ = ("_input",)
    _fields_ = (("type",   wintypes.DWORD),
                ("_input", _INPUT))

LPINPUT = ctypes.POINTER(INPUT)

def _check_count(result, func, args):
    if result == 0:
        raise ctypes.WinError(ctypes.get_last_error())
    return args

user32.SendInput.errcheck = _check_count
user32.SendInput.argtypes = (wintypes.UINT, # nInputs
                             LPINPUT,       # pInputs
                             ctypes.c_int)  # cbSize

def KeyDown(unicodeKey):
    key, unikey, uniflag = GetKeyCode(unicodeKey)
    x = INPUT( type=INPUT_KEYBOARD, ki= KEYBDINPUT( key, unikey, uniflag, 0))
    user32.SendInput(1, ctypes.byref(x), ctypes.sizeof(x))

def KeyUp(unicodeKey):
    key, unikey, uniflag = GetKeyCode(unicodeKey)
    extra = ctypes.c_ulong(0)
    x = INPUT( type=INPUT_KEYBOARD, ki= KEYBDINPUT( key, unikey, uniflag | KEYEVENTF_KEYUP, 0))
    user32.SendInput(1, ctypes.byref(x), ctypes.sizeof(x))

def KeyPress(unicodeKey):
    time.sleep(0.0001)
    KeyDown(unicodeKey)
    time.sleep(0.0001)
    KeyUp(unicodeKey)
    time.sleep(0.0001)


def GetKeyCode(unicodeKey):
    k = unicodeKey
    curKeyCode = 0
    if k == "up": curKeyCode = 0x26
    elif k == "down": curKeyCode = 0x28
    elif k == "left": curKeyCode = 0x25
    elif k == "right": curKeyCode = 0x27
    elif k == "home": curKeyCode = 0x24
    elif k == "end": curKeyCode = 0x23
    elif k == "insert": curKeyCode = 0x2D
    elif k == "pgup": curKeyCode = 0x21
    elif k == "pgdn": curKeyCode = 0x22
    elif k == "delete": curKeyCode = 0x2E
    elif k == "\n": curKeyCode = 0x0D

    if curKeyCode == 0:
        return 0, int(unicodeKey.encode("hex"), 16), KEYEVENTF_UNICODE
    else:
        return curKeyCode, 0, 0

I see you decided to stick with the `SendInput` code instead of `WriteConsoleInput`. FYI, lucasg has a [another version](http://stackoverflow.com/a/13615802/205580) of the key press code that I edited a couple of weeks ago. See the edit comment for a brief explanation of the changes I made. — Eryk Sun, Feb 13 '16 at 01:45
Ah cool, thanks. Yes for some reason it seemed that WriteConsoleInput wasn't able to send a newline which was kinda needed, so I went with this instead. It's possible there was a way around that but this was already working so I figured it would be fine. — Phylliida, Feb 13 '16 at 04:50

Key echo in Python in separate thread doesn't display first key stroke

1 Answers1

Linked