Controlling Keyboard with Hand gestures

Question

I have some opencv code which counts the number of fingers the user is holding up to the camera. the program is fully functional and works as expected.

However, I am trying to connect the keyboard with my program. For example, if the user holds up a 1, then the "W" key should be pressed and if the user holds up a 2, then the "A" key should be pressed, and so on.

I decided to use app script for this.

if count+1 == 1:       
    app('System Events').keystroke('w')
if count+1 == 2:
    app('System Events').keystroke('a')
if count+1 == 3: 
    app('System Events').keystroke('s')
if count+1 == 4:
    app('System Events').keystroke('d')
if count+1 == 5:
    app('System Events').keystroke('c')

This code works pretty well, but there are two things I am unable to figure out how to do:

1. How can I hold the key until the user removes the gesture from the camera? For example, if the user holds up one finger, then the program should press "W" as long as that the one finger is up.

2. For some reason, whenever I run this code, it has the camera but after some time, my computer just freezes and I can't do anything other than restart the computer. I feel like this is something to do with the keyboard stuff, so I set up an if statement:

key = cv2.waitKey(10)
if k == ord('q'):
    sys.exit()

However, this doesn't seem to work.

Would really appreciate if someone could help. Thanks in advance.

Answer 1

This question should be helpful to you detect key press in python?

Instead of using something like .keystroke('c') you could also try to minic the behaviour of e.g. a gaming controller or a keyboard.

They use a bit-wise representation for buttons, e.g.

0 0 0 0 0 0 0 0   

where the first 0 is UP, second is DOWN, ... A,B,X,Y ... and so on.

So for a D-PAD plus four action buttons you have 8 bits, one byte.

This can be converted to a number between 0 and 255 to represent the state of each button, e.g.

1 0 1 0 0 0 0 1

could be UP,LEFT,Y pressed and is a certain integer number.

You can read about that everywhere, e.g. https://www3.ntu.edu.sg/home/ehchua/programming/java/datarepresentation.html

You could make your image processing make return that number and process it to detect what is pressed using ttps://wiki.python.org/moin/BitwiseOperators

Yeah, you might not have a "real" controller, but you have something that acts like a controller. Many hardware input devices have that representation on the lower level, close to the hardware.

And usually when you are setting up things like what you are up to you can create a virtual input device. Ok, you can go with just a bunch of if statements and pulling keys separately if you are doing a bachelor or master thesis and create throwaway code. But if this goes a more professional way you might want to set up something like virtual USB keyboard.

We did similar things using gesture control to act like a mouse. But its the same like e.g. a virtual keyboard for sign language.

If you just want to show that you can move some point in some window that you are updating in an if loop with Python - go ahead.

But if you are planning to interact with an operating system you need to code something to fit its interface for human input devices (e.g. USB HID). In your case a virtual USB keyboard or USB mouse could work fine.

These devices will also be seen by PyGame, so you can look at how a gaming controller is used in PyGame and mimic one with your camera code.