I have a Raspberry Pi Zero which i have turned it into a virtual keyboard and virtual mouse. I've done it using this amazing (thank you, SeongTae Jeong!) repo on github - rasp_vusb. To shortly summarize how it works:
- You connect the PI Zero to a different computer via an USB cable and connect it to the Internet with an ethernet cable.
After connecting it to the computer it creates 3 new devices in the Device Manager (Windows):
- Human Interface Devices
- Usb Input Device
- Keyboards
- HID Keyboard Device
- Mice and other pointing devices
- HID-compliant mouse (Absolute position + buttons)
- HID-compliant mouse (Relative position + wheel)
- Human Interface Devices
There is a "server" running on the PI zero (code found in the rasp_vusb_server on the repo), which accepts your commands and then sends them as a mouse commands or as a keyboard commands to the device you are connected via USB.
You can send commands from a different computer to the PI zero (the server) by connecting to it's IP and sending the proper data. You can use the already built in program in the repo i provided which is located in the map InputController
Now this works perfectly on Windows 10 build 1803 and build 1903. It also works on Mac but I haven't tested the keyboard function on a Mac. The weird problem occurs when the Pi Zero is connected to a Windows 10 (pro or home) build 1809.
The keyboard commands, relative mouse commands (Meaning: move the cursor +50 pixel to the right from the position you are on atm) and mouse button (clicking) commands are working fine, but the absolute mouse commands (Meaning: move the cursor at X = 50, Y = 50 on the screen) and wheel commands do not work - The cursor keeps snapping at the upper left corner (coordinates, X=0, Y=0).
I tried this on many different computers and Windows builds, different Monitors (some had touch screens, others didn't), different Pi zeros. The problem only occurs on Windows 10 build 1809. I tried searching on the web about similar problems but all the answers lead to some touchscreen problems, but as I said in my case the touch screen is not the problem, as it works on different builds.
(I'm not actually sure if this has anything to do with the problem, because this is the code/command we send to the PI zero and then the PI zero sends it to the connected device) I started digging in the code and i found that when sending an absolute mouse command you send a byte array with a length of 15. I will explain it with an example for better understanding:
Let's say you want to send the an absolute mouse command, that you want to move the cursor to X=50 and Y=100 on the screen. As it's stated in the readme file in the repo you have to make some calculation depending on your screen resolution. So let's say we have a 1920x1080 resolution. Then we send X=(50 * 32767 / 1920) = 853 and Y=(100 * 32767 / 1080) = 3033. Then you have to call a function byte[] ConvertToMouseCommands("853 3033"). It's found in the repo in the InputController/MouseDevice.cs or Demo/MouseDevice.cs. This function translates the numbers to a byte array like so: (EDIT*) I will put next to it the byte array that is created when sending a relative mouse command like (X=+50, Y=+50) for comparison.
[0] = 11 <- Packet length (15 - 4) [0] = 7 (11-4)
[1] = 0 [1] = 0
[2] = 0 [2] = 0
[3] = 0 [3] = 0
[4] = 7 <- 7 means absolute command [4] = 6 <- 6 means relative commands
[5] = 85 <- (xPos & 0xff) [5] = 0
[6] = 3 <- (xPos & 0xff00) >> 8) [6] = 50 <- xPos (Just xPos!) you can only send +-127
[7] = 217 <- (yPos & 0xff) [7] = 50 <- yPos (Just yPos!) you can only send +-127
[8] = 11 <- (yPos & 0xff00) >> 8) [8] = 0
[9] = 0 [9] = 0
[10] = 0 [10] = 0
[11] = 0
[12] = 0
[13] = 0
[14] = 0
These are the byte arrays we send to the PI zero. When the PI zero receives them, it uses two function (depending if it's relative mouse command or absolute mouse command) to send them further.
The two functions are void MouseDevice::SendRelative(int fd, char *mouseInput, int keyLen) and void MouseDevice::SendAbsolute(int fd, char *mouseInput, int keyLen). They can be found in the repo in the rasp_vusb_server/MouseDevice.cpp. I don't want to paste the code here because this post is already long enough, but essentially they send the X coordinate, the Y coordinate, if the buttons are clicked or if the wheel on the mouse has rotated - It seems pretty straight forward to me.
As i said, the code works perfectly on other Windows builds, it just fails on the build 1809. I honestly don't even know how to start debugging something like this. Did something changed in that build - Should i send the bytes somewhat different? Is there some magical solution like disabling a driver or something (wishful thinking)? Should i try even older Windows 10 builds to see if it works?
I would be very thankful for any information that you can provide. Thank you!
