How can I control the keyboard and mouse with Python?

Question

How can I control the mouse and keyboard in Python?

The idea is to do the same as the Robot() class in Java. Be able to say: move the mouse from here to here, click there, write that whatever is on the screen.

For Windows there is win32api but I'm using mainly Linux.

For Linux there is Xlib but does it works for keyboard as well? (found only reference to the mouse)

Is there a cross-platform solution? (Linux, Windows and even OS X would be the great.)

Answer 1

I use dogtail (https://fedorahosted.org/dogtail/) to do such things, using this I have created an automated testing framework for my Linux(Ubuntu) app. That framework clicks buttons and types into text fields.

see the gedit example, https://fedorahosted.org/dogtail/browser/examples/gedit-test-utf8-procedural-api.py

So just use dogtail e.g

dogtail.rawinput.click(100, 100)

Answer 2

I can advise you PyAutoGUI, it allows to full control Mouse and Keyboard and get Screenshots and even you can locate images within the screen (like: where is the button?), very useful to automate clicks dynamically. It works for Windows, macOS, and Linux.

For example:

>>> import pyautogui
>>> screenWidth, screenHeight = pyautogui.size()
>>> pyautogui.moveTo(screenWidth / 2, screenHeight / 2)

Check out the Introduction page.

Answer 3

This totally works... on a Mac at least. This is for a click AND drag, etc.. but can be retrofitted accordingly.

#!/usr/bin/python
import sys
import time
from Quartz.CoreGraphics import * # imports all of the top-level symbols in the module

def mouseEvent(type, posx, posy):
    theEvent = CGEventCreateMouseEvent(None, type, (posx,posy), kCGMouseButtonLeft)
    CGEventPost(kCGHIDEventTap, theEvent)
def mousemove(posx,posy):
    mouseEvent(kCGEventMouseMoved, posx,posy);
def mouseclickdn(posx,posy):
    mouseEvent(kCGEventLeftMouseDown, posx,posy);
def mouseclickup(posx,posy):
    mouseEvent(kCGEventLeftMouseUp, posx,posy);
def mousedrag(posx,posy):
    mouseEvent(kCGEventLeftMouseDragged, posx,posy);

ourEvent = CGEventCreate(None);
currentpos=CGEventGetLocation(ourEvent); # Save current mouse position
mouseclickdn(60, 100);
mousedrag(60, 300);
mouseclickup(60, 300);
time.sleep(1);
mousemove(int(currentpos.x),int(currentpos.y)); # Restore mouse position

Answer 4

Here is an interessting Thread from Python Forum for you: Python Forum

Edit: There was also an interessting question on stackoverflow regarding mouse control...maybe it is a good starting point.. Mouse Control with Python

One of the Answers is refering to an Linux example...which heads you to an nice blog entry.

Answer 5

for the mouse, I've found pymouse which seems to work (I haven't fully tried it, a small hack needed for the click, cf the issues)

for the keyboard, I'm not sure Xlib can do the job. I'm still looking on how to write something but you can catch key event as explained here or in C here using Xlib (but I don't know C).

here is an example working on gnome only (not good enough yet)

In pymouse, they have a nice way to make it work on the 3 different platform but needs to make 3 code...

Answer 6

For console try ncurses or slang. In other situation try PyQt, PyGtk, TkInter.

ALL of this solution ARE cross-platform and work almost anywhere.

Answer 7

For linux there is Xlib but does it works for keyboard as well? (found only reference to the mouse)

Yes, it work for keyboard also.

Answer 8

A cross platform solution on linux, windows and mac is autopy. https://github.com/msanders/autopy/

It allows controlling mouse and keyboard, taking screenshots, and finding small bitmaps on larger bitmaps and should be very convenient if you want to automate some gui application you have no control on.