PyAutoGUI from Windows subsystem linux

Question

how can I execute mouse clicks in windows from Python code that is running in WSL?

I tried using PyAutoGUI, however, I get the following error:

...
File "/usr/lib/python3.6/os.py", line 669, in __getitem__
raise KeyError(key) from None
KeyError: 'DISPLAY'

Because I am using wsl, it decides that my platform is linux. If I hardcode it to windows, ctypes do have attribute win.dll in dc = ctypes.windll.user32.GetDC(0)

if sys.platform == 'win32':
    import ctypes

    if _PILLOW_INSTALLED:
        from PIL import ImageGrab

    # Makes this process aware of monitor scaling so the screenshots are correctly sized:
    try:
       ctypes.windll.user32.SetProcessDPIAware()
    except AttributeError:
        pass # Windows XP doesn't support this, so just do nothing.

    dc = ctypes.windll.user32.GetDC(0)

    class POINT(ctypes.Structure):
        _fields_ = [('x', ctypes.c_long),
                    ('y', ctypes.c_long)]

    def _winPosition():
        cursor = POINT()
        ctypes.windll.user32.GetCursorPos(ctypes.byref(cursor))
        return (cursor.x, cursor.y)
    position = _winPosition


    def _winScreenshot(filename=None):
        # TODO - Use the winapi to get a screenshot, and compare performance with ImageGrab.grab()
        # https://stackoverflow.com/a/3586280/1893164
        try:
            im = ImageGrab.grab()
            if filename is not None:
                im.save(filename)
        except NameError:
            raise ImportError('Pillow module must be installed to use screenshot functions on Windows.')
        return im
    screenshot = _winScreenshot

    def _winSize():
        return (ctypes.windll.user32.GetSystemMetrics(0), ctypes.windll.user32.GetSystemMetrics(1))
    size = _winSize

    def _winGetPixel(x, y):
        colorRef = ctypes.windll.gdi32.GetPixel(dc, x, y)  # A COLORREF value as 0x00bbggrr. See https://learn.microsoft.com/en-us/windows/win32/gdi/colorref
        red = colorRef % 256
        colorRef //= 256
        green = colorRef % 256
        colorRef //= 256
        blue = colorRef

        return (red, green, blue)
    getPixel = _winGetPixel


elif platform.system() == 'Linux':
    from Xlib.display import Display
    import errno

    scrotExists = False
    try:
            whichProc = subprocess.Popen(
                ['which', 'scrot'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
            scrotExists = whichProc.wait() == 0
    except OSError as ex:
        if ex.errno == errno.ENOENT:
            # if there is no "which" program to find scrot, then assume there
            # is no scrot.
            pass
        else:
            raise

    _display = Display(os.environ['DISPLAY'])

    def _linuxPosition():
        coord = _display.screen().root.query_pointer()._data
        return coord["root_x"], coord["root_y"]
    position = _linuxPosition

    def _linuxScreenshot(filename=None):
        if not scrotExists:
            raise NotImplementedError('"scrot" must be installed to use screenshot functions in Linux. Run: sudo apt-get install scrot')

        if filename is not None:
            tmpFilename = filename
        else:
            tmpFilename = '.screenshot%s.png' % (datetime.datetime.now().strftime('%Y-%m%d_%H-%M-%S-%f'))

        if scrotExists:
            subprocess.call(['scrot', '-z', tmpFilename])
            im = Image.open(tmpFilename)

            # force loading before unlinking, Image.open() is lazy
            im.load()

            if filename is None:
                os.unlink(tmpFilename)
            return im
        else:
            raise Exception('The scrot program must be installed to take a screenshot with PyScreeze on Linux. Run: sudo apt-get install scrot')
    screenshot = _linuxScreenshot

    def _linuxSize():
        return _display.screen().width_in_pixels, _display.screen().height_in_pixels
    size = _linuxSize

    def _linuxGetPixel(x, y):
        rgbValue = screenshot().getpixel((x, y))
        return rgbValue[0], rgbValue[1], rgbValue[2]
    getPixel = _linuxGetPixel

Does anyone know how to solve this problem?

I can't see the whole code but you could maybe try and hardcode the system as windows.. Really could do with more context here.. What have you tried?? — johnashu, May 12 '20 at 08:33
Isn't windows subsystem just a "bubble" inside Linux, can't you run the code in your Linux machine to automate things? Why do you need to run it inside the `WSL`? *(I'm by no means an expert on WSL, so I'm asking to see if I can figure this out)* — Torxed, May 12 '20 at 08:33
Updated the code. Tried hardcoding ctypes then do not have windll attribute. — Robert J Axe, May 12 '20 at 09:03

score 3 · Answer 1 · answered Aug 17 '21 at 19:05

Just to clarify... pyautogui will not work with WSL? Based on what I have read elsewhere this has to do with the lack of an X-server in the linux subsystem. Similar to how pyautogui will not work over a remote or headless terminal session.

So, try this with just plain old windows and it will work!

score -1 · Answer 2 · answered May 12 '20 at 08:40

-1

you could try adding this line AFTER importing pyautogui.

import pyautogui._pyautogui_win as platformModule

This should reassign / override the platformModule variable to the Windows version..

(In theory)

answered May 12 '20 at 08:40

johnashu

2,167
4
19
44

This doesn't work. – Blackgaurd Apr 26 '22 at 00:13

PyAutoGUI from Windows subsystem linux

2 Answers2