How can I make this while True loop run faster and work properly? It detects the presence of 3 unwanted items on screen, when gone, alert triggered

Question

I have a while True loop searching for the presence three different images. If any of the three are detected, 'space' is pressed, to search for another application. If an application is found without any of the three images, the loop is broken, and an alert is triggered notifying the user that there's work to be done.

The problem I'm having is, I think pyautogui detects that the image is gone immediately after enter is pressed, and the next application is loading in my browser, as it loads, nothing is there, it's a blank page. It only takes a fraction of a second to load the next application. When the next one loads in, often times, 1 of the 3 images are present but the loop already broke prematurely.

Here's the loop:

while True:
    pyautogui.press("space")
    img = pyautogui.locateOnScreen("ffd1.PNG", confidence=0.7)
    img = pyautogui.locateOnScreen("ffd2.PNG", confidence=0.7)
    img = pyautogui.locateOnScreen("ffd3.PNG", confidence=0.7)
    if img == None:
        ctypes.windll.user32.MessageBoxW(0, "There's work to be done!!", "There's work to be done!!", 0x1000)
        break

Is there a better way to do this? I want it to press 'space' quickly, and search for the lack of my images at the same time. I tried adding a time.sleep(.5) in there, and it didn't help at all. The loop still breaks between loading screens. :|

somebody pointed out that I'm only searching for one image. I fixed that here... It's still very slow.

while True:
    img = pyautogui.locateOnScreen("ffd1.PNG", confidence=0.7)
    if img == None and img2 == None and img3 == None:
        ctypes.windll.user32.MessageBoxW(0, "There's work to be done!!", "There's work to be done!!", 0x1000)
        break
    img2 = pyautogui.locateOnScreen("ffd2.PNG", confidence=0.7)
    if img == None and img2 == None and img3 == None:
        ctypes.windll.user32.MessageBoxW(0, "There's work to be done!!", "There's work to be done!!", 0x1000)
        break
    img3 = pyautogui.locateOnScreen("ffd3.PNG", confidence=0.7)
    if img == None and img2 == None and img3 == None:
        ctypes.windll.user32.MessageBoxW(0, "There's work to be done!!", "There's work to be done!!", 0x1000)
        break
    pyautogui.press("space")

tried rewriting using opencv and the thing just crashes all the time for no reason

img = cv2.imread('ffd1.PNG')
image_detected = False

while True:
    screenshot = pyautogui.screenshot()
    screenshot = np.array(screenshot)
    screenshot = cv2.cvtColor(screenshot, cv2.COLOR_RGB2BGR)
    
    res = cv2.matchTemplate(screenshot, img, cv2.TM_CCOEFF_NORMED)
    threshold = 0.8
    loc = np.where(res >= threshold)
    
    if np.any(loc) and not image_detected:
        print("Image detected on the screen! Pressed space key.")
        image_detected = True
        pyautogui.press('space')
        time.sleep(1)
        
    elif not np.any(loc) and image_detected:
        print("Image is no longer detected on the screen. Exiting...")
        break

your first two images are always ignored as the third one sets the value that is tested — JonSG, Feb 18 '23 at 22:38
Editing now. I fixed that part but it still doesn't work. :/ — bigbird342d, Feb 19 '23 at 00:17
ok it is working now but incredibly slow... so slow... is there any way to make this faster? :| — bigbird342d, Feb 19 '23 at 00:30
After the first image check if it is not none you might not have to check the others — JonSG, Feb 19 '23 at 00:46
it may be a tiny bit faster now. still incredibly slow. maybe openCV would do better here — bigbird342d, Feb 19 '23 at 12:35
check out https://stackoverflow.com/questions/56222990/template-matching-from-screen-capture — JonSG, Feb 19 '23 at 15:06

score 0 · Accepted Answer · answered Feb 19 '23 at 15:12

pyautogui is notoriously slow at screen capture and openCV and mss is/are alternatives to explore. That said, you are still often looping several times to set all values for img in order to do the test. Your opencv code is the closest to what I might expect.

For just pyautogui, try this:

while any(map(lambda f: pyautogui.locateOnScreen(f, confidence=0.7), ["ffd1.PNG", "ffd2.PNG", "ffd3.PNG"])):
    print("Image detected on the screen! Pressed space key.")
    pyautogui.press('space')
    time.sleep(1)

ctypes.windll.user32.MessageBoxW(0, "There's work to be done!!", "There's work to be done!!", 0x1000)

I'm not anticipating a huge improvement particularly if you have a larger monitor.

this is pretty good actually. I changed the time.sleep to (.3) and there's definitely improvement. I wasn't able to get this to work with openCV unfortunately. My code just keeps crashing. Thank you for your help with this! — bigbird342d, Feb 19 '23 at 17:01

How can I make this while True loop run faster and work properly? It detects the presence of 3 unwanted items on screen, when gone, alert triggered

1 Answers1