How can I search an area for pixel change?

Question

This is the code I am using to detect if a pixel (in this case pixel 510,510) turns to a certain color.

import PIL.ImageGrab
import mouse


while True:
    rgb = PIL.ImageGrab.grab(bbox = None)
    rgb2=(253, 146, 134)
    print (rgb.getpixel((510, 510)))
    if (rgb.getpixel((510, 510))) == rgb2:
        mouse.click()

I want to be able to search an area of my screen for any pixel that changes to a specified color, not just an individual pixel. How might I do that? I want to keep this running as fast as possible. I know most areas searched on an image or video would be a rectangle, but could it be a triangle to cut down on pixels searched? If not, the next sentences are irrelevant. How so? Would it work if I give the coords of each point in the triangle?

You could just grab the one pixel from the screen (from your code example) that you are interested by providing the `bbox` argument right at the pixel, e.g. `ImageGrab.grab(bbox=(510, 510, 511, 511,)).getpixel((0, 0))`. Or for an area just provide a bigger bounding box (covering the interested area) and iterate through `PIL.ImageGrab.grab().getdata()` and [use a loop to check if that one specific color exist](https://stackoverflow.com/questions/61810728/). Alternative, [find the difference between the current and previous screengrab](https://stackoverflow.com/questions/59461213/). — metatoaster, Jun 19 '22 at 03:54
@metatoaster So if I understand correctly, I'm using a bounding box and then using the other code you linked to detect if there is the color I'm looking for. Another question. Would I be able to set multiple (say 3) bounding boxes and depending on which box finds the color, have a different output for each? Not sure if that makes sense. — PaperWarrior105, Jun 19 '22 at 21:29
Yes, you could generalize this loop by putting them inside a function and yield some value, which would produce a generator. This generator function could take a `bbox`, and you can create multiple of them, which then all of them could iterated together and action base on what each of these generators return at the time inside the new loop. Doing it this way would synchronize the checks to a single concurrent sloop, or you could put them into separate threads which would have their own handlers should you want the actions fire asynchronously. — metatoaster, Jun 20 '22 at 02:13

Mark Setchell · Answer 1 · 2022-06-21T19:00:48.543

Make a black rectangular image just big enough to contain the shape you want to detect. Use np.zeros((h,w,3), np.uint8) to create it. It will be zero everywhere.
Draw the shape you want to detect in the black rectangle with colour=[1,1,1]. You now have an image that is 1 where you are interested in the pixels and 0 elsewhere. Do these first 2 steps outside your main loop.
Inside your loop, grab an area of screen the same size as your mask from steps 1 and 2. Multiply your image by the mask and all pixels you are not interested in will become zero. Test if your colour exists using np.where() or cv2.countNonZero(np.all(im==soughtColour, axis=-1))

As an alternative to drawing with colour=[1,1,1] at the second step, draw with colour=[255,255,255] and then in the third step use cv2.bitwise_and() instead of multiplying.

How can I search an area for pixel change?

1 Answers1