Get incremental update between two images

Question

Scenario: every 10 seconds I want to take a screenshot and upload it to a server. Of course, to make it more efficient, I should just take the difference from the last screenshot instead of the whole screen. I thought there might be an easy way to do this in python but I'm not finding any solutions.

Of course, I've found lots of answers like this one: CV - Extract differences between two images

but that solution doesn't really help me because I don't want the difference of both images, I just want the updated pixels and their most recent values.

Does anyone know of a good solution for this? Here's what I have so far:

def save_screen(path):
    import pyautogui
    pyautogui.screenshot().convert('L').save(path)

def get_incremental(image, image2):
    # ImageChops.difference from PIL is not sufficient
    # must I use numpy or something?
    pass

def upload_incremental(image):
    # upload to server using requests
    pass

def main(path):
    import time
    save_screen(path + '1')
    time.sleep(10)
    save_screen(path + '2')
    from PIL import Image
    image1 = Image.open(path + 1)
    image2 = Image.open(path + 2)
    upload_incremental(get_incremental(image1, image2))

do you think I'll have to use numpy to get the incremental updates between images? Algorithms Python - Difference Between Two Images

any other ideas? thanks!

numpy should be useful to make some calculus on these images, i.e., upload_incremental(image1-image2) after casting to ndarray. — Peter Julian, Jun 01 '21 at 22:15
You might be able to use scipy's sparse matrix (only stores non-zero elements), but that will only work if you have 0 noise. Essentially, you are asking for a poor man's video encoder (@ 0.1 FPS) and to then stream the result. You might be able to get some mileage out of the imageio+ffmpeg combo, but that would break the Request/Response communication pattern you seem to be going for here. — FirefoxMetzger, Jun 02 '21 at 04:29
Maybe you could provide say 3 representative images so we can see the screen sizes and sort of changes you are dealing with... — Mark Setchell, Jun 02 '21 at 06:35
@MarkSetchell its basically a screenshot every 10 seconds, so thats it. You can imagine that most of the time, when using a computer most of the screen stays exactly the same - unless you're watching a movie or playing a videogame. So as I'm typing this, it would have taken several screenshots and the only area of the image that would have changed would be this textbox on my screen. That is why there should be big bandwidth savings by only sending the updated pixels - or a bounded rectangle of pixels where some of them have changed. — MetaStack, Jun 02 '21 at 15:04

MetaStack · Accepted Answer · 2021-06-02T18:31:33.967

this is not the ideal solution but sufficient for now. get the largest inner rectangle of changes and return that:

import numpy as np
from PIL import Image, ImageChops

def crop_to_largest_changed_area(
    before: Image.Image,
    after: Image.Image
) -> 'tuple((left, top), Image.Image)':

    def black_out_clock(arr, yx, mode='L'):
        ''' assumes clock is at bottom right '''
        black = 0 if mode in ('L', '1') else (0, 0, 0)  # assume RGB
        arr[yx[0]-50:yx[0], yx[1]-120:yx[1]] = black
        return arr

    arr = np.array(ImageChops.difference(before, after))
    bbox = Image.fromarray(
        black_out_clock(
            arr=arr,
            yx=arr.shape,
            mode=after.mode)).getbbox()
    return (bbox[0:2], after.crop(bbox))

you could also take the individual pixels that changed from the array instead, but then you'd have to compress it yourself and I don't want to deal with all that, I'd rather grab more than I need to and leverage the png compression on it.

Get incremental update between two images

1 Answers1