How to get the MSE of each pixel in two images of the same dimension

Question

I have two images that will be identical in dimensions.

The images are numpy arrays and I am itterating the pixels and aquiring the r g b of each pixel like this:

for i in range(len(img1)):
    for j in range(len(img1[0])):
        pimg1 = img1[i][j]
        pimg2 = img2[i][j]

        r1 = pimg1[0]
        r2 = pimg2[0]
        g1 = pimg1[1]
        g2 = pimg2[1]
        b1 = pimg1[2]
        b2 = pimg2[2]

I then acquire the MSE between the two pixels like:

mse = math.sqrt(((r1 - r2) ** 2) + ((g1 - g2) ** 2) + ((b1 - b2) ** 2))

The problem is this is ridiculously slow. Is there a more efficient way to do this?

The final goal

I am looking to make all pixels that have a certain "thresholded" similarity, between the two images, black. And all pixels that have a larger difference the pixel of img2.

if mse > threshold:
    new_img[i][j] = pimg2
else:
    new_img[i][j] = [0, 0, 0] # black pixel

background image

entered image

I am capturing images like:

for frame in cam.camera.capture_continuous(raw, format="rgb", use_video_port=True):
    img = frame.array
    cv2.imwrite("image.png", img)

I am getting the images like:

dir = 'images/compare/'
bg = cv2.imread(dir+'bg.png')
img = cv2.imread(dir+'in.png')

Divakar · Accepted Answer · 2017-11-15T11:08:53.977

3

Simply use np.linalg.norm on the differences -

mse = np.linalg.norm(img1-img2,axis=2)

Faster one with np.einsum -

d = (img1-img2).astype(float)
mse = np.sqrt(np.einsum('...i,...i->...',d,d))

Runtime test -

In [46]: np.random.seed(0)
    ...: m,n = 1024,1024
    ...: img1 = np.random.randint(0,255,(m,n,3)).astype(np.uint8)
    ...: img2 = np.random.randint(0,255,(m,n,3)).astype(np.uint8)

In [47]: %timeit np.linalg.norm(img1-img2,axis=2)
10 loops, best of 3: 26.6 ms per loop

In [49]: %%timeit
    ...: d = (img1-img2).astype(float)
    ...: mse = np.sqrt(np.einsum('...i,...i->...',d,d))
100 loops, best of 3: 13 ms per loop

To create an output array that is set to black for pixels that have MSE values lesser than a certain threshold mse_thresh and select from img2 otherwise, here are the additional codes -

mask = mse >= mse_thresh
out = np.where(mask[...,None], img2, 0)

Stitching everything together - Using einsum to compute squared MSE values and comparing against the squared MSE threshold for major improvement and assigning back the output into img2 -

d = (img1-img2).astype(float)
mse = np.einsum('...i,...i->...',d,d)
img2[mse < mse_thresh**2] = 0

edited Nov 15 '17 at 11:08

answered Nov 15 '17 at 10:41

Divakar

218,885
19
262
358

The problem is that this still affects a fps of ~0.1fps to ~0.4fps :( – maxisme Nov 15 '17 at 11:03
@Maximilian Time to change the algorithm it seems :) – Divakar Nov 15 '17 at 11:04
@Maximilian Or would you be okay with writing the result into `img2`? – Divakar Nov 15 '17 at 11:05
@Maximilian Check out the last part (3 lines code). – Divakar Nov 15 '17 at 11:14
Please see the edit in my question where I have added the two test images. the difference extraction returns https://i.imgur.com/4O7ZX3s.png – maxisme Nov 15 '17 at 15:19
I think this is the technique https://stackoverflow.com/questions/27035672/cv-extract-differences-between-two-images – maxisme Nov 15 '17 at 15:22
@Maximilian Use loseless images (png, etc.)? See related discussion - https://stackoverflow.com/questions/47265000/image-deduction-with-pillow-and-numpy?noredirect=1#comment81481451_47265000 – Divakar Nov 15 '17 at 16:03
currently trying to pre process with blur to no avail – maxisme Nov 15 '17 at 16:26
Please see updated question for how I acquiring the PNGs – maxisme Nov 15 '17 at 16:35
@Maximilian Can you do the capture `cam.camera.capture_continuous` in png? Capturing in lossy jpegs and saving as PNG won't do it, as discussed in the earlier link. – Divakar Nov 15 '17 at 16:38
format="rgb" -> "Write the raw image data to a file in 24-bit RGB format" – maxisme Nov 15 '17 at 16:39
@Maximilian Upload the input images as png and share links? – Divakar Nov 15 '17 at 16:44
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/159080/discussion-between-divakar-and-maximilian). – Divakar Nov 15 '17 at 16:45
The RGB values in the image are uint8, so won't it cause problems with overflow doing `np.linalg.norm(img1-img2,axis=2)` ?I would have thought you need to take `img1.astype(float)` etc don't you? – Mark Setchell Sep 03 '19 at 17:10

How to get the MSE of each pixel in two images of the same dimension

The final goal

background image

entered image

1 Answers1