How to find rectangle of difference between two images

Question

I have two images the same size. What is the best way to find the rectangle in which they differ. Obviously I could go through the image 4 times in different directions, but i'm wondering if there's an easier way.

Example:

Answer 1

If you want a single rectangle, use int.MaxValue for the threshold.

var diff = new ImageDiffUtil(filename1, filename2);
var diffRectangles = diff.GetDiffRectangles(int.MaxValue);

If you want multiple rectangles, use a smaller threshold.

var diff = new ImageDiffUtil(filename1, filename2);
var diffRectangles = diff.GetDiffRectangles(8);

ImageDiffUtil.cs

using System;
using System.Collections.Generic;
using System.Drawing;
using System.Linq;

namespace diff_images
{
    public class ImageDiffUtil
    {
        Bitmap image1;
        Bitmap image2;

        public ImageDiffUtil(string filename1, string filename2)
        {
            image1 = Image.FromFile(filename1) as Bitmap;
            image2 = Image.FromFile(filename2) as Bitmap;
        }

        public IList<Point> GetDiffPixels()
        {
            var widthRange = Enumerable.Range(0, image1.Width);
            var heightRange = Enumerable.Range(0, image1.Height);

            var result = widthRange
                            .SelectMany(x => heightRange, (x, y) => new Point(x, y))
                            .Select(point => new
                            {
                                Point = point,
                                Pixel1 = image1.GetPixel(point.X, point.Y),
                                Pixel2 = image2.GetPixel(point.X, point.Y)
                            })
                            .Where(pair => pair.Pixel1 != pair.Pixel2)
                            .Select(pair => pair.Point)
                            .ToList();

            return result;
        }

        public IEnumerable<Rectangle> GetDiffRectangles(double distanceThreshold)
        {
            var result = new List<Rectangle>();

            var differentPixels = GetDiffPixels();

            while (differentPixels.Count > 0)
            {
                var cluster = new List<Point>()
                {
                    differentPixels[0]
                };
                differentPixels.RemoveAt(0);

                while (true)
                {
                    var left = cluster.Min(p => p.X);
                    var right = cluster.Max(p => p.X);
                    var top = cluster.Min(p => p.Y);
                    var bottom = cluster.Max(p => p.Y);
                    var width = Math.Max(right - left, 1);
                    var height = Math.Max(bottom - top, 1);
                    var clusterBox = new Rectangle(left, top, width, height);

                    var proximal = differentPixels
                                        .Where(point => GetDistance(clusterBox, point) <= distanceThreshold)
                                        .ToList();
                    proximal.ForEach(point => differentPixels.Remove(point));

                    if (proximal.Count == 0)
                    {
                        result.Add(clusterBox);
                        break;
                    }
                    else
                    {
                        cluster.AddRange(proximal);
                    }
                };
            }

            return result;
        }

        static double GetDistance(Rectangle rect, Point p)
        {
            var dx = Math.Max(rect.Left - p.X, 0);
            dx = Math.Max(dx, p.X - rect.Right);

            var dy = Math.Max(rect.Top - p.Y, 0);
            dy = Math.Max(dy, p.Y - rect.Bottom);
            return Math.Sqrt(dx * dx + dy * dy);
        }
    }
}

Form1.cs

using System.Drawing;
using System.Linq;
using System.Windows.Forms;

namespace diff_images
{
    public partial class Form1 : Form
    {
        public Form1()
        {
            InitializeComponent();

            var filename1 = @"Gelatin1.PNG";
            var filename2 = @"Gelatin2.PNG";

            var diff = new ImageDiffUtil(filename1, filename2);
            var diffRectangles = diff.GetDiffRectangles(8);

            var img3 = Image.FromFile(filename2);
            Pen redPen = new Pen(Color.Red, 1);
            var padding = 3;
            using (var graphics = Graphics.FromImage(img3))
            {
                diffRectangles
                    .ToList()
                    .ForEach(rect =>
                    {
                        var largerRect = new Rectangle(rect.X - padding, rect.Y - padding, rect.Width + padding * 2, rect.Height + padding * 2);
                        graphics.DrawRectangle(redPen, largerRect);
                    });
            }

            var pb1 = new PictureBox()
            {
                Image = Image.FromFile(filename1),
                Left = 8,
                Top = 8,
                SizeMode = PictureBoxSizeMode.AutoSize
            };

            var pb2 = new PictureBox()
            {
                Image = Image.FromFile(filename2),
                Left = pb1.Left + pb1.Width + 16,
                Top = 8,
                SizeMode = PictureBoxSizeMode.AutoSize
            };

            var pb3 = new PictureBox()
            {
                Image = img3,
                Left = pb2.Left + pb2.Width + 16,
                Top = 8,
                SizeMode = PictureBoxSizeMode.AutoSize
            };

            Controls.Add(pb1);
            Controls.Add(pb2);
            Controls.Add(pb3);
        }
    }
}

Answer 2

A naive approach would be to start at the origin, and work line by line, column by column. Compare each pixel, keeping note of the topmost, leftmost, rightmost, and bottommost, from which you can calculate your rectangle. There will be cases where this single pass approach would be faster (i.e. where there is a very small differing area)

Answer 3

Image processing like this is expensive, there are a lot of bits to look at. In real applications, you almost always need to filter the image to get rid of artifacts induced by imperfect image captures.

A common library used for this kind of bit whacking is OpenCV, it takes advantage of dedicated CPU instructions available to make this fast. There are several .NET wrappers available for it, Emgu is one of them.

Answer 4

I don't think there is an easier way.

In fact doing this will just be a (very) few lines of code, so unless you find a library that does that for you directly you won't find a shorter way.

Answer 5

Idea:

Consider an image as a 2D Array with each Array element as a pixel of the image. Hence, I would say Image Differencing is nothing but 2D Array Differencing.

Idea is to just scan through the array elements width-wise and find the place where there is a difference in pixel values. If example [x, y] co-ordinates of both 2D Array are different then our rectangle finding logic starts. Later on the rectangles would be used to patch the last updated Frame Buffer.

We need to scan through the boundaries of the rectangles for differences and if any difference is found in the boundary of rectangle, then the boundary will be increased width-wise or height-wise depending upon the type of scan made.

Consider I scanned width-wise of 2D Array and I found a location where there exist a co-ordinate which is different in both the 2D Arrays, I will create a rectangle with the starting position as [x-1, y-1] and with the width and height as 2 and 2 respectively. Please note that width and height refers to the number of pixels.

eg: Rect Info: X = 20 Y = 35 W = 26 H = 23

i.e width of the rectangle starts from co-ordinate [20, 35] -> [20, 35 + 26 - 1]. Maybe when you find the code you may be able to understand it better.

Also there are possibilities that there are smaller rectangles inside a bigger rectangle you have found, thus we need to remove the smaller rectangles from our reference because they mean nothing to us except that they occupu my precious space !!

The above logic would be helpful in the case of VNC Server Implementation where there would be a need of rectangles that denotes differences in the image that is currently taken. Those rectangles could be sent in the network to the VNC Client which can patch the rectangles in the local copy of Frame Buffer it possesses thereby displaying it on the VNC Client Display Board.

P.S.:

I will be attaching the code in which I implemented my own algorithm. I would request viewers to comment for any mistakes or performance tuning. I would also request viewers to comment about any better algorithm that would make life simpler.

Code:

Class Rect:

public class Rect {
    public int x; // Array Index
    public int y; // Array Index
    public int w; // Number of hops along the Horizontal
    public int h; // Number of hops along the Vertical

    @Override
    public boolean equals(Object obj) {
        Rect rect = (Rect) obj;
        if(rect.x == this.x && rect.y == this.y && rect.w == this.w && rect.h == this.h) {
            return true;
        }
        return false;
    }
}

Class Image Difference:

import java.awt.image.BufferedImage;
import java.io.File;
import java.io.IOException;
import java.util.LinkedList;

import javax.imageio.ImageIO;

public class ImageDifference {
 long start = 0, end = 0;

 public LinkedList<Rect> differenceImage(int[][] baseFrame, int[][] screenShot, int xOffset, int yOffset, int width, int height) {
  // Code starts here
  int xRover = 0;
  int yRover = 0;
  int index = 0;
  int limit = 0;
  int rover = 0;

  boolean isRectChanged = false;
  boolean shouldSkip = false;

  LinkedList<Rect> rectangles = new LinkedList<Rect>();
  Rect rect = null;

  start = System.nanoTime();

  // xRover - Rovers over the height of 2D Array
  // yRover - Rovers over the width of 2D Array
  int verticalLimit = xOffset + height;
  int horizontalLimit = yOffset + width;

  for(xRover = xOffset; xRover < verticalLimit; xRover += 1) {
   for(yRover = yOffset; yRover < horizontalLimit; yRover += 1) {

    if(baseFrame[xRover][yRover] != screenShot[xRover][yRover]) {
     // Skip over the already processed Rectangles
     for(Rect itrRect : rectangles) {
      if(( (xRover < itrRect.x + itrRect.h) && (xRover >= itrRect.x) ) && ( (yRover < itrRect.y + itrRect.w) && (yRover >= itrRect.y) )) {
       shouldSkip = true;
       yRover = itrRect.y + itrRect.w - 1;
       break;
      } // End if(( (xRover < itrRect.x + itrRect.h) && (xRover >= itrRect.x) ) && ( (yRover < itrRect.y + itrRect.w) && (yRover >= itrRect.y) ))
     } // End for(Rect itrRect : rectangles)

     if(shouldSkip) {
      shouldSkip = false;
      // Need to come out of the if condition as below that is why "continue" has been provided
      // if(( (xRover <= (itrRect.x + itrRect.h)) && (xRover >= itrRect.x) ) && ( (yRover <= (itrRect.y + itrRect.w)) && (yRover >= itrRect.y) ))
      continue;
     } // End if(shouldSkip)

     rect = new Rect();

     rect.x = ((xRover - 1) < xOffset) ? xOffset : (xRover - 1);
     rect.y = ((yRover - 1) < yOffset) ? yOffset : (yRover - 1);
     rect.w = 2;
     rect.h = 2;

     /* Boolean variable used to re-scan the currently found rectangle
      for any change due to previous scanning of boundaries */
     isRectChanged = true;

     while(isRectChanged) {
      isRectChanged = false;
      index = 0;


      /*      I      */
      /* Scanning of left-side boundary of rectangle */
      index = rect.x;
      limit = rect.x + rect.h;
      while(index < limit && rect.y != yOffset) {
       if(baseFrame[index][rect.y] != screenShot[index][rect.y]) {        
        isRectChanged = true;
        rect.y = rect.y - 1;
        rect.w = rect.w + 1;
        index = rect.x;
        continue;
       } // End if(baseFrame[index][rect.y] != screenShot[index][rect.y])

       index = index + 1;;
      } // End while(index < limit && rect.y != yOffset)


      /*      II      */
      /* Scanning of bottom boundary of rectangle */
      index = rect.y;
      limit = rect.y + rect.w;
      while( (index < limit) && (rect.x + rect.h != verticalLimit) ) {
       rover = rect.x + rect.h - 1;
       if(baseFrame[rover][index] != screenShot[rover][index]) {
        isRectChanged = true;
        rect.h = rect.h + 1;        
        index = rect.y;
        continue;
       } // End if(baseFrame[rover][index] != screenShot[rover][index])

       index = index + 1;
      } // End while( (index < limit) && (rect.x + rect.h != verticalLimit) )


      /*      III      */
      /* Scanning of right-side boundary of rectangle */
      index = rect.x;
      limit = rect.x + rect.h;
      while( (index < limit) && (rect.y + rect.w != horizontalLimit) ) {
       rover = rect.y + rect.w - 1;
       if(baseFrame[index][rover] != screenShot[index][rover]) {
        isRectChanged = true;
        rect.w = rect.w + 1;
        index = rect.x;
        continue;
       } // End if(baseFrame[index][rover] != screenShot[index][rover])

       index = index + 1;
      } // End while( (index < limit) && (rect.y + rect.w != horizontalLimit) )

     } // while(isRectChanged)


     // Remove those rectangles that come inside "rect" rectangle.
     int idx = 0;
     while(idx < rectangles.size()) {
      Rect r = rectangles.get(idx);
      if( ( (rect.x <= r.x) && (rect.x + rect.h >= r.x + r.h) ) && ( (rect.y <= r.y) && (rect.y + rect.w >= r.y + r.w) ) ) {
       rectangles.remove(r);
      } else {
       idx += 1;
      }  // End if( ( (rect.x <= r.x) && (rect.x + rect.h >= r.x + r.h) ) && ( (rect.y <= r.y) && (rect.y + rect.w >= r.y + r.w) ) ) 
     } // End while(idx < rectangles.size())

     // Giving a head start to the yRover when a rectangle is found
     rectangles.addFirst(rect);

     yRover = rect.y + rect.w - 1;
     rect = null;

    } // End if(baseFrame[xRover][yRover] != screenShot[xRover][yRover])
   } // End for(yRover = yOffset; yRover < horizontalLimit; yRover += 1)
  } // End for(xRover = xOffset; xRover < verticalLimit; xRover += 1)

  end = System.nanoTime();    
  return rectangles;
 }

 public static void main(String[] args) throws IOException { 
  LinkedList<Rect> rectangles = null;

  // Buffering the Base image and Screen Shot Image
  BufferedImage screenShotImg = ImageIO.read(new File("screenShotImg.png"));
  BufferedImage baseImg   = ImageIO.read(new File("baseImg.png"));

  int width  = baseImg.getWidth();
  int height = baseImg.getHeight();
  int xOffset = 0;
  int yOffset = 0;
  int length = baseImg.getWidth() * baseImg.getHeight();

  // Creating 2 Two Dimensional Arrays for Image Processing
  int[][] baseFrame = new int[height][width];
  int[][] screenShot = new int[height][width];

  // Creating 2 Single Dimensional Arrays to retrieve the Pixel Values  
  int[] baseImgPix   = new int[length];
  int[] screenShotImgPix  = new int[length];

  // Reading the Pixels from the Buffered Image
  baseImg.getRGB(0, 0, baseImg.getWidth(), baseImg.getHeight(), baseImgPix, 0, baseImg.getWidth());
  screenShotImg.getRGB(0, 0, screenShotImg.getWidth(), screenShotImg.getHeight(), screenShotImgPix, 0, screenShotImg.getWidth());

  // Transporting the Single Dimensional Arrays to Two Dimensional Array
  long start = System.nanoTime();

  for(int row = 0; row < height; row++) {
   System.arraycopy(baseImgPix, (row * width), baseFrame[row], 0, width);
   System.arraycopy(screenShotImgPix, (row * width), screenShot[row], 0, width);
  }

  long end = System.nanoTime();
  System.out.println("Array Copy : " + ((double)(end - start) / 1000000));

  // Finding Differences between the Base Image and ScreenShot Image
  ImageDifference imDiff = new ImageDifference();
  rectangles = imDiff.differenceImage(baseFrame, screenShot, xOffset, yOffset, width, height);

  // Displaying the rectangles found
  int index = 0;
  for(Rect rect : rectangles) {
   System.out.println("\nRect info : " + (++index));
   System.out.println("X : " + rect.x);
   System.out.println("Y : " + rect.y);
   System.out.println("W : " + rect.w);
   System.out.println("H : " + rect.h);

   // Creating Bounding Box
   for(int i = rect.y; i < rect.y + rect.w; i++) {    
    screenShotImgPix[ ( rect.x               * width) + i ] = 0xFFFF0000;
    screenShotImgPix[ ((rect.x + rect.h - 1) * width) + i ] = 0xFFFF0000;
   }

   for(int j = rect.x; j < rect.x + rect.h; j++) {
    screenShotImgPix[ (j * width) + rect.y                ] = 0xFFFF0000;
    screenShotImgPix[ (j * width) + (rect.y + rect.w - 1) ] = 0xFFFF0000;
   }

  }

  // Creating the Resultant Image
  screenShotImg.setRGB(0, 0, width, height, screenShotImgPix, 0, width);
  ImageIO.write(screenShotImg, "PNG", new File("result.png"));

  double d = ((double)(imDiff.end - imDiff.start) / 1000000);
  System.out.println("\nTotal Time : " + d + " ms" + "  Array Copy : " + ((double)(end - start) / 1000000) + " ms");

 }
}

Description:

There would be a function named

public LinkedList<Rect> differenceImage(int[][] baseFrame, int[][] screenShot, int width, int height)

which does the job of finding differences in the images and return a linkedlist of objects. The objects are nothing but the rectangles.

There is main function which does the job of testing the algorithm.

There are 2 sample images passed into the code in main function, they are nothing but the "baseFrame" and "screenShot" thereby creating the resultant image named "result".

I don't possess the desired reputation to post the resultant image which would be very interesting.

There is a blog which would provide the output Image Difference

Answer 6

So here comes the easy way if you know how to use Lockbit :)

        Bitmap originalBMP = new Bitmap(pictureBox1.ImageLocation);
        Bitmap changedBMP = new Bitmap(pictureBox2.ImageLocation);

        int width = Math.Min(originalBMP.Width, changedBMP.Width),
            height = Math.Min(originalBMP.Height, changedBMP.Height),

            xMin = int.MaxValue,
            xMax = int.MinValue,

            yMin = int.MaxValue,
            yMax = int.MinValue;

        var originalLock = originalBMP.LockBits(new Rectangle(0, 0, width, height), ImageLockMode.ReadWrite, originalBMP.PixelFormat);
        var changedLock = changedBMP.LockBits(new Rectangle(0, 0, width, height), ImageLockMode.ReadWrite, changedBMP.PixelFormat);

        for (int y = 0; y < height; y++)
        {
            for (int x = 0; x < width; x++)
            {
                //generate the address of the colour pixel
                int pixelIdxOrg = y * originalLock.Stride + (x * 4);
                int pixelIdxCh = y * changedLock.Stride + (x * 4);


                if (( Marshal.ReadByte(originalLock.Scan0, pixelIdxOrg + 2)!= Marshal.ReadByte(changedLock.Scan0, pixelIdxCh + 2))
                    || (Marshal.ReadByte(originalLock.Scan0, pixelIdxOrg + 1) != Marshal.ReadByte(changedLock.Scan0, pixelIdxCh + 1))
                    || (Marshal.ReadByte(originalLock.Scan0, pixelIdxOrg) != Marshal.ReadByte(changedLock.Scan0, pixelIdxCh))
                    )
                {
                    xMin = Math.Min(xMin, x);
                    xMax = Math.Max(xMax, x);

                    yMin = Math.Min(yMin, y);
                    yMax = Math.Max(yMax, y);
                }
            }
        }

        originalBMP.UnlockBits(originalLock);
        changedBMP.UnlockBits(changedLock);

        var result = changedBMP.Clone(new Rectangle(xMin, yMin, xMax - xMin, yMax - yMin), changedBMP.PixelFormat);

        pictureBox3.Image = result;

disclaim it looks like your 2 pictures contains more differences than we can see with the naked eye so the result will be wider than you expect but you can add a tolerance so it wil fit even if the rest isn't 100% identical

to speed things up you will maybe able to us Parallel.For but do it only for the outer loop

Answer 7

I don't think there can be anything better than exhaustively searching from each side in turn for the first point of difference in that direction. Unless, that is, you know a fact that in some way constrains the set of points of difference.