CV - Extract differences between two images

Question

I am currently working on an intrusion system based on video surveillance. In order to complete this task, I take a snapshot of the background of my scene (assume it's totally clean, no people or moving objects). Then, I compare the frame I get from the (static) video camera and look for the differences. I have to be able to check any differences, not only human shape or whatever, so I cannot specific feature extraction.

Typically, I have:

I am using OpenCV, so to compare I basically do:

cv::Mat bg_frame;
cv::Mat cam_frame;
cv::Mat motion;

cv::absdiff(bg_frame, cam_frame, motion);
cv::threshold(motion, motion, 80, 255, cv::THRESH_BINARY);
cv::erode(motion, motion, cv::getStructuringElement(cv::MORPH_RECT, cv::Size(3,3)));

Here is the result:

As you can see, the arm is stripped (due to color differential conflict I guess) and this is sadly not what I want.

I thought about add the use of cv::Canny() in order to detect the edges and fill the missing part of the arm, but sadly (once again), it only solves the problem in few situation not most of them.

Is there any algorithm or technique I could use to obtain an accurate difference report?

PS: Sorry for the images. Due to my newly subscription, I do not have enough reputation.

EDIT I use grayscale image in here, but I am open to any solution.

Answer 1

One problem in your code is cv::threshold which only uses 1 channel images. Finding the pixelwise "difference" between two images in only grayscale often leads to unintuitive results.

Since your provided images are a bit translated or the camera wasnt stationary, I've manipulated your background image to add some foreground:

background image:

foreground image:

code:

    cv::Mat diffImage;
    cv::absdiff(backgroundImage, currentImage, diffImage);

    cv::Mat foregroundMask = cv::Mat::zeros(diffImage.rows, diffImage.cols, CV_8UC1);

    float threshold = 30.0f;
    float dist;

    for(int j=0; j<diffImage.rows; ++j)
        for(int i=0; i<diffImage.cols; ++i)
        {
            cv::Vec3b pix = diffImage.at<cv::Vec3b>(j,i);

            dist = (pix[0]*pix[0] + pix[1]*pix[1] + pix[2]*pix[2]);
            dist = sqrt(dist);

            if(dist>threshold)
            {
                foregroundMask.at<unsigned char>(j,i) = 255;
            }
        }

giving this result:

with this difference image:

in general it is hard to compute a complete foreground/background segmentation from pixel-wise difference interpretations.

You will probably have to add postprocessing stuff to get a real segmentation, where you start from your foreground mask. Not sure whether there are any stable universal solutions yet.

As berak mentioned, in practice it won't be enough to use a single background image, so you will have to compute/manage your background image over time. There are plenty of papers covering this topic and afaik no stable universal solution yet.

here are some more tests. I converted to HSV color space: cv::cvtColor(backgroundImage, HSVbackgroundImagebg, CV_BGR2HSV); cv::cvtColor(currentImage, HSV_currentImage, CV_BGR2HSV); and performed the same operations in this space, leading to this result:

after adding some noise to the input:

I get this result:

so maybe the threshold is a bit too high. I still encourage you to have a look at HSV color space too, but you might have to reinterpret the "difference image" and rescale each channel to combine their difference values.

Answer 2

I use Python, this is my result:

The code:

# 2017.12.22 15:48:03 CST
# 2017.12.22 16:00:14 CST
import cv2
import numpy as np

img1 = cv2.imread("img1.png")
img2 = cv2.imread("img2.png")
diff = cv2.absdiff(img1, img2)
mask = cv2.cvtColor(diff, cv2.COLOR_BGR2GRAY)

th = 1
imask =  mask>th

canvas = np.zeros_like(img2, np.uint8)
canvas[imask] = img2[imask]

cv2.imwrite("result.png", canvas)

---

Update, here is C++ code:

//! 2017.12.22 17:05:18 CST
//! 2017.12.22 17:22:32 CST

#include <opencv2/opencv.hpp>
#include <iostream>
using namespace std;
using namespace cv;
int main() {

    Mat img1 = imread("img3_1.png");
    Mat img2 = imread("img3_2.png");

    // calc the difference
    Mat diff;
    absdiff(img1, img2, diff);

    // Get the mask if difference greater than th
    int th = 10;  // 0
    Mat mask(img1.size(), CV_8UC1);
    for(int j=0; j<diff.rows; ++j) {
        for(int i=0; i<diff.cols; ++i){
            cv::Vec3b pix = diff.at<cv::Vec3b>(j,i);
            int val = (pix[0] + pix[1] + pix[2]);
            if(val>th){
                mask.at<unsigned char>(j,i) = 255;
            }
        }
    }

    // get the foreground
    Mat res;
    bitwise_and(img2, img2, res, mask);

    // display
    imshow("res", res);
    waitKey();
    return 0;
}

---

Similar answers:

1. CV - Extract differences between two images

2. While finding a difference between 2 pictures OpenCV difference is bigger than it is supposed to be

Answer 3

Another technique to obtain the exact pixel differences between two images is to use the Structural Similarity Index (SSIM) first introduced in the paper Image Quality Assessment: From Error Visibility to Structural Similarity. This method can be used to determine if two images are identical and/or showcase differences due to tiny image discrepancies. SSIM is already implemented in the scikit-image library for image processing as skimage.metrics.structural_similarity()

The structural_similarity() function returns a score and a difference image, diff. The score represents the mean structural similarity index between the two input images and can fall between the range [-1,1] with values closer to one representing higher similarity. But since you're only interested in where the two images differ, the diff image is what we'll focus on. Specifically, the diff image contains the actual image differences with darker regions having more disparity. Larger areas of disparity are highlighted in black while smaller differences are in gray.

---

Using these two input images

We get this result

The SSIM score after comparing the two images show that they are very similar

Image Similarity: 95.8648%

from skimage.metrics import structural_similarity
import cv2

# Load images as grayscale
image1 = cv2.imread('1.png', 0)
image2 = cv2.imread('2.png', 0)

# Compute SSIM between the two images
(score, diff) = structural_similarity(image1, image2, full=True)

# The diff image contains the actual image differences between the two images
# and is represented as a floating point data type in the range [0,1] 
# so we must convert the array to 8-bit unsigned integers in the range
# [0,255] image1 we can use it with OpenCV
diff = (diff * 255).astype("uint8")
print("Image Similarity: {:.4f}%".format(score * 100))

cv2.imshow('diff', diff)
cv2.waitKey()

Answer 4

This is well-known classic computer vision problem called background subtraction. There are many approaches which can be used to solve this problem, most of them are already implemented, so I think you should first take a look at multiple existing algorithms, here is opensource implementation of most of them: https://github.com/andrewssobral/bgslibrary (I personally found SUBSENSE giving best results, but its deadly slow)