Find Frames in a video that matches an image

Question

I am trying to find frames that matches an image using opencv. I also want to find the timeframe at which the the image is found. The video is a masked video. The code so far:

 def occurence_counter(self):
        img = cv2.imread('ref_img.jpg', cv2.IMREAD_COLOR)
        # shrink
        img = cv2.resize(img, (10, 10))
        # convert to b&w
        img = color.rgb2gray(img)
        similarities = []
       
      result = self.parse_video(img, str(self.lineEdit.text()).strip(), 1, False)
        print result

def parse_video(self, image, video, n_matches, break_point=False, 
           verbose=False):

    similarities = [{'frame': 0, 'similarity': 0}]
    frame_count = 0

    cap = cv2.VideoCapture(video)
    while cap.isOpened():

        ret, frame = cap.read()

        if type(frame) == type(None):
            break

        # increment frame counter
        frame_count += 1

        # resize current video frame
        small_frame = cv2.resize(frame, (10, 10))
        # convert to greyscale
        small_frame_bw = color.rgb2gray(small_frame)

Answer 1

Finding the same frame is not so easy issue. There are many possible solutions.I will describe here the possible solutions in a very general way.

Template Matching

Template matching is algorithm that calculate the similarity of the corresponding pixels in the images. So if You are looking for very similar image (without rotation, translation, large intenisty changes) it is not so bad algorithm. It is not so fast for whole images. It is rather used to find the same fragment on several images or smaller image on bigger image, not to check similarity of two images. https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_template_matching/py_template_matching.html

For whole images it is easier to just simply subtract the images then use template matching. It is much faster. There must be an assumption that they are really similar to each other.

Histogram Comparision

You can use histogram comparision. It is the fastest way, but it is not accurate. Grass and apples are both green, but dissimilar to each other. It's usually better to use the HSV color space when it comes to color. https://docs.opencv.org/3.4.1/d8/dc8/tutorial_histogram_comparison.html

Feature matching

Algorithm is searching for simillar characteristic points on images. There are many algorithms to find features on images. They should be insensitive to scale change and rotation etc. But it depends on the feature extraction algoritm. https://docs.opencv.org/3.0-beta/doc/py_tutorials/py_feature2d/py_features_meaning/py_features_meaning.html#features-meaning

Other alogithms

Other algoritms are PSNR or SSIM. I have never used it, but there are used to calculate similarity for original and blur image or for similarity of whole video sequence. https://docs.opencv.org/3.4.2/d5/dc4/tutorial_video_input_psnr_ssim.html

You can too try to compare hashes of images. It is very interessting algorithm (for me), but it is not well documented. https://www.pyimagesearch.com/2017/11/27/image-hashing-opencv-python/

Feature matching are the most-used algorithm for this type of task. The reason for that is feature matching alogrithms can detect similar fragments of images when images are taken from a different angle, in different conditions, or only partially overlap. Structure From Motion algorithms are often using feature matching. https://hub.packtpub.com/exploring-structure-motion-using-opencv/ The solution to the problem always depends on the data we have. So there is no one answer.

Answer 2

If I am not mistaken what you want to do is called Template Matching, you can find opencv tutorial of that feature here. Also this thread might be useful for you, especially @Sam answer, which beyond Template Matching describe also Comparing Histograms and Feature Matching.