Comparing rotated images

Question

I'm looking for a way to compare images that are almost identical (95%+ similarity) but that could be rotated around a central axis.

I'm new to the whole computer graphics/vision field, and not really sure if there is a ready built tool or system for it, or even if this is the right stack exchange to ask on.

Currently, I'm investigating using C# to generate a bitmap histogram, then running it through Panda for analysis and to ideally group the images based on the similarity of histograms, though I've heard something like OpenCv + tesseract may be a viable alternative.

Answer 1

You need to compute the affine transformation matrix between your two images in order the get the information regarding scaling, translation and rotation.

How this matrix looks and how can you get the rotation difference?

From this answer:

I have used the following Java code (using OpenCV 3.2) which computes the scaling, translation and rotation difference between two Mat images. I hope you will find it useful.

static void calculateDifferences(Mat img1, Mat img2){

        // Initialization
        FeatureDetector detector = FeatureDetector.create(FeatureDetector.ORB);
        DescriptorExtractor descriptor = DescriptorExtractor.create(DescriptorExtractor.ORB);
        DescriptorMatcher matcher = DescriptorMatcher.create(DescriptorMatcher.BRUTEFORCE_HAMMING);

        // First image objects
        Mat img1_descriptors = new Mat();
        MatOfKeyPoint img1_keypoints_mat = new MatOfKeyPoint();

        // Detect KeyPoints for first image
        detector.detect(img1, img1_keypoints_mat);
        descriptor.compute(img1, img1_keypoints_mat, img1_descriptors);

        // Second image objects
        Mat img2_descriptors = new Mat();
        MatOfKeyPoint img2_keypoints_mat = new MatOfKeyPoint();

        // Detect KeyPoints for second image
        detector.detect(img2, img2_keypoints_mat);
        descriptor.compute(img2, img2_keypoints_mat, img2_descriptors);

        // Match KeyPoints
        MatOfDMatch matOfDMatch = new MatOfDMatch();
        matcher.match(img1_descriptors, img2_descriptors, matOfDMatch);

        // Filtering the matches
        List<DMatch> dMatchList = matOfDMatch.toList();
        Double max_dist = 0.0;
        Double min_dist = 100.0;

        for(int i = 0; i < img1_descriptors.rows(); i++){
            Double dist = (double) dMatchList.get(i).distance;
            if(dist < min_dist) min_dist = dist;
            if(dist > max_dist) max_dist = dist;
        }
        LinkedList<DMatch> good_matches = new LinkedList<>();
        for(int i = 0; i < img1_descriptors.rows(); i++){
            if(dMatchList.get(i).distance < 3*min_dist){
                good_matches.addLast(dMatchList.get(i));
            }
        }

        // Converting to MatOfPoint2f format
        LinkedList<Point> img1_points_list = new LinkedList<>();
        LinkedList<Point> img2_points_list = new LinkedList<>();

        List<KeyPoint> img1_keyPoints_list = img1_keypoints_mat.toList();
        List<KeyPoint> img2_keyPoints_list = img2_keypoints_mat.toList();

        int limit = good_matches.size();
        for(int i = 0; i < limit; i++){
            img1_points_list.addLast(img1_keyPoints_list.get(good_matches.get(i).queryIdx).pt);
            img2_points_list.addLast(img2_keyPoints_list.get(good_matches.get(i).trainIdx).pt);
        }

        MatOfPoint2f img1_point2f_mat = new MatOfPoint2f();
        img1_point2f_mat.fromList(img1_points_list);

        MatOfPoint2f img2_point2f_mat = new MatOfPoint2f();
        img2_point2f_mat.fromList(img2_points_list);

        // Computing the affine transform matrix
        Mat result = Video.estimateRigidTransform(img1_point2f_mat, img2_point2f_mat, true);
        printMat(result); // Printing the optimal affine transformation 2x3 array

        // The following variables correspond to the estimateRigidTransform result as shown here: https://stackoverflow.com/a/29511091/5165833
        double a = result.get(0,0)[0];
        double b = result.get(0,1)[0];
        double d = result.get(1,1)[0];
        double c = result.get(1,0)[0];
        double tx = result.get(0,2)[0];
        double ty = result.get(1,2)[0];

        // Solving for scale,translation and rotation as shown in the link above
        double scale_x = Math.signum(a) * Math.sqrt( (a*a) + (b*b) ); // Axis x scale difference
        double scale_y = Math.signum(d) * Math.sqrt( (c*c) + (d*d) ); // Axis y scale difference
        double translation = ty; // The translation difference
        double rotation_angle = Math.atan2(c,d); // Rotation difference

        // Printing results
        println("Scale_x diff: " + scale_x);
        println("Scale_y diff: " + scale_y);
        println("Translation diff: " + translation);
        println("Rotation diff: " + rotation_angle);
    }

    static void printMat(Mat m)
    {
        for (int x=0; x < m.height(); x++) {
            for (int y=0; y < m.width(); y++) {
                System.out.printf("%f",m.get(x,y)[0]);
                System.out.printf("%s"," ");
            }
            System.out.println();
        }
    }

Answer 2

You can compare image similarity using feature point extraction and matching using rotation and scale invariant algorithms.

Already rotation invariant feature matching algorithms are available in OpenCV - SIFT ,SURF but these are patented so you have to pay if you want to use them commercially.

Fortunately an efficient alternative to SIFT or SURF is available namely ORB (Oriented FAST and Rotated BRIEF)

These algorithms work fine in most of the images except a few like document images.

PS: As mentioned in the question (OpenCV + tesseract) alternatively can be used for document images by extracting the text contents in the image using Tesseract library(Opensource OCR engine)