How to check if feature descriptors and matches are correct?

Question

I'm trying to find common over laps between two images and for this I am using a ORB feature detector and BEBLID feature descriptor. Using these features, find the homography between them and align the images. The function code is as follows:

for pair in image_pairs:
    img1 = cv2.cvtColor(pair[0], cv2.COLOR_BGR2GRAY)
    img2 = cv2.cvtColor(pair[1], cv2.COLOR_BGR2GRAY)

    detector = cv2.ORB_create(1000)
    kpts1 = detector.detect(img1, None)
    kpts2 = detector.detect(img2, None)

    descriptor = cv2.xfeatures2d.BEBLID_create(0.75)
    kpts1, desc1 = descriptor.compute(img1, kpts1)
    kpts2, desc2 = descriptor.compute(img2, kpts2)

    method = cv2.DESCRIPTOR_MATCHER_BRUTEFORCE_HAMMING
    matcher = cv2.DescriptorMatcher_create(method)
    matches = matcher.match(desc1, desc2, None)

    matches = sorted(matches, key=lambda x: x.distance)

    percentage = 0.2
    keep = int(len(matches) * percentage)
    matches = matches[:keep]
    matchedVis = cv2.drawMatches(self.images[pair[0]], kpts1, self.images[pair[1]], kpts2, matches, None)
    cv2.imwrite("feature_match.png", matchedVis)

    ptsA = np.zeros((len(matches), 2), dtype="float")
    ptsB = np.zeros((len(matches), 2), dtype="float")

    for (i, m) in enumerate(matches):
        ptsA[i] = kpts1[m.queryIdx].pt
        ptsB[i] = kpts2[m.trainIdx].pt

    (H, mask) = cv2.findHomography(ptsA, ptsB, method=cv2.RANSAC)
    (h, w) = img2.shape[:2]
    aligned = cv2.warpPerspective(self.images[pair[0]], H, (w, h))
    cv2.imwrite("wrap.png", aligned)

A successful alignment of two images looks like:

And an unsuccessful alignment of two images looks like:

Some of the images in image_pairs list have no common overlaps and hence the alignment fails. Is there a way to detect such failures or even detect a successful alignment without explicitly looking at the warp.png images?

[mre] required. you haven't provided a MRE nor the required input data. this question, as it stands, is unanswerable. please also take the [tour] and review [ask] — Christoph Rackwitz, Feb 23 '23 at 16:44
as for the usual issues with feature matching and homographies... opencv has a function for drawing *matches*. use it, consider whether the illustration looks reasonable. -- do not use weird algorithms. use sift/akaze and flann. when that works, _then_ you can try weird algorithms. -- and don't forget Lowe's ratio test (your code does not contain that). -- do not write code from scratch. use the examples provided by OpenCV. — Christoph Rackwitz, Feb 23 '23 at 16:49
See https://opencv24-python-tutorials.readthedocs.io/en/latest/py_tutorials/py_feature2d/py_matcher/py_matcher.html and https://blog.francium.tech/feature-detection-and-matching-with-opencv-5fd2394a590 or search Google for "python opencv feature matching" for more — fmw42, Feb 23 '23 at 18:57
avoid the brute force matcher. it's inefficient. use `samples/python/find_obj.py` for a selection of descriptors (not just orb, which nobody should use) and how they need to be used with the FLANN matcher. -- this "opencv24-..." site is dead and outdated. up-to-date content to be found on official site: https://docs.opencv.org/4.x/dc/dc3/tutorial_py_matcher.html — Christoph Rackwitz, Feb 24 '23 at 03:44

score 2 · Accepted Answer · answered Feb 24 '23 at 10:17

This thread and its two top answers are a useful resource for what you are doing:
Detecting garbage homographies from findHomography in OpenCV?

One of the things the selected answer suggests is to check the determinant of the homography matrix. Where negative determinants signals a "flipped image", while a very large or very small determinant signals a blown-up and shrunken results respectively.

I think this is a great way to filter out most of the garbage homography RANSAC happens to give you.
Great supplementary material on determinants of linear transformations

However, in your example case with the image "twisting", I frustratingly cannot find a proper math term for describing the transformation.

I've personally dealt with a simpler problem, where I had 4 control points at the corner of every image I want to transform, but I manually implemented the "find homography" function from scratch. The issue arises when I mismatched the control points like this:

(Pardon my terrible hand-drawn illustration)

Where two of the control points are "swapped" and the transformed image is "twisted". With this illustration, I implemented a naive, brute-force check to verify that no two borders in the transformed image can intersect each other.

I found this Korean blog describing this issue, where they also appear to provide mathematical checks for any "abnormal conversions". According to the blog, checking for negative determinants is enough to catch these twisting transformations. They also provided additional checks for ensuring the image doesn't blow up in size or gets shrunken too much. Here is a Google translated excerpt:

Note that threshold values of sx, sy and P can be adjusted for your usecase.

Finally, the blog mentions that these checks will not catch "concave transformations" (see image in blog) and that you should manually check for them. Google translated text from blog:

However, the case of concave is not inspected under the above D<0 condition. To find the concave case, first transform the four points (0, 0), (1, 0), (0, 1), (1, 0) into a 2×2 submatrix and then check whether the transformed result is concave. should be inspected. To determine whether it is concave or not, refer to [Mathematics] - Calculating the Area (Area) of a Polygonal Figure.

How to check if feature descriptors and matches are correct?

1 Answers1