OpenCV feature matching multiple objects

Question

How can I find multiple objects of one type on one image. I use ORB feature finder and brute force matcher (opencv = 3.2.0).

My source code:

import numpy as np
import cv2
from matplotlib import pyplot as plt

MIN_MATCH_COUNT = 10

img1 = cv2.imread('box.png', 0)  # queryImage
img2 = cv2.imread('box1.png', 0) # trainImage

#img2 = cv2.cvtColor(img1, cv2.COLOR_BGR2GRAY)

# Initiate ORB detector
# 
orb = cv2.ORB_create(10000, 1.2, nlevels=9, edgeThreshold = 4)
#orb = cv2.ORB_create()

# find the keypoints and descriptors with SIFT
kp1, des1 = orb.detectAndCompute(img1, None)
kp2, des2 = orb.detectAndCompute(img2, None)

FLANN_INDEX_KDTREE = 0
index_params = dict(algorithm = FLANN_INDEX_KDTREE, trees = 5)
search_params = dict(checks = 50)

flann = cv2.FlannBasedMatcher(index_params, search_params)

des1 = np.float32(des1)
des2 = np.float32(des2)

# matches = flann.knnMatch(des1, des2, 2)

bf = cv2.BFMatcher()
matches = bf.knnMatch(des1, des2, k=2)

# store all the good matches as per Lowe's ratio test.
good = []
for m,n in matches:
    if m.distance < 0.7*n.distance:
        good.append(m)

if len(good)>3:
    src_pts = np.float32([ kp1[m.queryIdx].pt for m in good ]).reshape(-1,1,2)
    dst_pts = np.float32([ kp2[m.trainIdx].pt for m in good ]).reshape(-1,1,2)

    M, mask = cv2.findHomography(src_pts, dst_pts, cv2.RANSAC, 2)

    if M is None:
        print ("No Homography")
    else:
        matchesMask = mask.ravel().tolist()

        h,w = img1.shape
        pts = np.float32([ [0,0],[0,h-1],[w-1,h-1],[w-1,0] ]).reshape(-1,1,2)
        dst = cv2.perspectiveTransform(pts,M)

        img2 = cv2.polylines(img2,[np.int32(dst)],True,255,3, cv2.LINE_AA)

else:
    print ("Not enough matches are found - %d/%d" % (len(good),MIN_MATCH_COUNT))
    matchesMask = None

draw_params = dict(matchColor = (0,255,0), # draw matches in green color
                   singlePointColor = None,
                   matchesMask = matchesMask, # draw only inliers
                   flags = 2)

img3 = cv2.drawMatches(img1,kp1,img2,kp2,good,None,**draw_params)

plt.imshow(img3, 'gray'),plt.show()

But it can find only one instance of query image.

Query Image

Test Image

Result

So its found only one image from two. What I am doing wrong?

Find the first object, compute transformation, mask area of the found object, repeat until you get all the objects. — Michał Gacka, Mar 21 '17 at 23:08
@m3h0w, may be: 1. calculate features 2. compute transformation 3. find first object 4. mask area repeat until get all objects — V. Gai, Mar 22 '17 at 08:37
@m3h0w may be you know, why matcher can find only one object of two on test image — V. Gai, Mar 22 '17 at 09:28
Haven't got time to read the docs right now but I think that it is a fair assumption that the matching algorithm is looking for the best matching object and not for multiple objects. — Michał Gacka, Mar 22 '17 at 09:30
@V.Gai You can also check in the literature what are the common approaches to deal with this situation (strict feature matching deals with matching descriptors, there is no object assumption). The SIFT paper of Lowe proposes an approach based on Hough voting. I found recently this paper: [MAC-RANSAC: a robust algorithm for the recognition of multiple objects](https://hal.archives-ouvertes.fr/hal-00466487v2/document) but it should exist many others. — Catree, Mar 22 '17 at 10:08
maybe you can adapt my ideas from https://stackoverflow.com/questions/26543880/how-to-find-euclidean-distance-between-keypoints-of-a-single-image-in-opencv/26547225#26547225 and https://stackoverflow.com/questions/34310914/recognize-recurring-images-into-a-larger-one/34314697#34314697 from similar tasks — Micka, Oct 10 '18 at 07:51

score 11 · Accepted Answer · edited Jul 22 '21 at 19:51

My source to find multiple objects using ORB descriptors

import cv2
from matplotlib import pyplot as plt

MIN_MATCH_COUNT = 10

img1 = cv2.imread('box.png', 0)  # queryImage
img2 = cv2.imread('box1.png', 0) # trainImage

orb = cv2.ORB_create(10000, 1.2, nlevels=8, edgeThreshold = 5)

# find the keypoints and descriptors with ORB
kp1, des1 = orb.detectAndCompute(img1, None)
kp2, des2 = orb.detectAndCompute(img2, None)

import numpy as np
from sklearn.cluster import MeanShift, estimate_bandwidth

x = np.array([kp2[0].pt])

for i in xrange(len(kp2)):
    x = np.append(x, [kp2[i].pt], axis=0)

x = x[1:len(x)]

bandwidth = estimate_bandwidth(x, quantile=0.1, n_samples=500)

ms = MeanShift(bandwidth=bandwidth, bin_seeding=True, cluster_all=True)
ms.fit(x)
labels = ms.labels_
cluster_centers = ms.cluster_centers_

labels_unique = np.unique(labels)
n_clusters_ = len(labels_unique)
print("number of estimated clusters : %d" % n_clusters_)

s = [None] * n_clusters_
for i in xrange(n_clusters_):
    l = ms.labels_
    d, = np.where(l == i)
    print(d.__len__())
    s[i] = list(kp2[xx] for xx in d)

des2_ = des2

for i in xrange(n_clusters_):

    kp2 = s[i]
    l = ms.labels_
    d, = np.where(l == i)
    des2 = des2_[d, ]

    FLANN_INDEX_KDTREE = 0
    index_params = dict(algorithm = FLANN_INDEX_KDTREE, trees = 5)
    search_params = dict(checks = 50)

    flann = cv2.FlannBasedMatcher(index_params, search_params)

    des1 = np.float32(des1)
    des2 = np.float32(des2)

    matches = flann.knnMatch(des1, des2, 2)

    # store all the good matches as per Lowe's ratio test.
    good = []
    for m,n in matches:
        if m.distance < 0.7*n.distance:
            good.append(m)

    if len(good)>3:
        src_pts = np.float32([ kp1[m.queryIdx].pt for m in good ]).reshape(-1,1,2)
        dst_pts = np.float32([ kp2[m.trainIdx].pt for m in good ]).reshape(-1,1,2)

        M, mask = cv2.findHomography(src_pts, dst_pts, cv2.RANSAC, 2)

        if M is None:
            print ("No Homography")
        else:
            matchesMask = mask.ravel().tolist()

            h,w = img1.shape
            pts = np.float32([ [0,0],[0,h-1],[w-1,h-1],[w-1,0] ]).reshape(-1,1,2)
            dst = cv2.perspectiveTransform(pts,M)

            img2 = cv2.polylines(img2,[np.int32(dst)],True,255,3, cv2.LINE_AA)

            draw_params = dict(matchColor=(0, 255, 0),  # draw matches in green color
                               singlePointColor=None,
                               matchesMask=matchesMask,  # draw only inliers
                               flags=2)

            img3 = cv2.drawMatches(img1, kp1, img2, kp2, good, None, **draw_params)

            plt.imshow(img3, 'gray'), plt.show()

    else:
        print ("Not enough matches are found - %d/%d" % (len(good),MIN_MATCH_COUNT))
        matchesMask = None

Result images

Your answer is the best over all the searching, but I need a c++ implementation, I can't find alternatives of the `estimate_bandwidth` and `MeanShift` in c++, could you point me out? How can I segment the found good matches to clusters in c++?Thank you so much! — Suge, Sep 18 '18 at 04:17
Hi! I think you can find MeanShift for c++ here: https://docs.opencv.org/3.4.3/dc/d6b/group__video__track.html#ga432a563c94eaf179533ff1e83dbb65ea. And about clustring here: https://docs.opencv.org/2.4/modules/core/doc/clustering.html — V. Gai, Sep 19 '18 at 19:59
That `MeanShift` is far different with the one in `Python` `scikit.lean`, it used to track object in camera and can not track multiple objects. While the `kmeans` of `OpenCV` needs to estimate the clusters count first, this situation made me helpless. — Suge, Sep 20 '18 at 07:54
I post my trying to this thread, would you please take a look at it? Very appreciate it, you way is much too perfect!https://stackoverflow.com/questions/52425355/how-to-detect-multiple-objects-with-opencv-in-c — Suge, Sep 20 '18 at 12:38

score 1 · Answer 2 · answered Mar 25 '17 at 23:26

1

To solve this task can be used next approaches:

answered Mar 25 '17 at 23:26

V. Gai

450
3
9
30

OpenCV feature matching multiple objects

2 Answers2

Linked