Detect markers in 2D Image [Python/OpenCV]

Question

I would like to create a script in Python (with use OpenCV library) that determines which markers are in the picture. Markers look something like this:

Markers

After loading the image, the script should print which markers are in the picture (return numbers of markers). For example, if I load this picture:

Image with markers

For this image a script should return three numbers: 1, 2 and 3. I have a script, which load image and recognize figures (circles, squares etc. - in script below only squares) but I haven't any idea to recognise whole marker, which consists of several figures. Any ideas? Please for any suggestions about algorithm or any solution.

import numpy as np
import cv2

img = cv2.imread('B.jpg')
gray = cv2.imread('B.jpg',0)

ret,thresh = cv2.threshold(gray,120,255,1)

contours,h = cv2.findContours(thresh,1,2)

for cnt in contours:
    approx = cv2.approxPolyDP(cnt,0.01*cv2.arcLength(cnt,True),True)
    print len(approx)
    if len(approx)==4:
        print "square"
        cv2.drawContours(img,[cnt],0,(0,0,255))

cv2.imshow('img',img)
cv2.waitKey(0)
cv2.destroyAllWindows()

But obviously it isn't what I need it only surrounds rectangulars. Thanks for any tips and help.

1. find outer contours. 2. choose boundingbox for each contour (which should be identical to the contour since the external contours in your images are squares) 3. resize the image so that it had the same size as your reference markers. 4. use chamfer matching on canny edge image of resized image region and teference markers. 5. if one chamfer matching is smalö enough choose the smallest one as detected marker — Micka, May 04 '16 at 20:29
if you have additional perspective distortions you have to find quads instead of boundingBoxes and use perspective warping. — Micka, May 04 '16 at 20:31
I'd probably try feature matching as described here http://docs.opencv.org/3.1.0/d1/de0/tutorial_py_feature_homography.html#gsc.tab=0 but I did not get it working. Have also a look at this post: http://stackoverflow.com/questions/20259025/module-object-has-no-attribute-drawmatches-opencv-python — tfv, May 04 '16 at 20:52
Thanks for tips, I will read about it and I will try to use it in my script. — Lipstick, May 04 '16 at 21:04

score 0 · Answer 1 · answered May 05 '16 at 08:55

Firstly I try to use feature matching (proposition by @tfv user) but it probably isn't a good solution. In many approaches it gives too many matches (but images are completely different) or too few matches (when images are the same)

Now I will try use an algorithm by @Micka but I have a problem with second and third point. The code in my post above find squares. How do I save the new rectangles as new images?

score 0 · Answer 2 · answered Jun 07 '21 at 01:48

Feature matching will perform poorly because these markers have very few features (you want clusters of corners and fine details for feature matching).

These markers lend themselves better for a Template Matching scheme. The correlation between the input image and the templates will be the highest for the correct template type (given that the template is properly positioned, oriented, and scaled). The following code works for your example. It includes a lot of required preprocessing but the essence is to find the highest correlation: np.correlate(img.flatten(), templ.flatten()). The code could be improved in many ways to make it more robust against variations in marker position, scale, orientation, noise etc.

import matplotlib.pyplot as plt
import numpy as np
import cv2

# Detect and store the 6 templates (markers)
fig0, axs0 = plt.subplots(2)
imgs = cv2.imread("markers.png")
imgs_gray = cv2.cvtColor(imgs, cv2.COLOR_BGR2GRAY)
ret, imgs_th = cv2.threshold(imgs_gray, 100, 255, cv2.THRESH_BINARY)
xywh = np.zeros((0, 4), dtype=np.int32)
contours, hierarchies = cv2.findContours(
    imgs_th, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE
)
for idx, c in enumerate(contours):
    if hierarchies[0, idx, 3] == 0 and cv2.contourArea(c) > 20000:
        x, y, w, h = cv2.boundingRect(c)
        xywh = np.vstack((xywh, np.array([x, y, w, h])))
axs0[0].set_title("thresholded markers image")
axs0[0].imshow(imgs_th, cmap="gray")
sortx_xywh = xywh[np.argsort(xywh[:, 0])]
sortyx_xywh = sortx_xywh[np.argsort(sortx_xywh[:, 1])]
max_w = np.amax(sortyx_xywh[:, 2])
max_h = np.amax(sortyx_xywh[:, 3])
templates = np.zeros((max_h, max_w, 6))
for i, xy in enumerate(sortyx_xywh[:, 0:2]):
    templates[:, :, i] = imgs_th[xy[1] : xy[1] + max_h, xy[0] : xy[0] + max_w]

# Detect the marker regions in the input image
img_in = cv2.imread("input.jpg")
img_gray = cv2.cvtColor(img_in, cv2.COLOR_BGR2GRAY)
ret, img_th = cv2.threshold(img_gray, 100, 255, cv2.THRESH_BINARY)
xywh = np.zeros((0, 4), dtype=np.int32)
contours, hierarchies = cv2.findContours(img_th, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
for idx, c in enumerate(contours):
    if hierarchies[0, idx, 3] == 0 and cv2.contourArea(c) > 20000:
        x, y, w, h = cv2.boundingRect(c)
        xywh = np.vstack((xywh, np.array([x, y, w, h])))
axs0[1].set_title("thresholded input  image")
axs0[1].imshow(img_th, cmap="gray")
fig0.show()

# Use simplified template matching (correlation) to determine marker type and orientation
for xy in xywh[:, 0:2]:
    fig1, axs1 = plt.subplots(5, 6)
    img = img_th[xy[1] : xy[1] + max_h, xy[0] : xy[0] + max_w]
    axs1[0, 0].imshow(img, cmap="gray")
    axs1[0, 0].set_title("input image")
    corr = np.zeros((4, 6))
    for t in range(6):  # 6 templates
        templ = templates[:, :, t]
        for o in range(4):  # 4 orientations
            corr[o, t] = np.correlate(img.flatten(), templ.flatten())
            axs1[o + 1, t].imshow(templ, cmap="gray")
            axs1[o + 1, t].set_title("corr = {:.2e}".format(corr[o, t]))
            templ = np.rot90(templ)
    rot, typ = np.unravel_index(np.argmax(corr, axis=None), corr.shape)
    print("Input marker at ({},{}) is type {}, rotated {} degrees.".format(xy[0], xy[1], typ + 1, rot * 90))
    fig1.tight_layout(pad=0.001)
    fig1.show()

Detect markers in 2D Image [Python/OpenCV]

2 Answers2