In python, is how can many rectangles, which are not in order in numpy or list are macthed?

Question

I have developed two detectors based on deep learning (Faster R-CNN).

The first detector detects one image and returns the coordinates of bigger rectangles. For example:

 rectangles1 = [[10, 10, 50, 50], [60, 10, 150, ,150], [180, 180, 250, 250], [180, 400, 450, 450], [...]] # xmin, ymin, xmax, ymax

The second detector detects one image, which is same as above, and returns smaller rectangle.

 rectangles2 = [[10, 10, 30, 30], [60, 20, 150, 120], [160, 200, 270, 270], [200, 450, 425, 425], [...]] # xmin, ymin, xmax, ymax

If I plot these two (rectangles1, rectangles2), it can be shown like the picture below:

So, each coordinate of rectangle1 has only one corresponding coordinate of rectangle2. But, the returned coordinates of rectangles are not in order. For example,

rectangle1 = [rectangle1[0], rectangle1[2], rectangle1[4], rectangle1[3], rectangle1[1], rectangle[…]]
rectangle2 = [rectangle2[1], rectangle2[3], rectangle2[2], rectangle2[0], rectangle1[4], rectangle[…]]

What I want is that each coordinate of rectangle1 and one corresponding coordinate of rectangle2 are returned together. For example:

final = [[rectangle1[0], rectangle2[0]], [rectangle1[1], rectangle2[1]], [rectangle1[2], rectangle2[2], rectangle1[…], rectangle2[…]]

How could I make this code? Could you please give me a hint? I have watched some code in stackoverflow but they can only receive two rectangles. Thank you.

For making an image, you can use the code below:

import numpy as np
import cv2

img = np.zeros((800, 800, 3), np.uint8)

img1 = cv2.rectangle(img, (10, 10), (50, 50), (255, 0, 0), 3)
img1 = cv2.rectangle(img, (60, 10), (150, 150), (255, 0, 0), 3)
img1 = cv2.rectangle(img, (180, 180), (250, 250), (255, 0, 0), 3)
img1 = cv2.rectangle(img, (180, 400), (450, 450), (255, 0, 0), 3)

img2 = cv2.rectangle(img, (10, 10), (30, 30), (0, 0, 255), 3)
img2 = cv2.rectangle(img, (60, 20), (150, 120), (0, 0, 255), 3)
img2 = cv2.rectangle(img, (160, 200), (270, 270), (0, 0, 255), 3)
img2 = cv2.rectangle(img, (200, 450), (425, 425), (0, 0, 255), 3)

cv2.imshow('rectangle', img)
cv2.waitKey(0)
cv2.destroyAllWindows()

Thank you.

This is the same question you asked yesterday [How could I match two detected rectangles from image?](https://stackoverflow.com/questions/64832706/how-could-i-match-two-detected-rectangles-from-image), except now you want a for loop. — Ash, Nov 15 '20 at 10:52
Totally different. Previous question is that just find and calculate intersection of union of two detected rectangles with in arranged order. This question is to consider misaligned situation. Please don't come out swinging. — Seunghyeon, Nov 15 '20 at 12:51

Abhi25t · Accepted Answer · 2020-11-15T13:54:10.440

You need to calculate IOU of each possible pair of rectangles and select the highest IOU as best match:

Suppose two list of rectangles are (from your question above)

rectangles1 = [[10, 10, 50, 50], [60, 10, 150, 150], [180, 180, 250, 250], [180, 400, 450, 450]]
rectangles2 = [[10, 10, 30, 30], [60, 20, 150, 120], [160, 200, 270, 270], [200, 450, 425, 425]]

IOU function:

def intersection_over_union(box1, box2):
    # Get coordinates of the intersection 
    x1 = max(box1[0], box2[0])
    y1 = max(box1[1], box2[1])
    x2 = min(box1[2], box2[2])
    y2 = min(box1[3], box2[3])

    # Get the area of intersection rectangle
    intersection = max(0, x2 - x1 + 1) * max(0, y2 - y1 + 1)

    # Get the area of both rectangles
    box1Area = (box1[2] - box1[0] + 1) * (box1[3] - box1[1] + 1)
    box2Area = (box2[2] - box2[0] + 1) * (box2[3] - box2[1] + 1)

    iou = intersection / float(box1Area + box2Area - intersection)

    return iou

Code for matching:

matching_pairs = {}
for r1 in rectangles1:
    best_iou = 0   # Initiate with 0
    for r2 in rectangles2:
        iou = intersection_over_union(r1, r2)
        if iou > best_iou:
            matching_pairs[str(r1)] = r2

print the result

for rect in matching_pairs:
    print(rect, ':' ,matching_pairs[rect])

[10, 10, 50, 50] : [10, 10, 30, 30]

[60, 10, 150, 150] : [60, 20, 150, 120]

[180, 180, 250, 250] : [160, 200, 270, 270]

As you see the fourth pair is not returned because the intersection_over_union function assumes that the bounding boxes are returned in the standard format where following conditions are correct:

assert box1['x1'] <= box1['x2']
assert box1['y1'] <= box1['y2']
assert box2['x1'] <= box2['x2']
assert box2['y1'] <= box2['y2']

More details in this answer.

Since the fourth bounding box pair does not hold that condition, its best to use jaccard_score (another name for IOU) from sklearn

from sklearn.metrics import jaccard_score

box1 = [180, 400, 450, 450]
box2 = [200, 450, 425, 425]

img = np.zeros((800, 800, 3), np.uint8)  # use your image shape here or directly below

img1 = cv2.rectangle(np.zeros(img.shape), (box1[0], box1[1]), (box1[2], box1[3]), (1, 1, 1), -1) 
img2 = cv2.rectangle(np.zeros(img.shape), (box2[0], box2[1]), (box2[2], box2[3]), (1, 1, 1), -1)

jaccard_score(img1.ravel(),img2.ravel())

0.42515013385427974

I think that you are genius...Could I ask In example dataset rectangle1[3], which is [180, 400, 450, 450], rectangle2[3], which is [200, 450, 425, 425], are not returned and printed? — Seunghyeon, Nov 15 '20 at 12:42
import numpy as np import cv2 img = np.zeros((800, 800, 3), np.uint8) img1 = cv2.rectangle(img, (180, 400), (450, 450), (255, 0, 0), 3) img2 = cv2.rectangle(img, (200, 450), (425, 425), (0, 0, 255), 3) cv2.imshow('rectangle', img) cv2.waitKey(0) cv2.destroyAllWindows() — Seunghyeon, Nov 15 '20 at 12:54
Yeah there was a problem - it assumed that the bounding box follows some standard format for coordinates. Edited the answer to handle that condition as well. Let me know if you still need any help. — Abhi25t, Nov 15 '20 at 13:55

In python, is how can many rectangles, which are not in order in numpy or list are macthed?

1 Answers1