How to find accurate corner positions of a distorted rectangle from blurry image in python?

Question

I am looking for a procedure to detect the corners of an distorted rectangle accurately with OpenCV in Python.

I've tried the solution of different suggestions by googling, but through a sinusoidal superposition of a straight line (see the thresholded image) I probably can't detect the corners. I tried findContours and HoughLines so far without good results. Unfortunately I don't understand the C-Code from Xu Bin in how to find blur corner position with opencv?

This is my initial image:

After resizing and thresholding I apply canny edge detection to get following image:

contours, hierarchy = cv2.findContours(g_mask, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
box = cv2.minAreaRect(contour)
box = cv2.cv.BoxPoints(box) if imutils.is_cv2() else cv2.boxPoints(box)
box = np.array(box, dtype="float")
box = perspective.order_points(box)

I only get the following result with some extra drawing:

I thought line fitting would be a good way to solve the problem, but unfortunately I couldn't get HoughLines working and after looking in OpenCV Python - How to implement RANSAC to detect straight lines? RANSAC seems also difficult to apply for my problem.

Any help is highly appreciated.

Answer 1

Though this is old, this can at least help any others who have the same issue. In addition to nathancy's answer, this should allow you to find the very blurry corners with much more accuracy:

Pseudo Code

0. Resize if desired, not necessary though
1. Convert to grayscale
2. Apply blurring or bilateral filtering
3. Apply Otsu's threshold to get a binary image
4. Find contour that makes up rectangle
5. Approximate contour as a rectangle
6. Points of approximation are your rectangle's corners!

Code to do so

0. Resize:
   The function takes the new width and height, so I am just making the image 5 times bigger than it currently is.

img = cv2.resize(img, (img.shape[0] * 5, img.shape[1] * 5))

1. Grayscale conversion:
   Just converting to grayscale from OpenCV's default BGR colorspace.

gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

2. Blurring/bilateral filtering:
   You could use any number of techniques to soften up this image further, if needed. Maybe a Gaussian blur, or, as nathancy suggested, a bilateral filter, but no need for both.

# choose one, or a different function
blurred = cv2.GaussianBlur(gray, (5, 5), 0)
blurred = cv2.bilateralFilter(gray, 9, 75, 75)

3. Otsu's threshold
   Using the threshold function, pass 0 and 255 as the arguments for the threshold value and the max value. We pass 0 in because we are using the thresholding technique cv2.THRESH_OTSU which determines the value for us. This is returned along with the threshold itself, but I just set it to _ because we don't need it.

_, thresh = cv2.threshold(blurred, 0, 255, cv2.THRESH_OTSU)

4. Finding contour
   There is a lot more to contours than I will explain here, feel free to checkout docs. The important things to know for us is that it returns a list of contours along with a hierarchy. We don't need the hierarchy so it is set to _, and we only need the single contour it finds, so we set contour = contours[0].

contours, _ = cv2.findContours(thresh, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
contour = contours[0]

5. Approximate contour as a rectangle
   First we calculate the perimeter of the contour. Then we approximate it with the cv2.approxPolyDP function, and tell it the maximum distance between the original curve and its approximation with the 0.05 * perimeter. You may need to play around with the decimal for a better approximation.
   The approx is a numpy array with shape (num_points, 1, 2), which in this case is (4, 1, 2) because it found the 4 corners of the rectangle.

Feel free to read up more in the docs.

perimeter = cv2.arcLength(contour, True)
approx = cv2.approxPolyDP(contour, 0.05 * perimeter, True)

6. Find your skewed rectangle!
   You're already done! Here's just how you could draw those points. First we draw the circles by looping over them then grabbing the x and y coordinates, and then we draw the rectangle itself.

# drawing points
for point in approx:
    x, y = point[0]
    cv2.circle(img, (x, y), 3, (0, 255, 0), -1)

# drawing skewed rectangle
cv2.drawContours(img, [approx], -1, (0, 255, 0))

Finished Code

import cv2

img = cv2.imread("rect.png")

img = cv2.resize(img, (img.shape[0] * 5, img.shape[1] * 5))
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
blurred = cv2.bilateralFilter(gray, 9, 75, 75)
_, thresh = cv2.threshold(blurred, 0, 255, cv2.THRESH_OTSU)

contours, _ = cv2.findContours(thresh, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
contour = contours[0]

perimeter = cv2.arcLength(contour, True)
approx = cv2.approxPolyDP(contour, 0.05 * perimeter, True)

for point in approx:
    x, y = point[0]
    cv2.circle(img, (x, y), 3, (0, 255, 0), -1)
cv2.drawContours(img, [approx], -1, (0, 255, 0))

Answer 2

To detect corners, you can use cv2.goodFeaturesToTrack(). The function takes four parameters

corners = cv2.goodFeaturesToTrack(image, maxCorners, qualityLevel, minDistance)

• image - Input 8-bit or floating-point 32-bit grayscale single-channel image
• maxCorners - Maximum number of corners to return
• qualityLevel - Minimum accepted quality level of corners between 0-1. All corners below quality level are rejected
• minDistance - Minimum possible Euclidean distance between corners

Now that we know how to find corners, we have to find the rotated rectangle and apply the function. Here's an approach:

---

We first enlarge the image, convert to grayscale, apply a bilateral filter, then Otsu's threshold to get a binary image

Next we find the distorted rectangle by finding contours with cv2.findContours() then obtain the rotated bounding box highlighted in green. We draw this bounding box onto a mask

Now that we have the mask, we simply use cv2.goodFeaturesToTrack() to find the corners on the mask

Here's the result on the original input image and the (x, y) coordinates for each corner

Corner points

(377.0, 375.0)
(81.0, 344.0)
(400.0, 158.0)
(104.0, 127.0)

Code

import cv2
import numpy as np
import imutils

# Resize image, blur, and Otsu's threshold
image = cv2.imread('1.png')
resize = imutils.resize(image, width=500)
mask = np.zeros(resize.shape, dtype=np.uint8)
gray = cv2.cvtColor(resize, cv2.COLOR_BGR2GRAY)
blur = cv2.bilateralFilter(gray,9,75,75)
thresh = cv2.threshold(blur, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)[1]

# Find distorted rectangle contour and draw onto a mask
cnts = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnts = cnts[0] if len(cnts) == 2 else cnts[1]
rect = cv2.minAreaRect(cnts[0])
box = cv2.boxPoints(rect)
box = np.int0(box)
cv2.drawContours(resize,[box],0,(36,255,12),2)
cv2.fillPoly(mask, [box], (255,255,255))

# Find corners on the mask
mask = cv2.cvtColor(mask, cv2.COLOR_BGR2GRAY)
corners = cv2.goodFeaturesToTrack(mask, maxCorners=4, qualityLevel=0.5, minDistance=150)

for corner in corners:
    x,y = corner.ravel()
    cv2.circle(resize,(x,y),8,(155,20,255),-1)
    print("({}, {})".format(x,y))

cv2.imshow('resize', resize)
cv2.imshow('thresh', thresh)
cv2.imshow('mask', mask)
cv2.waitKey()