Billboard corner detection

Question

I was trying to detect billboard images on a random background. I was able to localize the billboard using SSD, this give me approximate bounding box around the billboard. Now I want to find the exact corners of the billboard for my application. I tried using different strategies which I came across such as Harris corner detection (using Opencv), finding intersections of lines using, Canny + morphological operations + contours. The details on the output is given below.

Harris corner detection The pseudocode for the harris corner detection is as follows:

img_patch_gray = np.float32(img_patch_gray)
harris_point = cv2.cornerHarris(img_patch_gray,2,3,0.04)
img_patch[harris_point>0.01*harris_point.max()]=[255,0,0]
plt.figure(figsize=IMAGE_SIZE)
plt.imshow(img_patch)

Here the red dots are the corners detected by the Harris corner detection algorithm and the points of interest are encircled in green.

Using Hough line detection Here I was trying to find the intersection of the lines and then choosing the points. Something similar to stackoverflow link, but it is very difficult to get the exact lines since billboards have text and graphics in it.

Contour based In this approach I have used canny edge detector, followed by dilation(3*3 kernel), followed by contour.

bin_img = cv2.Canny(gray_img_patch,100,250)
bin_img = dilate(bin_img, 3)
plt.imshow(bin_img, cmap='gray')
(_,cnts, _) = cv2.findContours(bin_img.copy(), 
cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)
cnts = sorted(cnts, key = cv2.contourArea, reverse = True)[:10]
cv2.drawContours(img_patch, [cnts[0]],0, (0,255,0), 1)

, . I had tried using approxPolyDp function from openCV but it was not as expected since it can also approximate larger or smaller contours by four points and in some images it might not form contours around the billboard frame.

I have used openCV 3.4 for all the image processing operations. used can be found here. Please note that the image discussed here is just for the illustration purpose and in general image can be of any billboard. Thanks in advance, any help is appreciated.

Answer 1

This is a very difficult task because the image containes a lot of noise. You can get an approximation of the contour but specific corners would be very hard. I have made an example on how I would make an approximation. It may not work on other images. Maybe it will help a bit or give you a new idea. Cheers!

import cv2
import numpy as np

# Read the image
img = cv2.imread('billboard.png')

# Blur the image with a big kernel and then transform to gray colorspace
blur = cv2.GaussianBlur(img,(19,19),0)
gray = cv2.cvtColor(blur,cv2.COLOR_BGR2GRAY)

# Perform histogram equalization on the blur and then perform Otsu threshold
equ = cv2.equalizeHist(gray)
_, thresh = cv2.threshold(equ,0,255,cv2.THRESH_BINARY+cv2.THRESH_OTSU)

# Perform opening on threshold with a big kernel (erosion followed by dilation)
kernel = np.ones((20,20),np.uint8)
opening = cv2.morphologyEx(thresh, cv2.MORPH_OPEN, kernel)

# Search for contours and select the biggest one
_, contours, hierarchy = cv2.findContours(opening,cv2.RETR_TREE,cv2.CHAIN_APPROX_NONE)
cnt = max(contours, key=cv2.contourArea)

# Make a hull arround the contour and draw it on the original image
mask = np.zeros((img.shape[:2]), np.uint8)
hull = cv2.convexHull(cnt)
cv2.drawContours(mask, [hull], 0, (255,255,255),-1)

# Search for contours and select the biggest one again
_, thresh = cv2.threshold(mask,0,255,cv2.THRESH_BINARY+cv2.THRESH_OTSU)
_, contours, hierarchy = cv2.findContours(thresh,cv2.RETR_TREE,cv2.CHAIN_APPROX_NONE)
cnt = max(contours, key=cv2.contourArea)

# Draw approxPolyDP on the image
epsilon = 0.008*cv2.arcLength(cnt,True)
approx = cv2.approxPolyDP(cnt,epsilon,True)
cv2.drawContours(img, [cnt], 0, (0,255,0), 5)

# Display the image
cv2.imshow('img', img)
cv2.waitKey(0)
cv2.destroyAllWindows()

Result: