Extract handwritten characters from a boxed form field image

Question

I am trying to extract handwritten characters from field boxes

My desired output would be the character segments with the boxes removed. So far, I've tried defining contours and filtering by area but that hasn't yielded any good results.

# Reading image and binarization
im = cv2.imread('test.png')

char_gray = cv2.cvtColor(im, cv2.COLOR_BGR2GRAY)
char_bw = cv2.adaptiveThreshold(char_gray, 255, cv2.ADAPTIVE_THRESH_MEAN_C, cv2.THRESH_BINARY, 75, 10)

# Applying erosion and dilation
kernel = np.ones((5,5), np.uint8) 
img_erosion = cv2.erode(char_bw, kernel, iterations=1)
img_dilation = cv2.dilate(img_erosion, kernel, iterations=1)  

# Find Canny edges 
edged = cv2.Canny(img_dilation, 100, 200) 

# Finding Contours 
edged_copy = edged.copy()
im2, cnts, hierarchy = cv2.findContours(edged_copy, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
print("Number of Contours found = " + str(len(cnts))) 

# Draw all contours 
cv2.drawContours(im, cnts, -1, (0, 255, 0), 3)

# Filter using area and save
for no, c in enumerate(cnts):
    area = cv2.contourArea(c)
    if area > 100:
        contour = c
        (x, y, w, h) = cv2.boundingRect(contour)
        img = im[y:y+h, x:x+w]
        cv2.imwrite(f'./cnts/cnt-{no}.png', img_dilation)

Please do some research on your problem before asking on this forum. Search this forum and Google for your type of problem. A search of Google shows: https://medium.com/coinmonks/a-box-detection-algorithm-for-any-image-containing-boxes-756c15d7ed26 and https://www.pyimagesearch.com/2017/07/10/using-tesseract-ocr-python/ — fmw42, Dec 26 '19 at 22:32
You can try [removing horizontal and vertical lines](https://stackoverflow.com/questions/46274961/removing-horizontal-lines-in-image-opencv-python-matplotlib) as a preprocessing step before extracting the characters — coffeewin, Dec 26 '19 at 23:26
@fmw42 Hi, I've tried that approach before but it fails to detect any ROIs on my images. — Srikanth Varma, Dec 27 '19 at 05:22
I gave it a shot with @coffeewin idea, but it doesn't seem to completely remove the lines, thus decreasing ocr accuracy. — Srikanth Varma, Dec 27 '19 at 05:22

score 2 · Accepted Answer · answered Dec 30 '19 at 23:59

Here's a simple approach:

Obtain binary image. We load the image, enlarge using imutils.resize(), convert to grayscale, and perform Otsu's thresholding to obtain a binary image
Remove horizontal lines. We create a horizontal kernel then perform morphological opening and remove the horizontal lines using cv2.drawContours
Remove vertical lines. We create a vertical kernel then perform morphological opening and remove the vertical lines using cv2.drawContours

Here's a visualization of each step:

Binary image

Detected lines/boxes to remove highlighted in green

Result

Code

import cv2
import numpy as np
import imutils

# Load image, enlarge, convert to grayscale, Otsu's threshold
image = cv2.imread('1.png')
image = imutils.resize(image, width=500)
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
thresh = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)[1]

# Remove horizontal
horizontal_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (25,1))
detect_horizontal = cv2.morphologyEx(thresh, cv2.MORPH_OPEN, horizontal_kernel, iterations=2)
cnts = cv2.findContours(detect_horizontal, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnts = cnts[0] if len(cnts) == 2 else cnts[1]
for c in cnts:
    cv2.drawContours(image, [c], -1, (255,255,255), 5)

# Remove vertical
vertical_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (1,25))
detect_vertical = cv2.morphologyEx(thresh, cv2.MORPH_OPEN, vertical_kernel, iterations=2)
cnts = cv2.findContours(detect_vertical, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnts = cnts[0] if len(cnts) == 2 else cnts[1]
for c in cnts:
    cv2.drawContours(image, [c], -1, (255,255,255), 5)

cv2.imshow('thresh', thresh)
cv2.imshow('image', image)
cv2.waitKey()

Extract handwritten characters from a boxed form field image

1 Answers1