Extracting hand writing text out in shape with OpenCV

Question

I am very new to OpenCV Python and I really need some help here.

So what I am trying to do here is to extract out these words in the image below.

The words and shapes are all hand drawn, so they are not perfect. I have did some coding below.

First of all, I grayscale the image

img_final = cv2.imread(file_name)
img2gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)

Then I use THRESH_INV to show the content

ret, new_img = cv2.threshold(image_final, 100 , 255, cv2.THRESH_BINARY_INV)

After which, I dilate the content

kernel = cv2.getStructuringElement(cv2.MORPH_CROSS,(3 , 3)) 
dilated = cv2.dilate(new_img,kernel,iterations = 3)

I dilate the image is because I can identify text as one cluster

After that, I apply boundingRect around the contour and draw around the rectangle

contours, hierarchy = cv2.findContours(dilated,cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_NONE) # get contours
index = 0
for contour in contours:

    # get rectangle bounding contour
    [x,y,w,h] = cv2.boundingRect(contour)

    #Don't plot small false positives that aren't text
    if w < 10 or h < 10:
        continue

    # draw rectangle around contour on original image
    cv2.rectangle(img,(x,y),(x+w,y+h),(255,0,255),2)

This is what I got after that.

I am only able to detect one of the text. I have tried many other methods but this is the closet results I have got and it does not fulfill the requirement.

The reason for me to identify the text is so that I can get the X and Y coordinate of each of the text in this image by putting a bounding Rectangle "boundingRect()".

Please help me out. Thank you so much

Answer 1

You can use the fact that the connected component of the letters are much smaller than the large strokes of the rest of the diagram.

I used opencv3 connected components in the code but you can do the same things using findContours.

The code:

import cv2
import numpy as np

# Params
maxArea = 150
minArea = 10

# Read image
I = cv2.imread('i.jpg')

# Convert to gray
Igray = cv2.cvtColor(I,cv2.COLOR_RGB2GRAY)

# Threshold
ret, Ithresh = cv2.threshold(Igray,0,255,cv2.THRESH_BINARY_INV+cv2.THRESH_OTSU)

# Keep only small components but not to small
comp = cv2.connectedComponentsWithStats(Ithresh)

labels = comp[1]
labelStats = comp[2]
labelAreas = labelStats[:,4]

for compLabel in range(1,comp[0],1):

    if labelAreas[compLabel] > maxArea or labelAreas[compLabel] < minArea:
        labels[labels==compLabel] = 0

labels[labels>0] =  1

# Do dilation
se = cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(25,25))
IdilateText = cv2.morphologyEx(labels.astype(np.uint8),cv2.MORPH_DILATE,se)

# Find connected component again
comp = cv2.connectedComponentsWithStats(IdilateText)

# Draw a rectangle around the text
labels = comp[1]
labelStats = comp[2]
#labelAreas = labelStats[:,4]

for compLabel in range(1,comp[0],1):

    cv2.rectangle(I,(labelStats[compLabel,0],labelStats[compLabel,1]),(labelStats[compLabel,0]+labelStats[compLabel,2],labelStats[compLabel,1]+labelStats[compLabel,3]),(0,0,255),2)