How to extract text from image after applying contour in python?

Question

So I have applied contouring on a big image and reached the following cropped part of the image:

But now without using any machine learning model, how do I actually get the image to a text variable? I came to know about template matching but I do not understand how do I proceed from here. I do have images of letters and numbers (named according to their image value) stored in a directory, but how do I match each of them and get the text as a string? I don't want to use any ML model or library like pyTesseract.

I would appreciate any help.

Edit:

The code I have tried for template matching.

def templateMatch(image):
    path = "location"

    for image_path in os.listdir(path + "/characters-images"):
        template = cv2.imread(os.path.join(path, "characters-images", image_path))
        template = cv2.cvtColor(template, cv2.COLOR_BGR2GRAY)

        template = template.astype(np.uint8)
        image = image.astype(np.uint8)

        res = cv2.matchTemplate(template, image, cv2.TM_SQDIFF_NORMED)
        mn, _, mnLoc, _ = cv2.minMaxLoc(res)

        if res is not None:
            return image_path.replace(".bmp", "")


def match(image):
    plate = ""
    # mask = np.zeros(image.shape, dtype=np.uint8)
    # print(image.shape)
    image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    # print(image.shape)
    # print(image)
    thresh = cv2.threshold(image, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)[1]

    cnts = cv2.findContours(thresh, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
    cnts = cnts[0] if len(cnts) == 2 else cnts[1]
    (cnts, _) = contours.sort_contours(cnts, method="left-to-right")

    for con in cnts:
        area = cv2.contourArea(con)

        if 800 > area > 200:
            x, y, w, h = cv2.boundingRect(con)
            # cv2.drawContours(mask, [c], 1, (255, 0, 0), 2)
            temp = thresh[y:y+h, x:x+w]

            character = templateMatching(temp)

            if character is not None:
                plate += character

    return plate

What have you tried template matching wise? You would need to match each letter and use the found positions to set those letters in order. — T A, Jan 06 '20 at 16:49
@TA I have added the code I have tried, but it breaks with an exception after one time and it just detectes 333333 and not other characters for the above image. — Panda, Jan 06 '20 at 16:53

score 3 · Answer 1 · answered Jan 06 '20 at 20:36

How do I actually get the image to a text variable? I came to know about template matching but I do not understand how do I proceed from here.

Template matching is used to locate a object in an image given a template, not to extract text from an image. Matching a template with the position of the object in the image will not help to get the text as a string. For examples on how to apply dynamic scale variant template matching, take a look at how to isolate everything inside of a contour, scale it, and test the similarity to an image? and Python OpenCV line detection to detect X symbol in image. I don't understand why would wouldn't want to use an OCR library. If you want to extract text from the image as a string variable, you should use some type of deep/machine learning. PyTesseract is probably the easiest. Here's a solution using PyTesseract

The idea is to obtain a binary image using Otsu's threshold then perform contour area and aspect ratio filtering to extract the letter/number ROIs. From here we use Numpy slicing to crop each ROI onto a blank mask then apply OCR using Pytesseract. Here's a visualization of each step:

Binary image

Detected ROIs highlighted in green

Isolated ROIs on a blank mask ready for OCR

We use the --psm 6 configuration option to tell Pytesseract to assume a uniform block of text. Look here for more configuration options. Result from Pytesseract:

XS NB 23

Code

import cv2
import numpy as np
import pytesseract

pytesseract.pytesseract.tesseract_cmd = r"C:\Program Files\Tesseract-OCR\tesseract.exe"

# Load image, create mask, grayscale, Otsu's threshold
image = cv2.imread('1.png')
mask = np.zeros(image.shape, dtype=np.uint8)
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
thresh = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)[1]

# Filter for ROI using contour area and aspect ratio
cnts = cv2.findContours(thresh, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
cnts = cnts[0] if len(cnts) == 2 else cnts[1]
for c in cnts:
    area = cv2.contourArea(c)
    peri = cv2.arcLength(c, True)
    approx = cv2.approxPolyDP(c, 0.05 * peri, True)
    x,y,w,h = cv2.boundingRect(approx)
    aspect_ratio = w / float(h)
    if area > 2000 and aspect_ratio > .5:
        mask[y:y+h, x:x+w] = image[y:y+h, x:x+w]

# Perfrom OCR with Pytesseract
data = pytesseract.image_to_string(mask, lang='eng', config='--psm 6')
print(data)

cv2.imshow('thresh', thresh)
cv2.imshow('mask', mask)
cv2.waitKey()

Thank you for the insight, however I am not trying to use pytesseract.image_to_string, rather I was thinking of matching two images and if the match is found, get the name of the image (which will be same as the image character). For example, when comparing a contour with all the images, image N named as N.jpg should have a match for first contour, then I can know N is the first character in the number plate. — Panda, Jan 07 '20 at 03:00
I am doing this without pytesseract to understand more about the algorithm and underlying principles, I would really appreciate some help — Panda, Jan 07 '20 at 04:19
That is a very inefficient way to do it since it will only work when the template is a large "bubble" letter, if you use that approach with regular text it will not work. If you really want to do it that way, take a look at the two links for template matching. One other thing you will have to account for is for multiple letters. If you template match for say the letter 'a', there will be many matches so you will need to determine which one you are looking for — nathancy, Jan 07 '20 at 20:38

score 0 · Answer 2 · answered Jan 06 '20 at 17:02

An option is to consider the bounding box around the characters and to compute the correlation score between a character at hand and those in the training set. You will keep the largest correlation score. (One of SAD, SSD, normalized grayscale correlation or just Hamming distance if your work on a binary image).

You will need to develop a suitable strategy to ensure that the tested characters and the learnt characters have compatible sizes and are properly overlaid.

How to extract text from image after applying contour in python?

2 Answers2