Reading OCR text from a dot matrix display image with Python and OpenCV

Question

I am looking to extract some text from a meter display using Python 3 and OpenCV. I have had some success with a lot of help from SO on the code below, and I can print text from a basic and 'tidy' image file. However when I look to extract from the attached dot matrix image the script is unable to pick out any text at all.

Is there a limitation to extracting in this kind of dot matrix text?

Here's what I am working with:

import cv2
import numpy as np
import pytesseract
from PIL import Image
from cv2 import boundingRect, countNonZero, cvtColor, drawContours, findContours, getStructuringElement, \
    imread, morphologyEx, pyrDown, rectangle, threshold

img = imread('test.JPG')
# down sample and use it for processing
adjusted = pyrDown(img)
# gray-scale image
img_gray = cvtColor(adjusted, cv2.COLOR_BGR2GRAY)

# morph gradient
morph_kernel = getStructuringElement(cv2.MORPH_ELLIPSE, (3, 3))
grad = morphologyEx(img_gray, cv2.MORPH_GRADIENT, morph_kernel)

# change to binary and morph
_, bw = threshold(src=grad, thresh=0, maxval=255, type=cv2.THRESH_BINARY+cv2.THRESH_OTSU)
morph_kernel = getStructuringElement(cv2.MORPH_RECT, (9, 1))
connected = morphologyEx(bw, cv2.MORPH_CLOSE, morph_kernel)
applyMask = np.zeros(bw.shape, np.uint8)

# get contours
im2, contour, hierarchy = findContours(connected, cv2.RETR_CCOMP, cv2.CHAIN_APPROX_SIMPLE)
# filter contours
for index in range(0, len(hierarchy[0])):
    rect = x, y, rectangle_width, rectangle_height = boundingRect(contour[index])

    # draw contour
    mask = drawContours(applyMask, contour, index, (255, 255, 2555), cv2.FILLED)

    # find non-zero pixels ratio
    r = float(countNonZero(applyMask)) / (rectangle_width * rectangle_height)
    if r > 0.5 and rectangle_height > 8 and rectangle_width > 8:
        rec_img = rectangle(adjusted, (x, y+rectangle_height), (x+rectangle_width, y), (0, 255, 0), 3)

        text = pytesseract.image_to_string(Image.fromarray(rec_img))
        print(text)

And here's the image I am trying to extract from:

Answer 1

In my experience, pytesseract doesn't work well in real word. Block Binary Pixel Sum could be better in your case.

Answer 2

This is the best solution I could find. Try blurring the image and then use OpenCV to pull the text. I was able to successfully pull out "Meter Input" but you may need another method to pull out the numbers.

import PIL
from PIL import Image
import PIL.ImageOps
import cv2 as cv
import pytesseract
from PIL import ImageFilter
from pytesseract import image_to_string
pytesseract.pytesseract.tesseract_cmd = r'C:\Program Files\Tesseract-OCR\tesseract.exe'

#open the image and check it out
image = Image.open('dotMatrixScreen.jpg')
display(image)

cropped_img = image.crop((230, 350, 1000, 570))
display(cropped_img)

full_pannel_blur = cropped_img.filter(ImageFilter.BoxBlur(radius=4))    
display(full_pannel_blur)

full_pannel_blur_grey = full_pannel_blur.convert("L")
full_pannel_blur_bw = full_pannel_blur_grey.point(lambda x: 0 if x<122 else 255, '1')
full_pannel_blur_bw.save("full_pannel_blur_bw.png")
display(full_pannel_blur_bw)

full_pannel_bw = full_pannel_blur_bw.convert("L")
full_pannel_bw.save("full_pannel_bw.png")

full_pannel_bw_cv = cv.imread("full_pannel_bw.png", 0)
display(Image.fromarray(full_pannel_bw_cv))

text = image_to_string(full_pannel_bw_cv)
print("Pytesseract got: ", '"' + text + '"')

Out: Pytesseract got: "Meter Index hes. S46m3"

Answer 3

This is very difficult to catch. Tried black & white image, but the leading zero's are missing:

import pytesseract
import cv2

image_file = "meter.jpg"

# load the input image, convert it from BGR to RGB channel ordering,
image = cv2.imread(image_file, cv2.IMREAD_UNCHANGED)
rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
print('Original Dimensions : ',image.shape)
 
scale_percent = 21 # percent of original size
width = int(rgb.shape[1] * scale_percent / 100)
height = int(rgb.shape[0] * scale_percent / 100)
dim = (width, height)
  
# resize image
resized = cv2.resize(rgb, dim, interpolation = cv2.INTER_AREA)
print('Resized Dimensions : ',resized.shape)

grayImage = cv2.cvtColor(resized, cv2.COLOR_BGR2GRAY)
(thresh, blackAndWhiteImage) = cv2.threshold(grayImage, 140, 255, cv2.THRESH_BINARY) # 0...255
bw = cv2.imshow('Black white image', blackAndWhiteImage)

# Configuration
options = r'--psm 6 --oem 3 -c tessedit_char_whitelist=0123456789.,:mIndexMtr'

# OCR the input image using Tesseract
text_bw = pytesseract.image_to_string(blackAndWhiteImage, config=options)
print(text_bw)

cv2.imshow('Resized', resized)
cv2.waitKey(0)
cv2.destroyAllWindows()

Output:

MeterIndex
065.846m3

Image:

Answer 4

This looks exactly like the gas meter I've found how to read. It was tricky and I'll be the first to admit to it being a bodge, but it works.

I use an ESP32Cam as my camera, running ftp and MQTT to ftp a photo on command to my home automation PC. i use a Meccano motor to turn a cam which pushes the red button in on the meter. it's driven by another ESP, also via MQTT. The MQTT payload sends a duration in mS for how long the motor should activate to make roughly one turn. The same payload on the same topic tells the ESP Cam how long to wait before taking a picture (plus 1000mS). there's a limited time before the meter reading goes dark again and only so many times a day you can do a reading...

On my Home Automation PC I run an Android emulator called MEmu. On this I run Tasker, MQTT Client (integrates a a plugin to Tasker) and the Text Scanner App, which does the actual OCR.

When the ESPCam has completed the ftp of the picture to the HA PC,it sends it's own MQTT message which is the trigger used by Tasker to launch a Task sequence that uses AutoInput plugin to press the various buttons to navigate Text Scanner to pick up the file from it's default /use/Memu/Picture folder, scan it, copy the output, use a short Javascriptlet to strip off the "Meter Index" and "m3", leaving just the meter reading, which is sent to the MQTT plugin and on to my HA system. Simples!

It's like watching one of those Heath Robinson contraptions boil an egg but it works. Also, I tried every other way possible, believe me, including Python, Tesseract and using a mobile phone to take the photos. The phone worked OK for while but the camera stopped working.

So, there you go!