Cluster bounding boxes and draw line on them (OpenCV, Python)

Question

With this code I created some bounding boxes around the characters in the below image:

import csv
import cv2
from pytesseract import pytesseract as pt

pt.run_tesseract('bb.png', 'output', lang=None, boxes=True, config="hocr")

# To read the coordinates
boxes = []
with open('output.box', 'rt') as f:
    reader = csv.reader(f, delimiter=' ')
    for row in reader:
        if len(row) == 6:
            boxes.append(row)

# Draw the bounding box
img = cv2.imread('bb.png')
h, w, _ = img.shape
for b in boxes:
    img = cv2.rectangle(img, (int(b[1]), h-int(b[2])), (int(b[3]), h-int(b[4])), (0, 255, 0), 2)

cv2.imshow('output', img)
cv2.waitKey(0)

OUTPUT

What I would like to have is this:

The program should draw a perpendicular line on the X axis of the bounding box (only for the first and third text-area. The one in the middle must not be interested in the process).

The goal is this (and of there is another way to achieve it, please explain): once I have this two lines (or, better, group of coordinates), using a mask to cover this two areas.

Is it possible ?

Source image:

CSV as requested: print(boxes)

[['l', '56', '328', '63', '365', '0'], ['i', '69', '328', '76', '365', '0'], ['n', '81', '328', '104', '354', '0'], ['e', '108', '328', '130', '354', '0'], ['1', '147', '328', '161', '362', '0'], ['m', '102', '193', '151', '227', '0'], ['i', '158', '193', '167', '242', '0'], ['d', '173', '192', '204', '242', '0'], ['d', '209', '192', '240', '242', '0'], ['l', '247', '193', '256', '242', '0'], ['e', '262', '192', '292', '227', '0'], ['t', '310', '192', '331', '235', '0'], ['e', '334', '192', '364', '227', '0'], ['x', '367', '193', '398', '227', '0'], ['t', '399', '192', '420', '235', '0'], ['-', '440', '209', '458', '216', '0'], ['n', '481', '193', '511', '227', '0'], ['o', '516', '192', '548', '227', '0'], ['n', '553', '193', '583', '227', '0'], ['t', '602', '192', '623', '235', '0'], ['o', '626', '192', '658', '227', '0'], ['t', '676', '192', '697', '235', '0'], ['o', '700', '192', '732', '227', '0'], ['u', '737', '192', '767', '227', '0'], ['c', '772', '192', '802', '227', '0'], ['h', '806', '193', '836', '242', '0'], ['l', '597', '49', '604', '86', '0'], ['i', '610', '49', '617', '86', '0'], ['n', '622', '49', '645', '75', '0'], ['e', '649', '49', '671', '75', '0'], ['2', '686', '49', '710', '83', '0']]

EDIT:

To use zindarod answer, you need tesserocr. Installation through pip install tesserocr can give you various errors. I found wheel version of it (after hours trying to install and solve errors, see my comment below the answer...): here you can find/download it.

Hope this helps..

Answer 1

Googles tesseract-ocr already has this functionality in page segmentation method(psm). You just need to use a better python wrapper, which exposes more of tesseract's functionalities than pytesseract does. One of the better ones is tesserocr.

A simple example with your image:

  import cv2
  import numpy as np
  import tesserocr as tr
  from PIL import Image

  cv_img = cv2.imread('text.png', cv2.IMREAD_UNCHANGED)

  # since tesserocr accepts PIL images, converting opencv image to pil
  pil_img = Image.fromarray(cv2.cvtColor(cv_img,cv2.COLOR_BGR2RGB))

  #initialize api
  api = tr.PyTessBaseAPI()
  try:
    # set pil image for ocr
    api.SetImage(pil_img)
    # Google tesseract-ocr has a page segmentation methos(psm) option for specifying ocr types
    # psm values can be: block of text, single text line, single word, single character etc.
    # api.GetComponentImages method exposes this functionality
    # function returns:
    # image (:class:`PIL.Image`): Image object.
    # bounding box (dict): dict with x, y, w, h keys.
    # block id (int): textline block id (if blockids is ``True``). ``None`` otherwise.
    # paragraph id (int): textline paragraph id within its block (if paraids is True).
    # ``None`` otherwise.
    boxes = api.GetComponentImages(tr.RIL.TEXTLINE,True)
    # get text
    text = api.GetUTF8Text()
    # iterate over returned list, draw rectangles
    for (im,box,_,_) in boxes:
      x,y,w,h = box['x'],box['y'],box['w'],box['h']
      cv2.rectangle(cv_img, (x,y), (x+w,y+h), color=(0,0,255))
  finally:
    api.End()

  cv2.imshow('output', cv_img)
  cv2.waitKey(0)
  cv2.destroyAllWindows()

Answer 2

I am late here, searching for something else. I have never used the tesser wrappers, they just seem to get in the way for no real benefit. All they are doing is abstracting away the call to the subprocess?

This is how I access psm configuration through the args passed to a subprocess. I have included oem, pdf and hocr parameters as well just for completeness but this is not necessary, you can just pass the psm parameter. Do make the help call at the terminal as there are 13 psm options and 4 for oem. Depending on what you are doing the quality can be highly dependent on psm.

It is possible to Pipe in and out using subprocess.Popen() or if you are feeling adventurous you can do it asynchronously with asyncio.create_subprocess_exec()in much the same way.

import subprocess

# args 
# 'tesseract' - the executable name
# path to the image file
# output file name - no extension tesser will add .txt .pdf .hocr etc etc
# optional params
# -psm x to set the page segmentation mode see more with tesseract --help-psm at the cli
# -oem x to set ocr engine mode see more with tesseract --help-osm
# can add a mode parameter to the end of the args list to get output in :
# searchable pdf - just add a parameter 'pdf' as below
# hOCR output (html) - just add 'hocr' as below

args = ['tesseract', 'Im1.tiff', 'Im1', '-psm 1', '-oem 2']

# args = ['tesseract', 'Im1.tiff', 'Im1', '-psm 1', '-oem 2', 'pdf']
# args = ['tesseract', 'Im1.tiff', 'Im1', '-psm 1', '-oem 2', 'hocr']

try:
    proc = subprocess.check_call(args)
    print('subprocess retcode {r}'.format(r=proc))
except subprocess.CalledProcessError as exp:
    print('subprocess.CalledProcessError : ', exp)