detect boxes/tables and remove them

Question

How to remove all vertical and horizontal lines that form boxes/tables

I have searched and tried.. But can't make it work

Have tried to search for it the last couple of days.. have found a few examples which doesn't work.. Have tried to get the pieces together..

cv:Mat img = cv::imread(input, CV_LOAD_IMAGE_GRAYSCALE);

cv::Mat grad;
cv::Mat morphKernel = cv::getStructuringElement(cv::MORPH_ELLIPSE, cv::Size(3, 3));
cv::morphologyEx(img, grad, cv::MORPH_GRADIENT, morphKernel);

cv::Mat res;
cv::threshold(grad, res, 0, 255, cv::THRESH_BINARY | cv::THRESH_OTSU);

// find contours
cv::Mat mask = cv::Mat::zeros(res.size(), CV_8UC1);
std::vector<std::vector<cv::Point>> contours;
std::vector<cv::Vec4i> hierarchy;
cv::findContours(res, contours, CV_RETR_EXTERNAL, CV_CHAIN_APPROX_SIMPLE);

for(int i = 0; i < contours.size(); i++){
    cv::Mat approx;
    double peri = cv::arcLength(contours[i], true);
    cv::approxPolyDP(contours[i], approx, 0.04 * peri, true);
    int num_vertices = approx.rows;

    if(num_vertices == 4){
        cv::Rect rect = cv::boundingRect(contours[i]);

        // this is a rectangle
    }
}

Answer 1

You could try something like that :

• threshold your image
• compute connected components
• remove particules for which at least 3 of 4 bounding box tops are in touch with particule

This should give you something like that :

Here is the associated source code :

#include <iostream>
#include <opencv2/core.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/highgui.hpp>
#include <limits>

using namespace cv;

struct BBox {

    BBox() :
        _xMin(std::numeric_limits<int>::max()),
        _xMax(std::numeric_limits<int>::min()),
        _yMin(std::numeric_limits<int>::max()),
        _yMax(std::numeric_limits<int>::min())
    {}

    int _xMin;
    int _xMax;
    int _yMin;
    int _yMax;
};

int main()
{
    // read input image
    Mat inputImg = imread("test3_1.tif", IMREAD_GRAYSCALE);

    // create binary image
    Mat binImg;
    threshold(inputImg, binImg, 254, 1, THRESH_BINARY_INV);

    // compute connected components
    Mat labelImg;
    const int nbComponents = connectedComponents(binImg, labelImg, 8, CV_32S);

    // compute associated bboxes
    std::vector<BBox> bboxColl(nbComponents);
    for (int y = 0; y < labelImg.rows; ++y) {

        for (int x = 0; x < labelImg.cols; ++x) {

            const int curLabel = labelImg.at<int>(y, x);
            BBox& curBBox = bboxColl[curLabel];
            if (curBBox._xMin > x)
                curBBox._xMin = x;
            if (curBBox._xMax < x)
                curBBox._xMax = x;
            if (curBBox._yMin > y)
                curBBox._yMin = y;
            if (curBBox._yMax < y)
                curBBox._yMax = y;
        }
    }

    // parse all labels
    std::vector<bool> lutTable(nbComponents);
    for (int i=0; i<nbComponents; ++i) {

        // check current label width
        const BBox& curBBox = bboxColl[i];
        if (curBBox._xMax - curBBox._xMin > labelImg.cols * 0.3)
            lutTable[i] = false;
        else
            lutTable[i] = true;
    }

    // create output image
    Mat resImg(binImg);
    MatConstIterator_<int> iterLab = labelImg.begin<int>();
    MatIterator_<unsigned char> iterRes = resImg.begin<unsigned char>();
    while (iterLab != labelImg.end<int>()) {

        if (lutTable[*iterLab] == true)
            *iterRes = 1;
        else
            *iterRes = 0;

        ++iterLab;
        ++iterRes;
    }

    // write result
    imwrite("resImg3_1.tif", resImg);
}

I simply remove all labels for which with is greater than 30% of image total width. Your image is quite noisy so I can't use bounding box tops touches as said before, sorry...

Don't know if this will match with all your images but you could add some geometrical filters to improve this first version.

Regards,

Answer 2

You can use LineSegmentDetector for this purpose:

import numpy as np
import cv2

image = cv2.imread("image.png")
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
# This is the detector, you might have to play with the parameters
lsd = cv2.createLineSegmentDetector(0, _scale=0.6)

lines, widths, _, _ = lsd.detect(gray)

if lines is not None:
    for i in range(0, len(lines)):
        l = lines[i][0]
        # Much slower version of Euclidean distance
        if np.sqrt((l[0]-l[2])**2 + (l[1]-l[3])**2) > 50:
            # You might have to tweak the threshold as well for other images
            cv2.line(image, (l[0], l[1]), (l[2], l[3]), (255, 255, 255), 3, 
                     cv2.LINE_AA)
cv2.imwrite("result.png", image)

Output:

As you can see, the lines aren't completely removed in the top image so I am leaving the tweaking part to you. Hope it helps!

Answer 3

I'd like to use this answer box to make a few comments.

First off, its way easier to see progress, if you can easily see what the output looks like visually. With that in mind, here is an update to your code with an emphasis on viewing interim results. I'm using VS Studio Community 2017, and OpenCV version 4.0.1 (64bit) in Win10 for anyone who wants to repeat this exercise. There were a few routines used that required updates for OpenCV 4...

#include "pch.h"
#include <iostream>
#include <opencv2/opencv.hpp>
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>

int main()
{    
    cv::Mat img = cv::imread("0zx9Q.png", cv::IMREAD_GRAYSCALE );        // --> Contour size = 0x000000e7 hex (231 each)
    // cv::Mat img = cv::imread("0zx9Q.png", cv::IMREAD_REDUCED_GRAYSCALE_2); // --> Contour size = 0x00000068 hex (104 each)
    // cv::Mat img = cv::imread("0zx9Q.png", cv::IMREAD_REDUCED_GRAYSCALE_4); // --> Contour size = 0x0000001f hex (31 each)
    // cv::Mat img = cv::imread("0zx9Q.png", cv::IMREAD_REDUCED_GRAYSCALE_8); // --> Contour size = 0x00000034 hex (52 each)

    if (!img.data)                              // Check for invalid input
    {
        std::cout << "Could not open or find the image" << std::endl;
        return -1;
    }

    // cv::namedWindow("Display Window - GrayScale Image", cv::WINDOW_NORMAL);   // Create a window for display.
    // cv::imshow("Display Window - GrayScale Image", img);                      // Show our image inside it.
    // cv::waitKey(0);                                                           // Wait for a keystroke in the window

    cv::Mat imgOriginal = cv::imread("0zx9Q.png", cv::IMREAD_UNCHANGED);
    cv::namedWindow("Display Window of Original Document", cv::WINDOW_NORMAL);   // Create a window for display.

    cv::Mat grad;
    cv::Mat morphKernel = cv::getStructuringElement(cv::MORPH_RECT, cv::Size(25, 25));
    // MORPH_ELLIPSE, contourSize: 0x00000005    when 60,60... but way slow...
    // MORPH_ELLIPSE, contourSize: 0x00000007    when 30,30...  
    // MORPH_ELLIPSE, contourSize: 0x00000007    when 20,20... 
    // MORPH_ELLIPSE, contourSize: 0x0000000a    when 15,15...
    // MORPH_ELLIPSE, contourSize: 0x0000007a    when 5,5...
    // MORPH_ELLIPSE, contourSize: 0x000000e7    when 3,3 and IMREAD_GRAYSCALE 
    // MORPH_CROSS,   contourSize: 0x0000008e    when 5,5 
    // MORPH_CROSS,   contourSize: 0x00000008    when 25,25 
    // MORPH_RECT,    contourSize: 0x00000007    when 25,25 

    cv::morphologyEx(img, grad, cv::MORPH_GRADIENT, morphKernel);

    cv::Mat res;
    cv::threshold(grad, res, 0, 255, cv::THRESH_BINARY | cv::THRESH_OTSU);

    // find contours
    cv::Mat mask = cv::Mat::zeros(res.size(), CV_8UC1);
    std::vector<std::vector<cv::Point>> contours;
    std::vector<cv::Vec4i> hierarchy;
    cv::findContours(res, contours, cv::RETR_EXTERNAL, cv::CHAIN_APPROX_SIMPLE);

    int contourSize = contours.size();
    std::cout << " There are a total of " << contourSize << " contours. \n";
    for (int i = 0; i < contourSize; i++) {
        cv::Mat approx;
        double peri = cv::arcLength(contours[i], true);
        cv::approxPolyDP(contours[i], approx, 0.04 * peri, true);
        int num_vertices = approx.rows;
        std::cout << " Contour # " << i << " has " << num_vertices << " vertices.\n";
        if (num_vertices == 4) {
            cv::Rect rect = cv::boundingRect(contours[i]);
            cv::rectangle(imgOriginal, rect, cv::Scalar(255, 0, 0), 4);
        }
    }

    cv::imshow("Display Window of Original Document", imgOriginal);           // Show our image inside it.
    cv::waitKey(0);                                                           // Wait for a keystroke in the window
}

With that said, the parameters for getStructuringElement() matter huge. I spent a lot of time trying different choices, with very mixed results. And it turns out there are a whole lot of findContours() responses that don't have four vertices. I suspect the whole findContours() approach is probably flawed. I often would get false rectangles identified around text characters in words and phrases. Additionally the lighter lines surrounding some boxed areas would be ignored.

Instead, I think I'd be looking hard at straight line detection, via techniques discussed here, if such a response exists for a C++ and not python. Perhaps here, or here? I hoping line detection techniques would ultimately get better results. And hey, if the documents / images selected always include a white background, it would be pretty easy to solid rectangle them OUT of the image, via LineTypes: cv::FILLED

Info here provided, not as an answer to the posted question but as a methodology to visually determine success in the future.