Improving Flood-fill Result Under Sensible Light Conditions with OpenCV

Question

I am trying to do contour detection on image with tools, due to the metal nature of the tools they tend to reflect the light and produce glares. In order to limit this effect I am using floodFill, but this is very sensitive to the parameters.

For example here is my original image:

and this is the flooded one

Which looks great, not I try with a second image:

As you can see on the flooded image some tools are not correctly handled and as such the contour detection will fail to give a good result.

Here is the updated version:

import numpy as np
import cv2


def getFilteredLabelIndex(stats, widthLowLimit=50, heightLowLimit=50, areaLowLimit=7000):
    ret = []
    for i in range(1, stats.shape[0]):
        # extract the connected component statistics for the current label
        w = stats[i, cv2.CC_STAT_WIDTH]
        h = stats[i, cv2.CC_STAT_HEIGHT]
        area = stats[i, cv2.CC_STAT_AREA]

        keepWidth = w > widthLowLimit
        keepHeight = h > heightLowLimit
        keepArea = area > areaLowLimit

        if all((keepWidth, keepHeight, keepArea)):
            ret.append(i)

    return ret

# load our input image, convert it to grayscale, and blur it slightly
impath = "q8djf.png"
originalImage = cv2.imread(impath)

birdEye = originalImage 

seed = (35, 35)
originalImage = np.maximum(originalImage, 10)
foreground = originalImage.copy()

# Use floodFill for filling the background with black color
cv2.floodFill(foreground, None, seed, (0, 0, 0),
              loDiff=(5, 5, 5), upDiff=(5, 5, 5))
              
cv2.imshow("foreground", foreground)

gray = cv2.cvtColor(foreground, cv2.COLOR_BGR2GRAY)
cv2.imshow("gray", gray)

threshImg = cv2.threshold(gray, 1, 255, cv2.THRESH_BINARY)[1]
cv2.imshow("threshImg", threshImg)

(numLabels, labels, stats, centroids) = cv2.connectedComponentsWithStats(
    threshImg, 4, cv2.CV_32S)

filteredIdx = getFilteredLabelIndex(stats)

for i in filteredIdx:
    componentMask = (labels == i).astype("uint8") * 255

    kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (5, 5))
    componentMask = cv2.dilate(componentMask, kernel, iterations=3)

    ctrs, _ = cv2.findContours(
        componentMask, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)
    ctrs = sorted(ctrs, key=cv2.contourArea, reverse=True)
    cntrs = max(ctrs, key=cv2.contourArea)

    cv2.drawContours(birdEye, [cntrs], -1, (255, 0, 255), 3)
    cv2.imshow("contour", birdEye)

cv2.imshow("original contour", birdEye)
cv2.waitKey(0)
cv2.destroyAllWindows()

Any suggestion would be welcome.

control the lighting conditions or use much more advanced techniques like deep learning instance segmentation or object detection. Intermediate advanced techniques could be superpixel segmenration shape matching, keypoint matching, but typically all of them have some limirations together with their advantages. — Micka, Apr 16 '21 at 15:25
If your background is always as nice as in those rwo images you could try gradient or edge extraction and clustering the results to estimate objects. You dan then use convex-hull to estimate the contour. — Micka, Apr 16 '21 at 15:28
Hi Micka, thanks for the tips. I am not sure how deep learning could help since there is no way to know the shape of the tool. It's not like identifying a face or a car for which you can train a network. In my case the tools could be of any nature (please correct me if I am wrong). As for your second point yes, the background will alway be as nice it's a plain sheet of A4 paper. In fact i am using cv2.connectedComponentsWithStats image to identify the different objects but again this will cut out part of the objects depending on the light conditions. — user2097439, Apr 16 '21 at 16:50
Deep learning can generalize very well and is one a few techniques that works well in uncontrolled real life situations. Your current approach is very simple and so you would have to control the environment, or find a way to automatically normalize the image or the parameters (according to the image) or to take a lot of assumptions. — Micka, Apr 16 '21 at 17:59
@Micka do you suggest I try to train a DNN to find out the objects or to remove the background or to directly find the contours ? I guess finding a suitable set of images to train on will be quite difficult to be honest. Wandering if a YoloV5 would work out of the box. — user2097439, Apr 16 '21 at 20:06
really depends on your actual use-case, variability of environment, etc.. I strongly guess that you will have to earn that experience yourself or to hire an experienced computer vision expert. I mostly wanted to tell that computer vision for non-toy problems (or controlled environment) is hard. And yes, collecting and annotating data for deep learning is hard, too. If you want to try the deep learning way, either object detection or instance segmentation or semantic segmentation could be points to start with. — Micka, Apr 16 '21 at 20:40
The good thing might be: You will need lots of annotated data for testing anyways, so you could go both ways: 1. write cv algorithms that solve like 60-90% of the samples and use that to semi-auto-annotate your test data and 2. later use the data to start DNN training approaches. — Micka, Apr 16 '21 at 20:42

score 1 · Answer 1 · answered Apr 16 '21 at 21:53

Sharpening the input image may help.

The sharpening operation enlarges the difference between the object and the background.

I am not sure how robust it's going to be for other images, but it's improve the robustness of the current solution.

I used the sharpening solution from the following post: How can I sharpen an image in OpenCV?, but with different parameters for stronger sharpening.

The following code demonstrates the solution:

# https://stackoverflow.com/questions/4993082/how-can-i-sharpen-an-image-in-opencv
# Sharpen the image
blur = cv2.GaussianBlur(birdEye, (0, 0), 3)
sharp_foreground = cv2.addWeighted(birdEye, 2, blur, -1, 0);
sharp_foreground = np.maximum(sharp_foreground, 10)

# Use floodFill for filling the background with black color
# Use loDiff and upDiff (10, 10, 10) instead of (5, 5, 5) for exaggerating the problem
cv2.floodFill(sharp_foreground, None, seed, (0, 0, 0),
              loDiff=(10, 10, 10), upDiff=(10, 10, 10))

With sharpening (sharp_foreground):

Without sharpening:

Note:
The example uses loDiff=(10, 10, 10) and upDiff=(10, 10, 10) just for demonstration.
Try keeping the values lower.

Improving Flood-fill Result Under Sensible Light Conditions with OpenCV

1 Answers1