Trim (noised) whitespaces in images using Python

Question

Say that I have an image as shown below:

As you can see, the image contains blank/white space (noised with small green dots) in addition to a big green square (the square is randomly put in the left-bottom corner). What I want to do is to crop the image/trim the white space to only get the big green square in the end.
Thanks to Stackoverflow community, I found a way to trim the white space (if it does not contain noise) in this answer; here is the code that I used:

from PIL import Image, ImageChops

def trim(im):
    bg = Image.new(im.mode, im.size, im.getpixel((0,0)))
    diff = ImageChops.difference(im, bg)
    diff = ImageChops.add(diff, diff, 2.0, -100)
    diff.show()
    bbox = diff.getbbox()
    if bbox:
        return im.crop(bbox)

im = Image.open("noised.jpg")
im = trim(im)
im.save('output.png')

Since the blank/white space contains noise, the cropped image that I got looks like this :

So the script cropped the image when it met the first non white (noised) pixel.
My question is: Is there a way to denoise the white space without affecting the image (green square) or to directly trim the white space even if it is noised?
Thank you for your help!

Answer 1

My suggestion is using OpenCV and Numpy libraries purely.

Assuming your original image has the filename of "input_green.jpg". The method is based on contour detection algorithm.

First of all, import and load the input image, and make the relevant gray image object.

import cv2
import numpy as np
img = cv2.imread("input_green.jpg", 1)
image_gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)

Secondly, apply image blur and binarization methods, and then detect any contours.

image_gray = 255 - cv2.GaussianBlur(image_gray, (3,3), 0)
# clean the image based on gray value
image_gray = np.where(image_gray <= 50, 0, image_gray)
image_gray = np.where(image_gray > 50, 255, image_gray)
_,image_gray = cv2.threshold(image_gray,0,255,cv2.THRESH_BINARY+cv2.THRESH_OTSU)
# draw a bounding box around the whole image
h, w = image_gray.shape[:2]
cv2.rectangle(image_gray, (0,0), (w, h), 0, 5)

Thirdly, detect and sort the contours by area

_,contours,_ = cv2.findContours(image_gray,cv2.RETR_LIST,cv2.CHAIN_APPROX_SIMPLE)
contours_sorted = sorted(contours, key=cv2.contourArea, reverse=True)

Finally, grab the biggest contour as the green rectangle on the image, and crop it from the original image.

green_rect = contours_sorted[0]
img_copy = img.copy()
cv2.drawContours(img_copy, [green_rect], 0, (0, 0, 255), 2)
cv2.imshow("result", img_copy)
#cv2.imwrite("result_green_rect.jpg", img_copy)
cv2.waitKey()

Tempory result:

# crop the green rect
(x, y, w, h) = cv2.boundingRect(green_rect)
crop_result = img[y:y+h, x:x+w]
cv2.imshow("crop_result", crop_result)
#cv2.imwrite("result_green_rect_crop.jpg", crop_result)
cv2.waitKey()

Cropped Result:

Answer 2

This can be done with OpenCV by detecting the green areas, then picking out the biggest area. There are several ways to go about detecting the green area, including cv2.inRange and cv2.threshold.

1. Identifying Green Regions

cv2.inRange

With inRange, you can identify colors within a certain range. For example:

lower_bound = (0,100,0)
upper_bound = (10,255,10)

So pixels with colors between lower_bound and upper_bound can be identified to create a mask with

mask = cv2.inRange(im, lower_bound, upper_bound)

Here's the mask of green areas:

cv2.threshold

Similarly, thresholding will create a mask of green areas. First, turn the image into grayscale.

imgray = cv2.cvtColor(im, cv2.COLOR_RGB2GRAY)

However, if you threshold this image, it will identify the white areas, so we want the inverse of the threshold found by cv2.THRESH_BINARY_INV

ok, thresh = cv2.threshold(imgray, 200, 255, cv2.THRESH_BINARY_INV)

Here is the threshold image:

As can be seen, thresh and mask identify the green regions.

2. Contours

Once we have the mask or threshold image, we can identify the white regions by looking for contours. I will use the mask image (thresh could be used just as well).

(im2, contours, hierarchy)  = cv2.findContours(mask, cv2.RETR_EXTERNAL,
                                cv2.CHAIN_APPROX_SIMPLE)

We specifically want the contours, which give a set of points that can be used to outline the green areas. We want the contour that creates the largest contour area, which we can find by first putting them in order from largest to smallest, then taking the first one.

ordered_cnts = sorted(contours, key=cv2.contourArea, reverse=True)
largest_cnt = ordered_cnts[0]

largest_cnt is the following set of points:

[[[  0 701]]
 [[  0 999]]
 [[298 999]]
 [[298 701]]
 [[289 701]]
 [[288 702]]
 [[287 701]]]

Those points can be used to outline the green box in the bottom left of the image. We just want a single rectangle, so we can outline the entire contour by finding the smallest rectangle that would surround all those points.

rect = cv2.minAreaRect(largest_cnt)
box = cv2.boxPoints(rect)

box gives a list of points that are the four corners of rect. We can use numpy to convert to integer points and get the limits of the box to crop the image.

box = np.array(box, dtype=int)
x0, y0 = np.min(box,axis=0)
x1, y1 = np.max(box,axis=0)
crop = im[y0:y1, x0:x1]

The crop image:

Combined Code

lower_bound = (0,100,0)
upper_bound = (10,255,10)
mask = cv2.inRange(im, lower_bound, upper_bound)
(im2, cnts, hierarchy) = cv2.findContours(mask, cv2.RETR_EXTERNAL,
                                    cv2.CHAIN_APPROX_SIMPLE)
ordered_cnts = sorted(cnts, key=cv2.contourArea, reverse=True)
largest_cnt = ordered_cnts[0]
rect = cv2.minAreaRect(largest_cnt)
box = cv2.boxPoints(rect)
box = np.array(box, dtype=int)
x0, y0 = np.min(box,axis=0)
x1, y1 = np.max(box,axis=0)
crop = im[y0:y1, x0:x1]