2

I am working on a computer vision problem, in which one step in the problem is find the locations where the objects are close to each other. Example, in the image below I am interesting in finding the regions marked in gray.

Input :

enter image description here

Output :

enter image description here

My current approach is to first invert the image, followed by morphological gradient follower by erosion, then removing some non-interesting contours. Script is as follows:

img = cv2.imread('mask.jpg', 0)
img = (255 - img)

kernel = np.ones((11,11), np.uint8) 
gradient = cv2.morphologyEx(img, cv2.MORPH_GRADIENT, kernel)

kernel = np.ones((5,5), np.uint8) 
img_erosion = cv2.erode(gradient, kernel, iterations=3) 

img_erosion[img_erosion > 200] = 255
img_erosion[img_erosion <= 200] = 0

def get_contours(mask):
    contours, hierarchy = cv2.findContours(mask,cv2.RETR_TREE,cv2.cv2.CHAIN_APPROX_NONE)
    return contours

cnts = get_contours(img_erosion)

img_new = np.zeros_like(img_erosion)
img_h, img_w = img_erosion.shape
for i in cnts:
    if cv2.contourArea(i) > 30:
        print(cv2.boundingRect(i), cv2.contourArea(i))
        x, y, h, w = cv2.boundingRect(i)
        if h/w > 5 or w/h > 5 or cv2.contourArea(i) > 100:  ## Should be elongated 
            if (x - 10 > 0) and (y - 10 > 0): ## Check if near top or left edge
                if (img_w - x > 10) and (img_h - y > 10): ## Check if near bottom or right edge

                    cv2.drawContours(img_new, [i], -1, (255,255,255), 2)
kernel = np.ones((3,3), np.uint8) 
img_new = cv2.dilate(img_new, kernel, iterations=2)
plt.figure(figsize=(6,6))
plt.imshow(img_new)

Result is:

enter image description here

But, using this approach, I am required to adjust many parameters, and it's failing in many cases when the orientation is different or edges are slightly far, or if "L" shaped edges etc.

I am new to image processing, is there any other method that can help me to solve this task efficiently?

Edit : Attaching some more images

enter image description here

enter image description here

(Mostly rectangular polygons, but lot of variation in size and relative positions)

Jeru Luke
  • 20,118
  • 13
  • 80
  • 87
  • 1
    Cool question! Do you have some examples of images that cause your algorithm difficulty please? – Mark Setchell Feb 09 '19 at 10:18
  • 1
    I don't understand the problem, what do you mean by "finding the locations where the objects are next to each other"? The gray boxes in the example images aren't near each other, and you're only detecting one of three gray boxes in your example output. Can you expand a bit more on expected output? I'm also unsure on the input, you give three total examples; the first is a ternary image (three colors), the other two are binary images. What is the expected range of possibilities? – alkasm Feb 09 '19 at 10:57
  • Sorry for providing the confusing images. You can assume the input to be just a grayscale image (white as the objects, 255 value). Regarding the output, i am interested in finding the regions where the objects are close (It can be a threshold of 5-10 pixels). The first image I gave is confusing you i guess. So the input there is black and white image. I have added gray color as region which I am looking to find (This is the input image for that first image. https://imgur.com/a/9ZB7t9t). Let me know if I am clear or not. Thanks. – stupid_cannon Feb 09 '19 at 11:12
  • @AlexanderReynolds Edited the post. Now input and output should be clear I guess. – stupid_cannon Feb 09 '19 at 11:15
  • Oh! Now I understand, the gray was meant to show how those pieces are near each other. Cool. Okay, so you're looking to find just the places where they're close (i.e. the areas highlighted in gray), or do you actually want the contours or blobs merged together in some way? – alkasm Feb 09 '19 at 11:15
  • I want to identify those regions only, for some other processing. Not going to merge the blobs. – stupid_cannon Feb 09 '19 at 11:19
  • Do you need all "connecting regions" in a single mask together, in a bunch of separate masks, or something else? Do you need them to reference which blobs they connect somehow? I'm not sure why you're flipping and doing erosion---the more sensible thing to do here is dilation :) – alkasm Feb 09 '19 at 11:29
  • I will be using them as a separate mask. Or if i can just get the contours of these "connecting regions" that is also fine. Though finally I am creating a separate mask of these connecting regions (Like in the result image (the 3rd image)) – stupid_cannon Feb 09 '19 at 11:33
  • Is there some global threshold in terms of distance between objects and you want to find all points where the distance is less than the threshold? Or do you need to know set of points A mark objects 2 pixels apart, set of points B mark objects 5 pixels apart and set of points C mark objects 10 pixels apart? – Mark Setchell Feb 09 '19 at 17:05
  • Yes. It's a threshold based on how close two objects are. So if objects are more than 15 pixels apart, I am not interested in those. Only when it's less than 15 pixels, I want to find the region where they are close. I am not looking for the exact values of how far the objects are, I am fine if there is some error (I guess I can work with around 20/30% error also) – stupid_cannon Feb 09 '19 at 19:15

1 Answers1

3

The best way to do this probably is via the Stroke Width Transform. This isn't in OpenCV, though it is in a few other libraries and you can find some implementations floating around the internet. The stroke width transform finds the minimum width between the nearest edges for each pixel in the image. See the following figure from the paper:

Stroke Width Transform example

Thresholding this image then tells you where there are edges separated by some small distance. E.g., all the pixels with values < 40, say, are between two edges that are separated by less than 40 pixels.

So, as is probably clear, this is pretty close to the answer that you want. There would be some additional noise here, like you'd also get values that are between the square ridges on the edge of your shapes...which you'd have to filter out or smooth out (contour approximation would be a simple way to clean them up as a preprocessing step, for example).

However, while I do have a prototype SWT programmed, it's not a very good implementation, and I haven't really tested it (and actually forgot about it for a few months.......maybe a year) so I'm not going to put it out right now. But, I do have another idea that is a little simpler and doesn't necessitate reading a research paper.


You have multiple blobs in your input image. Imagine if you had each one separately in its own image, and you grew each blob by however much distance you're willing to put between them. If you grew each blob by say 10 pixels, and they overlap, then they'd be within 20 pixels of each other. However this doesn't give us the full overlap region, just a portion of where the two expanded blobs overlapped. A different, but similar way to measure this is if the blobs grew by 10 pixels, and overlapped, and furthermore overlapped the original blobs before they were expanded, then the two blobs are within 10 pixels of each other. We're going to use this second definition to find nearby blobs.

def find_connection_paths(binimg, distance):

    h, w = binimg.shape[:2]
    overlap = np.zeros((h, w), dtype=np.int32)
    overlap_mask = np.zeros((h, w), dtype=np.uint8)
    kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (distance, distance))

    # grows the blobs by `distance` and sums to get overlaps
    nlabels, labeled = cv2.connectedComponents(binimg, connectivity=8)
    for label in range(1, nlabels):
        mask = 255 * np.uint8(labeled == label)
        overlap += cv2.dilate(mask, kernel, iterations=1) // 255
    overlap = np.uint8(overlap > 1)

    # for each overlap, does the overlap touch the original blob?
    noverlaps, overlap_components = cv2.connectedComponents(overlap, connectivity=8)
    for label in range(1, noverlaps):
        mask = 255 * np.uint8(overlap_components == label)
        if np.any(cv2.bitwise_and(binimg, mask)):
            overlap_mask = cv2.bitwise_or(overlap_mask, mask)
    return overlap_mask

Connecting regions

Now the output isn't perfect---when I expanded the blobs, I expanded them outwardly with a circle (the dilation kernel), so the connection areas aren't exactly super clear. However, this was the best way to ensure it'll work on things of any orientation. You could potentially filter this out/clip it down. An easy way to do this would be to get each connecting piece (shown in blue), and repeatedly erode it down a pixel until it doesn't overlap the original blob. Actually OK let's add that:

def find_connection_paths(binimg, distance):

    h, w = binimg.shape[:2]
    overlap = np.zeros((h, w), dtype=np.int32)
    overlap_mask = np.zeros((h, w), dtype=np.uint8)
    overlap_min_mask = np.zeros((h, w), dtype=np.uint8)
    kernel_dilate = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (distance, distance))

    # grows the blobs by `distance` and sums to get overlaps
    nlabels, labeled = cv2.connectedComponents(binimg)
    for label in range(1, nlabels):
        mask = 255 * np.uint8(labeled == label)
        overlap += cv2.dilate(mask, kernel_dilate, iterations=1) // 255
    overlap = np.uint8(overlap > 1)

    # for each overlap, does the overlap touch the original blob?
    noverlaps, overlap_components = cv2.connectedComponents(overlap)
    for label in range(1, noverlaps):
        mask = 255 * np.uint8(overlap_components == label)
        if np.any(cv2.bitwise_and(binimg, mask)):
            overlap_mask = cv2.bitwise_or(overlap_mask, mask)

    # for each overlap, shrink until it doesn't touch the original blob
    kernel_erode = cv2.getStructuringElement(cv2.MORPH_RECT, (3, 3))
    noverlaps, overlap_components = cv2.connectedComponents(overlap_mask)
    for label in range(1, noverlaps):
        mask = 255 * np.uint8(overlap_components == label)
        while np.any(cv2.bitwise_and(binimg, mask)):
            mask = cv2.erode(mask, kernel_erode, iterations=1)
        overlap_min_mask = cv2.bitwise_or(overlap_min_mask, mask)

    return overlap_min_mask

Minimum overlapping regions

Of course, if you still wanted them to be a little bigger or smaller you could do whatever you like with them, but this looks pretty close to your requested output so I'll leave it there. Also, if you're wondering, I have no idea where the blob on the top right went. I can take another pass at this last piece later. Note that the last two steps could be combined; check if there's overlap, if it is, cool---shrink it down and store it in the mask.

alkasm
  • 22,094
  • 5
  • 78
  • 94