Python/OpenCV — Centroid Determination in Bacterial Clusters

Question

I am currently working on developing an algorithm to determine centroid positions from (Brightfield) microscopy images of bacterial clusters. This is currently a major open problem in image processing.

This question is a follow-up to: Python/OpenCV — Matching Centroid Points of Bacteria in Two Images.

Currently, the algorithm is effective for sparse, spaced-out bacteria. However, it becomes totally ineffective when the bacteria become clustered together.

In these images, notice how the bacterial centroids are located effectively.

Bright-Field Image #1

Bright-Field Image #2

Bright-Field Image #3

However, the algorithm fails when the bacteria cluster at varying levels.

Bright-Field Image #4

Bright-Field Image #5

Bright-Field Image #6

Bright-Field Image #7

Bright-Field Image #8

Original Images

Bright-Field Image #1

Bright-Field Image #2

Bright-Field Image #3

Bright-Field Image #4

Bright-Field Image #5

Bright-Field Image #6

Bright-Field Image #7

Bright-Field Image #8

I'd like to optimize my current algorithm so it's more robust for these type of images. This is the program I'm running.

import cv2
import numpy as np
import os

kernel = np.array([[0, 0, 1, 0, 0],
                   [0, 1, 1, 1, 0],
                   [1, 1, 1, 1, 1],
                   [0, 1, 1, 1, 0],
                   [0, 0, 1, 0, 0]], dtype=np.uint8)


def e_d(image, it):
    image = cv2.erode(image, kernel, iterations=it)
    image = cv2.dilate(image, kernel, iterations=it)
    return image


path = r"(INSERT IMAGE DIRECTORY HERE)"
img_files = [file for file in os.listdir(path)]


def segment_index(index: int):
    segment_file(img_files[index])


def segment_file(img_file: str):
    img_path = path + "\\" + img_file
    print(img_path)
    img = cv2.imread(img_path)
    img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    # Applying adaptive mean thresholding
    th = cv2.adaptiveThreshold(img, 255, cv2.ADAPTIVE_THRESH_MEAN_C, cv2.THRESH_BINARY_INV, 11, 2)
    # Removing small noise
    th = e_d(th.copy(), 1)

    # Finding contours with RETR_EXTERNAL flag and removing undesired contours and
    # drawing them on a new image.
    cnt, hie = cv2.findContours(th, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
    cntImg = th.copy()
    for contour in cnt:
        x, y, w, h = cv2.boundingRect(contour)
        # Eliminating the contour if its width is more than half of image width
        # (bacteria will not be that big).
        if w > img.shape[1] / 2:
            continue
        cntImg = cv2.drawContours(cntImg, [cv2.convexHull(contour)], -1, 255, -1)

    # Removing almost all the remaining noise.
    # (Some big circular noise will remain along with bacteria contours)
    cntImg = e_d(cntImg, 3)

    # Finding new filtered contours again
    cnt2, hie2 = cv2.findContours(cntImg, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)

    # Now eliminating circular type noise contours by comparing each contour's
    # extent of overlap with its enclosing circle.
    finalContours = []  # This will contain the final bacteria contours
    for contour in cnt2:
        # Finding minimum enclosing circle
        (x, y), radius = cv2.minEnclosingCircle(contour)
        center = (int(x), int(y))
        radius = int(radius)

        # creating a image with only this circle drawn on it(filled with white colour)
        circleImg = np.zeros(img.shape, dtype=np.uint8)
        circleImg = cv2.circle(circleImg, center, radius, 255, -1)

        # creating a image with only the contour drawn on it(filled with white colour)
        contourImg = np.zeros(img.shape, dtype=np.uint8)
        contourImg = cv2.drawContours(contourImg, [contour], -1, 255, -1)

        # White pixels not common in both contour and circle will remain white
        # else will become black.
        union_inter = cv2.bitwise_xor(circleImg, contourImg)

        # Finding ratio of the extent of overlap of contour to its enclosing circle.
        # Smaller the ratio, more circular the contour.
        ratio = np.sum(union_inter == 255) / np.sum(circleImg == 255)

        # Storing only non circular contours(bacteria)
        if ratio > 0.55:
            finalContours.append(contour)

    finalContours = np.asarray(finalContours)

    # Finding center of bacteria and showing it.
    bacteriaImg = cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)

    for bacteria in finalContours:
        M = cv2.moments(bacteria)
        cx = int(M['m10'] / M['m00'])
        cy = int(M['m01'] / M['m00'])

        bacteriaImg = cv2.circle(bacteriaImg, (cx, cy), 5, (0, 0, 255), -1)

    cv2.imshow("bacteriaImg", bacteriaImg)
    cv2.waitKey(0)

# Segment Each Image
for i in range(len(img_files)):
    segment_index(i)

Ideally I would like at least to improve on a couple of the posted images.

Answer 1

The mask is always the weak point in identifying objects, and the most important step. This will improve identifying images with high numbers of bacteria. I have modified your e_d function by adding an OPEN and another ERODE pass with the kernal, and changed the it (number of iterations) variable (to 1, 2 instead of 1,3) for your code to do this. This is by no means a finished effort, but I hope it will give you an idea of what you might try to enhance it further. I used the images you provided, and since they already have a red dot, this may be interfering with my result images... but you can see it is able to identify more bacteria on most. Some of my results show two dots, and the image with only one bacteria, I missed it, each quite possibly because it was already marked. Try it with the raw images and see how it does.

Also, since the bacteria are relatively uniform in both size and shape, I think you could work with the ratio and/or average of height to width of each bacteria to filter out the extreme shapes (small or large) and the skinny, long shapes too. You can measure enough bacteria to see what is the average contour length, or height and width, or height/width ratio, etc., to find reasonable tolerances rather than the proportion to the image size itself. Another suggestion, would be to rethink how you are masking the images all together, possibly to try it in two steps. One to find the boundary of the long shape containing the bacteria, and then to find the bacteria within it. This assumes all of your images will be similar to these, and if that is so, it may help to eliminate the stray hits outside of this boundary, that are never bacteria.

#!usr/bin/env python
# https://stackoverflow.com/questions/63182075/python-opencv-centroid-determination-in-bacterial-clusters
import cv2
import numpy as np
import os

kernel = np.array([[0, 0, 1, 0, 0],
                   [0, 1, 1, 1, 0],
                   [1, 1, 1, 1, 1],
                   [0, 1, 1, 1, 0],
                   [0, 0, 1, 0, 0]], dtype=np.uint8)


def e_d(image, it):
    print(it)
    image = cv2.erode(image, kernel, iterations=it)
    image = cv2.dilate(image, kernel, iterations=it)
    image = cv2.morphologyEx(image, cv2.MORPH_OPEN, kernel, iterations = 1)
    image = cv2.morphologyEx(image, cv2.MORPH_ERODE, kernel, iterations = 1)
    return image


#path = r"(INSERT IMAGE DIRECTORY HERE)"
path = r"E:\stackimages"
img_files = [file for file in os.listdir(path)]


def segment_index(index: int):
    segment_file(img_files[index])


def segment_file(img_file: str):
    img_path = path + "\\" + img_file
    print(img_path)
    head, tail = os.path.split(img_path)
    img = cv2.imread(img_path)
    img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    cv2.imshow("bacteriaImg-1", img)
    cv2.waitKey(0)
    # Applying adaptive mean thresholding
    th = cv2.adaptiveThreshold(img, 255, cv2.ADAPTIVE_THRESH_MEAN_C, cv2.THRESH_BINARY_INV, 11, 2)
    # Removing small noise
    th = e_d(th.copy(), 1)

    # Finding contours with RETR_EXTERNAL flag and removing undesired contours and
    # drawing them on a new image.
    cnt, hie = cv2.findContours(th, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
    cntImg = th.copy()
    for contour in cnt:
        x, y, w, h = cv2.boundingRect(contour)
        # Eliminating the contour if its width is more than half of image width
        # (bacteria will not be that big).
        
        if w > img.shape[1] / 2:
            continue
  
        else:
           
            cntImg = cv2.drawContours(cntImg, [cv2.convexHull(contour)], -1, 255, -1)


    # Removing almost all the remaining noise.
    # (Some big circular noise will remain along with bacteria contours)
    cntImg = e_d(cntImg, 2)
    cv2.imshow("bacteriaImg-2", cntImg)
    cv2.waitKey(0)

    # Finding new filtered contours again
    cnt2, hie2 = cv2.findContours(cntImg, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)

    # Now eliminating circular type noise contours by comparing each contour's
    # extent of overlap with its enclosing circle.
    finalContours = []  # This will contain the final bacteria contours
    for contour in cnt2:
        # Finding minimum enclosing circle
        (x, y), radius = cv2.minEnclosingCircle(contour)
        center = (int(x), int(y))
        radius = int(radius)

        # creating a image with only this circle drawn on it(filled with white colour)
        circleImg = np.zeros(img.shape, dtype=np.uint8)
        circleImg = cv2.circle(circleImg, center, radius, 255, -1)

        # creating a image with only the contour drawn on it(filled with white colour)
        contourImg = np.zeros(img.shape, dtype=np.uint8)
        contourImg = cv2.drawContours(contourImg, [contour], -1, 255, -1)

        # White pixels not common in both contour and circle will remain white
        # else will become black.
        union_inter = cv2.bitwise_xor(circleImg, contourImg)

        # Finding ratio of the extent of overlap of contour to its enclosing circle.
        # Smaller the ratio, more circular the contour.
        ratio = np.sum(union_inter == 255) / np.sum(circleImg == 255)

        # Storing only non circular contours(bacteria)
        if ratio > 0.55:
            finalContours.append(contour)

    finalContours = np.asarray(finalContours)

    # Finding center of bacteria and showing it.
    bacteriaImg = cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)

    for bacteria in finalContours:
        M = cv2.moments(bacteria)
        cx = int(M['m10'] / M['m00'])
        cy = int(M['m01'] / M['m00'])

        bacteriaImg = cv2.circle(bacteriaImg, (cx, cy), 5, (0, 0, 255), -1)

    cv2.imshow("bacteriaImg", bacteriaImg)
    cv2.waitKey(0)


# Segment Each Image
for i in range(len(img_files)):
    segment_index(i)

Answer 2

Here's some code that you can try and see if it works for you. It uses an alternative approach to segmenting images. You can fiddle around with parameters to see what combination gives you most acceptable results.

import numpy as np
import cv2
import matplotlib.pyplot as plt


# Adaptive threshold params
gw = 11
bs = 7
offset = 5

bact_aspect_min = 2.0
bact_aspect_max = 10.0
bact_area_min = 20 # in pixels
bact_area_max = 1000

url = "/path/to/image"
img_color = cv2.imread(url)
img = cv2.cvtColor(img_color, cv2.COLOR_BGR2GRAY)
rows, cols = img.shape

img_eq = img.copy()
cv2.equalizeHist(img, img_eq)

img_blur = cv2.medianBlur(img_eq, gw)
th = cv2.adaptiveThreshold(img_blur, 255, cv2.ADAPTIVE_THRESH_MEAN_C, cv2.THRESH_BINARY_INV, bs, offset)

_, contours, hier = cv2.findContours(th.copy(), cv2.RETR_CCOMP, cv2.CHAIN_APPROX_SIMPLE)
for i in range(len(contours)):
    # Filter closed contours
    rect = cv2.minAreaRect(contours[i])
    area = cv2.contourArea(contours[i])
    (x, y), (width, height), angle = rect
    if min(width, height) == 0:
        continue
        
    aspect_ratio = max(width, height) / min(width, height)
    
    if hier[0][i][3] != -1 and \
    bact_aspect_min < aspect_ratio < bact_aspect_max and \
    bact_area_min < area < bact_area_max:
        M = cv2.moments(contours[i])
        cx = int(M['m10'] / M['m00'])
        cy = int(M['m01'] / M['m00'])
        img_color = cv2.circle(img_color, (cx, cy), 3, (255, 0, 0), cv2.FILLED)

plt.imshow(img_color)

It seems that your bacterias seem fused/overlapped in most of the images and it is extremely hard to gauge their size when they are fused and to separate them. Best way is to run this code snippet in Jupyter/ipywidgets with a range of parameter values and see what works best. Good luck!

EDIT 1

I have updated the code to use a slight bit different technique and idea. Basically using l2 contours (holes) to ascertain bacteria, this is much more in line with the shape of the bacteria. You can, again, fiddle around with the parameters to see what works best. Set of parameters in the code gave me satisfactory results. You may want to filter the image a bit more to remove false positives.

Couple of other tricks can be used in addition to the one in the latest code:

1. Try out ADAPTIVE_THRESH_GAUSSIAN_C
2. Try equalized image without blurring
3. Use level 1 contours along with level 2
4. Use different size constraints for l1 and l2 contours.

I think a combination of all these should provide you with a pretty decent result.