Python/OpenCV — Matching Centroid Points of Bacteria in Two Images

Question

I am working on an algorithm to match the centroids of bacteria using computer vision.

As I'm an undergraduate and beginner to computer vision, I do not have code specifically for this problem. Just to provide some background, I'm using the following functions in my GUI.

The 'bact' variable refers to Bacteria objects, which stores each bacteria's ID, position, etc.

 def identify_fluor(img, frame: int):

    darkBlue = (139, 0, 0)

    for bact in fluor_at_frame(frame):
    
        pos = tuple([int(coord) for coord in bact.position[frame]])
        img = cv2.circle(img, pos, 5, darkBlue, -1)

    return img

 def identify_bright(img, frame: int):

    darkRed = (0, 0, 139)

    for bact in bright_at_frame(frame):

        pos = tuple([int(coord) for coord in bact.position[frame]])
        img = cv2.circle(img, pos, 5, darkRed, -1)

    return img

These centroids are found using the best software available in current image-processing literature. As you can notice, the processing images on the right (bright-field) is significantly underdeveloped and is a significant hurdle and nuisance for bacteriology researchers.

We need these images on the right to be processed because they have a significantly greater sampling rate of images (1 second [Right] vs. 11 seconds [Left]). The fluorescence images (left) accumulate chemical damage when sampled too frequently, losing their florescence.

These are some instances when the images align perfectly:

Sample 1 of Bacteria Match:

Sample 2 of Bacteria Match:

Sample 3 of Bacteria Match:

In these cases, the images on the right are at an intermediate stage before reaching the next aligned image.

Sample 4 of Bacteria Match

Sample 5 of Bacteria Match

Sample 6 of Bacteria Match

Bright-Field Images

Sample 1 of Bright-Field

Sample 2 of Bright-Field

Sample 3 of Bright-Field

Additional Links

Sample 4 of Bright-Field

Sample 5 of Bright-Field

Sample 6 of Bright-Field

Sample 7 of Bright-Field

Sample 8 of Bright-Field

Sample 9 of Bright-Field

Note: This is not homework. I am doing a research project trying to gain information on the temporal dynamics of bacteria. I am trying to achieve a working solution on one of the samples of the images.

Edit #1: For clarification, I am trying to find the centroids of the bacteria on the right using the bacteria on the left.

Edit #2: I am not looking to match the images by applying a linear transformation. A computer vision algorithm is sought.

Edit #3: Additional bright-field images have been added separately for testing purposes.

Answer 1

My approach works directly on the right image.

The code is shared below and explained with comments:

I am creating a function at the beginning which erodes and dilates the image with a circular kernel, specified number of times.

kernel = np.array([[0, 0, 1, 0, 0], 
                   [0, 1, 1, 1, 0], 
                   [1, 1, 1, 1, 1], 
                   [0, 1, 1, 1, 0], 
                   [0, 0, 1, 0, 0]], dtype=np.uint8)
def e_d(image, it):
    image = cv2.erode(image, kernel, iterations=it)
    image = cv2.dilate(image, kernel, iterations=it)
    return image

Note: Image on the right is read in the grayscale format in the variable "img".

# Applying adaptive mean thresholding
th = cv2.adaptiveThreshold(img,255,cv2.ADAPTIVE_THRESH_MEAN_C, cv2.THRESH_BINARY_INV,11,2)
# Removing small noise
th = e_d(th.copy(), 1)

# Finding contours with RETR_EXTERNAL flag and removing undesired contours and 
# drawing them on a new image.
cnt, hie = cv2.findContours(th, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
cntImg = th.copy()
for contour in cnt:
    x,y,w,h = cv2.boundingRect(contour)
    # Eliminating the contour if its width is more than half of image width
    # (bacteria will not be that big).
    if w > img.shape[1]/2:      
        continue
    cntImg = cv2.drawContours(cntImg, [cv2.convexHull(contour)], -1, 255, -1)

# Removing almost all the remaining noise. 
# (Some big circular noise will remain along with bacteria contours)
cntImg = e_d(cntImg, 5)


# Finding new filtered contours again
cnt2, hie2 = cv2.findContours(cntImg, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)

# Now eliminating circular type noise contours by comparing each contour's 
# extent of overlap with its enclosing circle.
finalContours = []      # This will contain the final bacteria contours
for contour in cnt2:
    # Finding minimum enclosing circle
    (x,y),radius = cv2.minEnclosingCircle(contour)
    center = (int(x),int(y))
    radius = int(radius)

    # creating a image with only this circle drawn on it(filled with white colour)
    circleImg = np.zeros(img.shape, dtype=np.uint8)
    circleImg = cv2.circle(circleImg, center, radius, 255, -1)

    # creating a image with only the contour drawn on it(filled with white colour)    
    contourImg = np.zeros(img.shape, dtype=np.uint8)
    contourImg = cv2.drawContours(contourImg, [contour], -1, 255, -1)

    # White pixels not common in both contour and circle will remain white 
    # else will become black.
    union_inter = cv2.bitwise_xor(circleImg, contourImg)
    
    # Finding ratio of the extent of overlap of contour to its enclosing circle. 
    # Smaller the ratio, more circular the contour.
    ratio = np.sum(union_inter == 255) / np.sum(circleImg == 255)
    
    # Storing only non circular contours(bacteria)
    if ratio > 0.55:
        finalContours.append(contour)

finalContours = np.asarray(finalContours)


# Finding center of bacteria and showing it.
bacteriaImg = cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)

for bacteria in finalContours:
    M = cv2.moments(bacteria)
    cx = int(M['m10']/M['m00'])
    cy = int(M['m01']/M['m00'])

    bacteriaImg = cv2.circle(bacteriaImg, (cx, cy), 5, (0, 0, 255), -1)
    
cv2.imshow("bacteriaImg", bacteriaImg)
cv2.waitKey(0)

NOTE: I am only taking the image on the right and my image's size is (221, 828). If your input image is smaller or bigger than this, adjust the value of the number of iterations of erosion and dilation for removing noise accordingly to get good results.

Here are the output images:

Also, as you can see in the third image, the leftmost bacteria, its center is marked not exactly at the center. This is happening because, in the code, I have used convex hull of the contours in one place. You can solve this by keeping a track of all the contours and then at the end, take the center of the initial contour.

I am sure that this code can also be modified and made better but this is what I could think of right now. Any suggestions are most welcomed.

Answer 2

This seems to be an easy calibration problem.

Find two corresponding points left and right (i.e. the same points in the real world). If your setup is fixed, you can do that "manually" and once for all. You may have to add markers for this purpose (or use two distant bacteria centers that you match visually). If the setup is not fixed, add the markers anyway and design them so that they are easy to locate by image processing.

Now you have a simple linear relation between the left and right coordinates by solving

XR = a XL + b

for the two points. Then using one of the points to find c,

YR = a YL + c

holds.

Now knowing a, b, c, every point on the left can be mapped to the right. From your sample images, I have determined that

a ~ 1.128
b ~ 773
c ~ -16

very grossly.

---

Do not attempt any kind of matching of the shapes, rely on the geometric transformation of the coordinates.