I am attempting to develop a pipeline for stabilizing images of a fluid experiment using opencv in Python. Here is an example raw image (actual size: 1920x1460)
This pipeline should be able to stabilize both for low-frequency drift and high frequency "jitter" that occasionally happens when valves are opened/closed during the experiment. My current approach, following the example here, is to apply a bilateral filter followed by adaptive thresholding to bring out the channels in the image. Then, I use goodFeaturesToTrack to find corners in the thresholded image. However, there is a substantial amount of noise in the image because of low contrast some optical effects in the corners of the image. Although I can find corners of the channel, as shown here, they move around quite a bit frame to frame, seen here. I tracked the amount of x and y pixel shift in each frame relative to the first frame calculated from calcOpticalFlowPyrLK, and computing a rigid transform using estimateRigidTransform, shown here. In this plot I can see low-frequency drift from frames 0:200, and a sharp jump around frame ~225. These jumps match what is observed in the video. However, the substantial noise (with amplitude of ~5-10 pixels) does not match what is observed in the video. If I apply these transforms to my image stack, I get instead increased jitter that does not stabilize the image. Moreover, if I try and compute the transform for one frame to the next (rather than all frames to the first), after processing a handful of frames I will get None returns for the rigid transform matrix, probably because the noisiness prevents a rigid transform from being calculated.
Here's a sample of how I am calculating the transform:
# Load required libraries
import numpy as np
from skimage.external import tifffile as tif
import os
import cv2
import matplotlib.pyplot as plt
from sklearn.externals._pilutil import bytescale
#Read in file and convert to 8-bit so it can be processed
os.chdir(r"C:\Path\to\my\processingfolder\inputstack")
inputfilename = "mytestfile.tif"
input_image = tif.imread(inputfilename)
input_image_8 = bytescale(input_image)
n_frames, vid_height, vid_width = np.shape(input_image_8)
transforms = np.zeros((n_frames-1,3),np.float32)
prev_image = starting_image
prev_f = cv2.bilateralFilter(prev_image,9,75,75)
prev_t = cv2.adaptiveThreshold(prev_f,255,cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY,49,2)
prev_pts = cv2.goodFeaturesToTrack(prev_t,maxCorners=100,qualityLevel=0.5,minDistance=10,blockSize=25,mask=None)
for i in range(1,n_frames-2):
curr_image = input_image_8[i]
curr_f = cv2.bilateralFilter(curr_image,9,75,75)
curr_t = cv2.adaptiveThreshold(curr_f,255,cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY,49,2)
#Detect features through optical flow:
curr_pts, status, err = cv2.calcOpticalFlowPyrLK(prev_t,curr_t,prev_pts,None)
#Sanity check
assert len(prev_pts) == len(curr_pts)
#Filter to only the valid points
idx = np.where(status==1)[0]
prev_pts = prev_pts[idx]
curr_pts = curr_pts[idx]
#Find transformation matrix
m = cv2.estimateRigidTransform(prev_pts,curr_pts, fullAffine=False) #will only work with OpenCV-3 or less
# Extract translation
dx = m[0,2]
dy = m[1,2]
# Extract rotation angle
da = np.arctan2(m[1,0], m[0,0])
# Store transformation
transforms[i] = [dx,dy,da]
print("Frame: " + str(i) + "/" + str(n_frames) + " - Tracked points : " + str(len(prev_pts)))
How can I process my images differently so that I am picking out the lines of these channels without the noisiness in detecting corners? This stabilization/alignment does not need to happen on the fly, it can be applied to the whole stack after the fact.
