Object Tracking on Images like Google Street Views

Question

Question:

Is it possible to use a tracking algorithm on street level images like Google panoramic street views? It is possible to track over video and each video frame is equivalent to an image, but these images have 5 meters in between.

What I have tried:

I have tried the Deep Sort tracking algorithm but it is not accurate and mostly looses the objects. I couldn't find much information on how to track over a set of images instead of video on Google.

Note:

I have a directory full of panoramic images which were each taken 5 meters apart. I see the same objects in multiple images but am not able to track them.

Any help or guidance wis appreciated.

Answer 1

Tracking a set of images instead of video shouldn't be a problem, it's the same as a video with a slower frame rate. I think the most likely reason the matching is failing is due to fish eye distortion (a result of the 360 imaging) which you will need to remove before you can then match the signs.

360 degree cameras generally use 2 or more wide angle cameras with fish eye lenses to capture the photos and then stitch them together in software. While this gives satisfactory 360 degree images, fish eye lenses add a lot of positive radial distortion. This means that as the object you want to track moves through the field of view of the camera, it gets distorted and then no longer "looks" like the original object.

Usually you would have access to the original cameras and could perform camera calibration to get a camera and distortion matrix which you could then use to undistort your images as detailed in the OpenCV docs. This is a good place for more background on where distortion comes from and how to deal with it.

Without calibration parameters, there are a few things you could try:

Estimate Camera and Distortion Matrices

This answer on the Signal Processing StackExchange mentions how to do this:

Compute the homography using findHomography then use warpPerspective to warp your images

The full post has more detail on how to do this, but it is fairly straightforward and I've used it before with decent success. findHomography will give you the parameters you need to pass to warpPerspective to remove the distortion without knowing the intrinsic camera parameters.

If that doesn't work for some reason, you could try the following less sophisticated approach:

Estimate a fisheye (radial) distortion via trial and error and pass it to undistort to correct the images

This answer and this answer detail how to do this. You won't know the distortion parameters so you can try some and see what values have better or worse results. I would only try this if the first method doesn't work.

Extra Info

I found this slightly dated research paper that integrates undistortion into a fast tracking algorithm.

Answer 2

You need a filter type spheric like a your camera objective with advance or a kalman filter, kalman filter is algorithm to noise or error prediction,

you need transform spheric coordenate to 2 dimension , after you need use

kf = cv.KalmanFilter(4, 2)
kf.measurementMatrix = np.array([[1, 0, 0, 0], [0, 1, 0, 0]], np.float32)
kf.transitionMatrix = np.array([[1, 0, 1, 0], [0, 1, 0, 1], [0, 0, 1, 0], [0, 0, 0, 1]], np.float32)

def Estimate(self, coordX, coordY):
    ''' This function estimates the position of the object'''
    measured = np.array([[np.float32(coordX)], [np.float32(coordY)]])
    self.kf.correct(measured)
    predicted = self.kf.predict()
    return predicted