OpenCV findFundamentalMat very unstable and sensitive

Question

I'm working on a Project for my University where we want a Quadrcopter to stabilize himself with his camera. Unfortunately the Fundamental matrix reacts very sensible to little changes within the featurpoints, i'll give you examples later on.

I think my matching already works pretty good thanks to ocv. I'm using SURF Features and match them with the knn-Method:

    SurfFeatureDetector surf_detect;
    surf_detect = SurfFeatureDetector(400);

    //detect keypoints
    surf_detect.detect(fr_one.img, fr_one.kp);
    surf_detect.detect(fr_two.img, fr_two.kp);

    //extract keypoints
    SurfDescriptorExtractor surf_extract;
    surf_extract.compute(fr_one.img, fr_one.kp, fr_one.descriptors);
    surf_extract.compute(fr_two.img, fr_two.kp, fr_two.descriptors);

    //match keypoints
    vector<vector<DMatch> > matches1,matches2;
    vector<DMatch> symMatches,goodMatches;
    FlannBasedMatcher flann_match;

    flann_match.knnMatch(fr_one.descriptors, fr_two.descriptors, matches1,2);
    flann_match.knnMatch(fr_two.descriptors, fr_one.descriptors, matches2,2);

    //test matches in both ways
    symmetryTest(matches1,matches2,symMatches);

    std::vector<cv::Point2f> points1, points2;
    for (std::vector<cv::DMatch>::const_iterator it= symMatches.begin();
       it!= symMatches.end(); ++it)
    {
        //left keypoints
        float x= fr_one.kp[it->queryIdx].pt.x;
        float y= fr_one.kp[it->queryIdx].pt.y;
        points1.push_back(cv::Point2f(x,y));
        //right keypoints
        x = fr_two.kp[it->trainIdx].pt.x;
        y = fr_two.kp[it->trainIdx].pt.y;
        points2.push_back(cv::Point2f(x,y));
    }

    //kill outliers with ransac
    vector<uchar> inliers(points1.size(),0);
    findFundamentalMat(Mat(points1),Mat(points2),
                inliers,CV_FM_RANSAC,3.f,0.99f);

    std::vector<uchar>::const_iterator
    itIn= inliers.begin();
    std::vector<cv::DMatch>::const_iterator
    itM= symMatches.begin();
    for ( ;itIn!= inliers.end(); ++itIn, ++itM)
    {
        if (*itIn)
        {
            goodMatches.push_back(*itM);
        }
    }

Now i want to compute the Fundamental Matrix with these matches. I'm using the 8POINT method for this example - i already tried it with LMEDS and RANSAC - there it only get's worse because there are more matches which change.

    vector<int> pointIndexes1;
    vector<int> pointIndexes2;
    for (vector<DMatch>::const_iterator it= goodMatches.begin();
         it!= goodMatches.end(); ++it) {
             pointIndexes1.push_back(it->queryIdx);
             pointIndexes2.push_back(it->trainIdx);
    }
    vector<Point2f> selPoints1, selPoints2;
    KeyPoint::convert(fr_one.kp,selPoints1,pointIndexes1);
    KeyPoint::convert(fr_two.kp,selPoints2,pointIndexes2);

    Mat F = findFundamentalMat(Mat(selPoints1),Mat(selPoints2),CV_FM_8POINT);

When i call these calculations within a loop on the same pair of images the result of F varies very much - theres no way to extract movement from such calculations.

I generated an example where i filtered out some matches so that you can see the effect i mentioned for yourselves.

http://abload.de/img/div_c_01ascel.png

http://abload.de/img/div_c_02zpflj.png

Is there something wrong with my code or do i have to think about other reasons like image-quality and so on ?

Thanks in advance for the Help ! derfreak

Answer 1

To summarize what others have already stated and elaborate in more detail,

1. As currently implemented in OpenCV, the 8-point algorithm has no outlier rejection. It is a least-squares algorithm and cannot be used with RANSAC or LMEDS because these flags override the 8-point flag. It is recommended that the input points are normalized to improve the condition number of the matrix in the linear equation, as stated in "In Defence of the 8-point Algorithm". However, the OpenCV implementation automatically normalizes the input points, so there is no need to normalize them manually.

2. The 5-point and 7-point algorithms both have outlier rejection, using RANSAC or LMEDS. If you are using RANSAC, you may need to tune the threshold to get good results. The OpenCV documentation shows that the default threshold for RANSAC is 1.0, which in my opinion is a bit large. I might recommend using something around 0.1 pixels. On the other hand, if you are using LMEDS you won't need to worry about the threshold, because LMEDS minimizes the median error instead of counting inliers. LMEDS and RANSAC both have similar accuracy if the correct threshold is used and both have comparable computation time.

3. The 5-point algorithm is more robust than the 7-point algorithm because it only has 5 degrees of freedom (3 rotation and 2 for the unit-vector translation) instead of 7 (the additional 2 parameters are for the camera principle points). This minimal parameterization allows the rotation and translation to be easily extracted from the matrix using SVD and avoids the planar structure degeneracy problem.

4. However, in order to get accurate results with the 5-point algorithm, the focal length must be known. The paper suggests that the focal length should be known within 10%, otherwise the 5-point algorithm is no better than the other uncalibrated algorithms. If you haven't performed camera calibration before, check out the OpenCV camera calibration tutorial. Also, if you are using ROS, there is a nice camera calibration package.

5. When using the OpenCV findEssentialMat function I recommend first passing the pixel points to undistortPoints. This not only reverses the effect of lens distortion, but also transforms the coordinates to normalized image coordinates. Normalized image coordinates (not to be confused with the normalization done in the 8-point algorithm) are camera agnostic coordinates that do not depend on any of the camera intrinsic parameters. They represent the angle of the bearing vector to the point in the real world. For example, a normalized image coordinate of (1, 0) would correspond to a bearing angle of 45 degrees from the optical axis of the camera in the x direction and 0 degrees in the y direction.

6. After using RANSAC to obtain a good hypothesis, the best estimate can be improved by using iterative robust non-linear least-squares. This is mentioned in the paper and described in more detail in "Bundle Adjustment - A Modern Synthesis". Unfortunately, it appears that the OpenCV implementation of the 5-point algorithm does not use any iterative refinement methods.

Answer 2

Even if your algorithm is correct, 8 point F matrix computation is very error prone due to image noise. The lesser correspondences you use the better. The best you can do is doing 5 point Essential (E) matrix computation, but that would require you to pre-calibrate the camera and convert the detected pixel image points after SIFT/SURF to normalized pixels (metric pixel locations). Then apply Nister's 5-point algorithm either from the freely available Matlab implementation or from Bundler (c++ implementation by Noah Snavely). In my experience with SfM, 5-point E matrix is much much better/stable than 7 or 8 point F matrix computation. And ofcourse do RANSAC after 5 point to get more robust estimates. Hope this helps.

Answer 3

The 8-point algorithm is the simplest method of computing fundamental matrix, but if care is taken you can perform it well. The key to obtain the good results is proper careful normalization of the input data before constructing the equations to solve. Many of algorithms can do it.

Pixels point coordinate must be changed to camera coordinates, I don't see that you are doing these. As I understand, your

vector<int> pointIndexes1; is expressed in the pixel coordinates. You must known the intrinsic camera parameters, if you want get more stable results. You may find them by many methods: tutorial openCV. Then you have two options of normalize it. You may apply for your fundamental matrix,

Mat E = K.t() * F * K; where K is Intrinsic Camera Parameters.[see on Wiki]

However this assumption is not accurate. If camera calibration matrix K is known, then you may apply inverse to the point x to obtain the point expressed in camera normalized coordinates.

pointNormalize1= K.inv()*pointIndexes1 where pointIndexes1(2), z is equal 1.

In the case of the 8PA, a simple transformation of points improve and hence in the stability of the results. The suggested normalization is a translation and scaling of each image so that the centroid of the reference points is at origin of the coordinates and the RMS distance of the points from the origin is equal to ![sqrt{2}]. Note that it is recommended that the singularity condition should be enforced before denormalization.

Reference: check it if : you are still interested