OpenCV + Python: Calculate stereo reprojection error

Question

I want to do something similar to this question, but for stereoCalibrate() instead of calibrateCamera(). That is, compute the reprojection error for a stereo camera calibration.

My reduced example looks like this:

import cv2
import numpy as np

def calibrate_stereo(w, h, objpoints, imgpoints_l, imgpoints_r):
    stereocalib_criteria = (cv2.TERM_CRITERIA_COUNT + cv2.TERM_CRITERIA_EPS , 1000, 1e-6)
    retval, A1, D1, A2, D2, R, T, E, F = cv2.stereoCalibrate(objpoints,imgpoints_l, imgpoints_r,None,None,None,None, (w,h), flags=0, criteria=stereocalib_criteria)

    return (retval, (A1,D1,A2,D2, R, T, E, F))

def calc_rms_stereo(objectpoints, imgpoints_l, imgpoints_r, A1, D1, A2, D2, R, T):
    tot_error = 0
    total_points = 0

    for i, objpoints in enumerate(objectpoints):
        # calculate world <-> cam1 transformation
        _, rvec_l, tvec_l,_ = cv2.solvePnPRansac(objpoints, imgpoints_l[i], A1, D1)

        # compute reprojection error for cam1
        rp_l, _ = cv2.projectPoints(objpoints, rvec_l, tvec_l, A1, D1)
        tot_error += np.sum(np.square(np.float64(imgpoints_l[i] - rp_l)))
        total_points += len(objpoints)

        # calculate world <-> cam2 transformation
        rvec_r, tvec_r  = cv2.composeRT(rvec_l,tvec_l,cv2.Rodrigues(R)[0],T)[:2]

        # compute reprojection error for cam2
        rp_r,_ = cv2.projectPoints(objpoints, rvec_r, tvec_r, A2, D2)
        tot_error += np.square(imgpoints_r[i] - rp_r).sum()
        total_points += len(objpoints)

    mean_error = np.sqrt(tot_error/total_points)

    return mean_error


if __name__ == "__main__":    
    # omitted: reading values for w,h, objectPoints, imgpoints_l, imgpoints_r from file (format as expected by the OpenCV functions)
    # [...]

    rms, (A1,D1,A2,D2,R,T,_,_) = calibrate_stereo(w, h, objectpoints, imgpoints_l, imgpoints_r)

    print("RMS (stereo calib): {}".format(rms))

    rms_2 = calc_rms_stereo(objectpoints, imgpoints_l, imgpoints_r, A1, D1, A2, D2, R, T)    
    print("RMS (custom calculation):", rms_2)

Sample output:

RMS (stereo calib): 0.14342257926694932
RMS (custom calculation): 0.356273345751

As far as I can tell, the computation in the source code of stereoCalibrate() is very similar to mine. What am I missing?

OpenCV 3.3.0 on Ubuntu

What I understood of the source code of stereo calibrate they do the distance between the point ( norm(a-b) ) added to the error and then they divide and get the root square... I think you add the squares of all the points components difference and do square root... Maybe I am wrong — api55, Nov 16 '17 at 13:17
@api55 that's what i thought too. I assume you mean [this line](https://github.com/opencv/opencv/blame/master/modules/calib3d/src/calibration.cpp#L2152), but they are indeed using a [square norm](https://stackoverflow.com/questions/32887671/does-opencv-offer-a-squared-norm-function-for-cvpoint) — TheBender, Nov 16 '17 at 13:33
exactly that one. Also, it makes sense. RMSE is the root of the mean of the squared error, however it depends of what you define as error. As far as I know, the reprojection error is the distance of a point projected in 2D with a point in 2D used to generate the 3D point in the first place... — api55, Nov 16 '17 at 13:50
Yes, i agree. But norm(a-b)^2 (i.e. squared distance between points a & b) should be equal to the sum of the squared elementwise differences, i.e. (a[0]-b[0])^2 + (a[1]-b[1])^2 no? — TheBender, Nov 16 '17 at 14:54
Hmmm, actually it is true, I didn't see that they were using `NORM_L2SQR` which is the squared version..... then everything looks the same, strange... I do not spot the difference :( — api55, Nov 16 '17 at 15:03

score 1 · Accepted Answer · answered Jan 23 '18 at 12:00

I solved it after implementing a custom stereo calibration algorithm based on the OpenCV implementation.

The difference between the reprojection error calculated inside cv2.stereoCalibrate() and my custom calculation stems from different values for the extrinsic parameters rvec_l and tvec_l. These vectors describe the rotation and translation between the left camera and the calibration pattern for each image. cv2.solvePnpRansac() yields optimized values based only on the reprojection error of the left image, while in cv2.stereoCalibrate() those values are optimized together with R and T based on the reprojection error in both images of each stereo pair.

If one wants to exactly replicate the RMS value that is returned by cv2.stereoCalibrate(), one has to modify the C/C++ source code of cv::stereoCalibrate() to return the optimized extrinsic parameters as well (cv::calibrateCamera() does that already for monocular calibration).

I didn't get the point. From the error's perspective cv2.stereocalibrate is a better option it seems, however you are suggesting changing the code of the same. Can you please explain this. — user169703, Aug 01 '22 at 18:22

OpenCV + Python: Calculate stereo reprojection error

1 Answers1