OpenCV - Calibrate fisheye lens error (Ill-conditioned matrix)

Question

I am trying to calibrate a fisheye lens following these instructions https://medium.com/@kennethjiang/calibrate-fisheye-lens-using-opencv-333b05afa0b0 where you can find the full code I'm using for the calibration part.

I arrive at this point where:

N_OK = len(objpoints)
K = np.zeros((3, 3))
D = np.zeros((4, 1))
rvecs = [np.zeros((1, 1, 3), dtype=np.float64) for i in range(N_OK)]
tvecs = [np.zeros((1, 1, 3), dtype=np.float64) for i in range(N_OK)]  
rms, _, _, _, _ = \
    cv2.fisheye.calibrate(
        objpoints,
        imgpoints,
        gray.shape[::-1],
        K,
        D,
        rvecs,
        tvecs,
        calibration_flags,
        (cv2.TERM_CRITERIA_EPS+cv2.TERM_CRITERIA_MAX_ITER, 30, 1e-3)
    )
print("Found " + str(N_OK) + " valid images for calibration")
print("DIM=" + str(_img_shape[::-1]))
print("K=np.array(" + str(K.tolist()) + ")")
print("D=np.array(" + str(D.tolist()) + ")")

I get this error:

Traceback (most recent call last)
<ipython-input-10-deaca9981fe4> in <module>()
     13         tvecs,
     14         calibration_flags,
---> 15         (cv2.TERM_CRITERIA_EPS+cv2.TERM_CRITERIA_MAX_ITER, 30, 1e-3)
     16     )
     17 print("Found " + str(N_OK) + " valid images for calibration")

error: C:\ci\opencv_1512688052760\work\modules\calib3d\src\fisheye.cpp:1414: 
error: (-3) CALIB_CHECK_COND - Ill-conditioned matrix for input array 0 in 
function cv::internal::CalibrateExtrinsics

I don't understand what's going on and I could only find so little information around the internet, does anyone have experienced something similar and know how to solve this?

Thanks

These are the images of the checkerboard I'm using:

Gabriel Devillers · Accepted Answer · 2020-04-05T15:47:01.153

15

I think it is because your variable calibration_flags has CALIB_CHECK_COND set. Try disabling this flag. Without it I was able to undistort your images (see links below).

I am not sure what this check is for (the documentation is not very explicit). This flag reject some images¹ of my gopro hero 3 even when the chessboard is visible and detected. In my case one image among 20 is not passing this test. This image has the chessboard close to the left border.

¹ in OpenCV versions >= 3.4.1 the error message tells you which image is not passing the test

edited Apr 05 '20 at 15:47

answered Mar 23 '18 at 11:53

Gabriel Devillers

3,155
2
30
53

That's great, thank you! Were you able to preserve all the pixels outside the crop area by any chance? – Francesco Apr 04 '18 at 17:59
Yes you can do that by chosing the appropriate parameters `P` and `size` in function [fisheye::initUndistortRectify](https://docs.opencv.org/3.4.1/db/d58/group__calib3d__fisheye.html#ga0d37b45f780b32f63ed19c21aa9fd333). You should be able to compute them with function [getOptimalNewCameraMatrix](https://docs.opencv.org/3.4.1/d9/d0c/group__calib3d.html#ga7a6c4e032c97f03ba747966e6ad862b1). See [this QA](https://stackoverflow.com/questions/34316306/opencv-fisheye-calibration-cuts-too-much-of-the-resulting-image). But as user alexisrozhkov said it might not be possible to keep all pixels. – Gabriel Devillers Apr 06 '18 at 07:46
I followed the same instructions and removed the CALIB_CHECK_COND flag as Gabriel suggested. I used the same images as Francesco provided to get K and D. But I still see some distortion using K and D. I don't see such distortion on the undistorted images provided by Gabriel. So I am wondering whether you are using the same code, or you made some modification. Here is an example: Distorted image: https://i.stack.imgur.com/DNBuD.jpg My undistorted image: https://i.stack.imgur.com/GTAE0.jpg Gabriel's undistorted image: https://i.stack.imgur.com/rmJ5Q.jpg – huisinro Aug 12 '19 at 22:44
@huisinro I am afraid I cannot find the code I used to produce these images, sorry. But I suspect it was close to the example found online. – Gabriel Devillers Aug 13 '19 at 08:27
Gabriel, thanks for the update. We actually figured it out now. – huisinro Aug 13 '19 at 17:51
Has anyone figured out what `CALIB_CHECK_COND` flag does exactly? – nim.py Apr 16 '22 at 13:52

score 1 · Answer 2 · answered Nov 11 '19 at 19:26

1

I did not find the code in python so I manually check the images with chessboard on the edge and delete them one by one until the error is gone.

answered Nov 11 '19 at 19:26

Ahmadiah

476
5
12

score 1 · Answer 3 · answered Dec 18 '19 at 22:38

As @Ahmadiah mentioned, the "ill conditioned" thing can happen when the checkerboard falls near the edge of the image. One way to handle this is to remove images one by one and try again when they cause calibration to fail. Here is an example where we do that:

def calibrate_fisheye(all_image_points, all_true_points, image_size):
    """ Calibrate a fisheye camera from matching points.
    :param all_image_points: Sequence[Array(N, 2)[float32]] of (x, y) image coordinates of the points.  (see  cv2.findChessboardCorners)
    :param all_true_points: Sequence[Array(N, 3)[float32]] of (x,y,z) points.  (If from a grid, just put (x,y) on a regular grid and z=0)
        Note that each of these sets of points can be in its own reference frame,
    :param image_size: The (size_y, size_x) of the image.
    :return: (rms, mtx, dist, rvecs, tvecs) where
        rms: float - The root-mean-squared error
        mtx: array[3x3] A 3x3 camera intrinsics matrix
        dst: array[4x1] A (4x1) array of distortion coefficients
        rvecs: Sequence[array[N,3,1]] of estimated rotation vectors for each set of true points
        tvecs: Sequence[array[N,3,1]] of estimated translation vectors for each set of true points
    """
    assert len(all_true_points) == len(all_image_points)
    all_true_points = list(all_true_points)  # Because we'll modify it in place
    all_image_points = list(all_image_points)
    while True:
        assert len(all_true_points) > 0, "There are no valid images from which to calibrate."
        try:
            rms, mtx, dist, rvecs, tvecs = cv2.fisheye.calibrate(
                objectPoints=[p[None, :, :] for p in all_true_points],
                imagePoints=[p[:, None, :] for p in all_image_points],
                image_size=image_size,
                K=np.zeros((3, 3)),
                D=np.zeros((4, 1)),
                flags=cv2.fisheye.CALIB_RECOMPUTE_EXTRINSIC + cv2.fisheye.CALIB_CHECK_COND + cv2.fisheye.CALIB_FIX_SKEW,
                criteria=(cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 1e-6),
            )
            print('Found a calibration based on {} well-conditioned images.'.format(len(all_true_points)))
            return rms, mtx, dist, rvecs, tvecs
        except cv2.error as err:
            try:
                idx = int(err.message.split('array ')[1][0])  # Parse index of invalid image from error message
                all_true_points.pop(idx)
                all_image_points.pop(idx)
                print("Removed ill-conditioned image {} from the data.  Trying again...".format(idx))
            except IndexError:
                raise err

Wouldn't `err.message.split('array ')[1][0]` only work for 1-digit indexes? `int(err.message.split()[-4])` (for python 2) or `int(str(err).split()[-4])` (for python 3) should work for any indexes — roygbiv, Jul 09 '21 at 16:35

score -1 · Answer 4 · answered Jan 15 '21 at 03:39

my opencv version is 4.0+, I sloved it by this method

rms, _, _, _, _ = \
cv2.fisheye.calibrate(
    objpoints,
    imgpoints,
    #gray.shape[::-1],
    gray.shape,
    K,
    D,
    rvecs,
    tvecs,
    calibration_flags,
    (cv2.TERM_CRITERIA_EPS+cv2.TERM_CRITERIA_MAX_ITER, 30, 1e-3)
)

OpenCV - Calibrate fisheye lens error (Ill-conditioned matrix)

4 Answers4

Linked