3

I saw that OpenCV's solvePnP() function assume that your camera parameters are from a pinhole model. But I calibrated my camera using cv.fisheye module, so I wanted to know how to use solvePnP with parameters obtained from that fisheye module.

How can I use my fisheye camera parameters with solvePnP()?

off99555
  • 3,797
  • 3
  • 37
  • 49

1 Answers1

6

According to docs.opencv.org, you have {K, D, rvecs, tvecs} from cv::fisheye::calibrate().

You can remove the effect of K and D from the input coordinates distorted using cv.fisheye.undistortPoints() See here.

So the routine must be:

  1. undistorted = cv.fisheye.undistortPoints(distorted, K, D)
  2. cv.solvePnP(objPoints, undistorted, I, D) where I=np.eye(3), D=np.zeros((1,5))

good luck

Yongduek Seo
  • 76
  • 3
  • How would I be sure that this will work? I saw that internally `solvePnP` use `cv.undistortPoints` instead of `cv.fisheye.undistortPoints`. I tried evaluating both these functions with same arguments and they return different result (even when np.eye(3) and np.zeros((1,5)) used as camera parameters). Do these functions need to return the same output with identify camera matrix? – off99555 Nov 27 '19 at 05:33
  • 1
    It seems the solution is working. Though I don't understand why it's working. The way I prove is by experimentation. The idea is to go from object pose to image points then back again at object pose. And see if the object pose is preserved. I generated a random `rvec` and `tvec`, do `imgpoints = cv.fisheye.projectPoints(objectPoints,rvec,tvec,K,D)`, then use `imgpoints` to do `solvePnP` as described in the answer. The output `rvec`, and `tvec` obtained by solvePnP matches very closely (has 1e-7 error) to the orginal rotation and translation. So this is proof by experiment that it's working! – off99555 Nov 27 '19 at 08:16
  • Good to hear it works. The camera model in `cv.fishey` uses a different lens distortion model as is found in the `docs.opencv.org`, which is why `cv.fisheye.undistortPoints()` should be used to remove the distortion effect. – Yongduek Seo Nov 27 '19 at 12:55
  • I understand that you should use `cv.fisheye.undistortPoints()` to remove the distortion effect. But I don't undersetand why `cv.undistortPoints(imgpoints, np.eye(3), np.zeros(4))` is not equal to `cv.fisheye.undistortPoints(imgpoints, np.eye(3), np.zeros(4))`. The first function makes sense to me, it returns the same value as `imgpoints`. But the 2nd function returns something different that I do not understand. Why they both don't return `imgpoints`? – off99555 Nov 27 '19 at 18:24
  • It is interesting to examine why. The reason is simply due to the [computation procedure of the distortion model of fisheye lens in opencv](https://docs.opencv.org/4.1.2/db/d58/group__calib3d__fisheye.html) which I also found myself very curious; this is quite contrary to the normal pin hole projection case. A related [stackoverflow q&a was found to be here](https://stackoverflow.com/questions/31089265/what-are-the-main-references-to-the-fish-eye-camera-model-in-opencv3-0-0dev). – Yongduek Seo Nov 28 '19 at 15:46
  • Great! This helped me to understand the fisheye camera model and why we should use empty 'I' and 'D' params to solvePnP by undistorted image. Thank you again! – Yuanhui Nov 29 '21 at 03:59