4

I understand the above question is vast and vague. But to put into context: I'm trying to determine the accuracy of pose and position estimation of a camera. I have spent weeks calibrating and trying different methods and different sized boards, lighting, distances etc.

The methods I have tried:

Description:

  • Using about 20 images of various orientations and locations in front of the camera for each
  • For a 9x6 checkerboard of sizes 25mm, 32mm and 50mm blocks
  • At both resolutions of 1280x720 and 1920x1080
  • At distances ranging between 500mm for the smaller board to 2000mm for the larger one

In all cases I have followed strict guidelines as per this link How to verify the correctness of calibration of a webcam?

In all combinations of the above factors, I get results that are a few millimeters off (+- 15mm) My intrinsics for 25mm blocks at 1280x720 for the above methods span as follows:

  • OpenCV's built in calibration method > fx = 1269.4 fy = 1269.49 cx = 639.5 cy = 359.5
  • Matlab Calibration Toolbox > fx = 1259.53 fy = 1260.76 cx = 661.3 cy = 306.5
  • Matlab calibration App > fx = 1255.1 fy = 1254.8 cx = 652.6 cy = 340.7
  • Manually adjusting values. > Various results, none stable with exception of cx = 639.5 cy = 359.5

This should not be the case. The position of the camera relative to the origin of the board should be accurate to an average of a millimeter or two, if not submillimeter accuracy when dealing with distances of under 1 meter. Unless I am mistaken?

My question is, what is an ideal, simple calibration method, for an HD webcam, with little distortion?

Community
  • 1
  • 1
Grim
  • 81
  • 1
  • 9
  • openCVs and Matlabs calibration methods are the same, as far as i know... – Ander Biguri Dec 02 '14 at 19:06
  • 3
    I'm not sure how to interpret your statement ["In all cases I have followed strict guidelines as per ..."](http://stackoverflow.com/questions/12794876/how-to-verify-the-correctness-of-calibration-of-a-webcam/12821056#12821056) - because if I follow the link, under #4 I read: *"Take lots of measurements and pictures. You want hundreds of measurements (corners) per image, and tens of images. Where data is concerned, the more the merrier. A 10x10 checkerboard is the absolute minimum I would consider. I normally worked at 20x20."* - yet you are using a 9x6 board? – Schorsch Dec 02 '14 at 19:30
  • 1
    This is true. My project is to determine accuracies at distances of about 3 meters. Using smaller blocks became harder to detect, and using 10x10 rendered errors in detecting rotations of 180 degrees. I also read a few sources (one of which were in stackoverflow and cannot find) stating it was not necessary to have so many images. I found that my reprojection error increases with more Images. I am left with 20 images after rejecting high error ones. – Grim Dec 02 '14 at 20:07

2 Answers2

13

There are many possible sources of error.

First of all, while all three of the calibration implementations you have tried use essentially the same algorithm, there are enough differences that explain the discrepancies in the results.

The main difference is in the checkerboard corner detection. The Caltech Calibration Toolbox does not have automatic checkerboard detection, and uses a second optimization pass to refine the corners. Both OpenCV and the Camera Calibrator App do detect the checkerboard automatically, but the algorithm used in the Camera Calibrator App is far better. It is more robust, meaning that it is likely to detect a board when OpenCV does not, and its sub-pixel localization is more precise. My point is that in these three approaches you are calibrating using different data points. So it is not surprising that your results are different.

Once you have calibrated, what kind of reprojection errors are you getting? The Camera Calibrator App shows you a bar graph of the reprojection errors. You should look at it, and exclude the images that give you high errors. Ideally, you want your mean reprojection error to be less than half a pixel. The lower the better.

Now I have to ask you how you are measuring the distance from the camera to the checkerboard? The extrinsics you are getting from the calibration represent the transformation from the checkerboard's coordinate system into the camera's coordinate system, whose origin is inside the camera case, at its optical center. This is hard to measure accurately. A better way would be to place a flat object of a known size on the checkerboard and measure it using the camera. In fact, you can measure the distances between the detected checkerboard corners. Note that the detection accuracy is another source of error.

Another thing, please be sure not to save your calibration images as jpeg. The compression artifacts will affect the accuracy of the checkerboard corner detection. Use a lossless format like tiff or png.

Dima
  • 38,860
  • 14
  • 75
  • 115
  • Thank you for the answer. I understand that the results would be different, just expected them to be more similar. I get reprojection errors of under 0.199 for both matlab App and Caltec's toolbox. Both of which is after rejecting high error images. – Grim Dec 02 '14 at 20:12
  • Your jpeg information is interesting. I used jpeg for caltec's toolbox, but used png for both OpenCV and Matlab's App. I will try tiff and get back with results. – Grim Dec 02 '14 at 20:14
  • 0.199 sounds good... What about your +/- 15mm error? How are you measuring that? – Dima Dec 02 '14 at 20:15
  • Definitely do not use jpeg at any point during acquisition. Obviously, if you convert a jpeg image to png, you will still have the artifacts. – Dima Dec 02 '14 at 20:17
  • I'm measuring the distance from the origin of the chessboard, to the lens, and also from the center line (principal point line) to the lens, both with a standard measuring tape. (I understand this measuring tool is not very scientific, but it's all I have). – Grim Dec 02 '14 at 20:21
  • Doesn't sound very precise... Try measuring the size of the checkerboard squares, and comparing to the actual size. Undistort the image, call detect checkerboardPoints(), then use the pointsToWorld() method of cameraParameters. – Dima Dec 02 '14 at 20:24
  • 2
    I can confirm the following: Using PNG or TIFF in the Matlab Toolbox instead of JPG, shaved off about 3mm of the accuracy. Also as you said, "...camera's coordinate system, whose origin is inside the camera case..." this gave me the correct results. Various tests between 300mm and 1500mm gave me accuracies not exceeding about 3mm (with a measuring tape). Also, and most importantly, the checkerboard block sizes were measured with a caliper which also took off about 5mm, even thought it only went from 25mm to 25.3mm, which makes sense. Thank you very much for your help Dima. – Grim Dec 02 '14 at 21:30
3

Coming to this party a tad late, but if it can help...

The differences are not all that surprising, given that the three applications, though relying mostly on the same algorithms (and, for OpenCV and the Matlab kit, even sharing an author), have different implementations with different performance. If you are interested in making a fair comparison, you should at least drive them with the same sets of measurements (i.e. the locations of the detected corners, with subpixel refinement), so that any difference in output be purely due to the implementation. It has already correctly been pointed out that, by using different image formats, you were effectively using different measurement sets.

The variation you observe in the principal point is also unsurprising: it is quite hard to estimate it accurately because

  • The reprojection error is not very sensitive to its location, i.e. the cost function you are trying to optimize is mostly "flat" in the (cx, cy) subspace around the optimum (see in-depth discussion in this paper), and
  • It is intrinsically confused with the center of the nonlinear distortion and by the component parallel to to image plane of the world-to-camera translation.

The latter point implies that small, hard-to-detect variations in the principal point will result in significant 3D positional errors unless additional constraints are added, that make the constrained error function not flat about quantitities of interest, i.e. the absolute position and orientation of the camera with respect to the calibration rig.

Convenient constraints to add are:

  • A prior model for the relative motion of calibration target and camera. For example, if your checkerboard target is attached to a turntable, you can optimize for the center and axis of its motion (3 + 2 parameters), rather than the 6 x num_images parameters for the poses of the individual target positions.
  • Multiple camera positions with respect to a non-planar fixed rig (or rig-related point-and-direction, as above).
Francesco Callari
  • 11,300
  • 2
  • 25
  • 40