2

i am trying to evaluate distortion correction by line fitting algorithm. Now i want to make a decision whether the given image is corrected or not. So should i consider RMSE for overall image, because i get RMSE for each line in image. Please suggest me how to make decision.

user4973313
  • 39
  • 8

1 Answers1

0

Afraid you are doing it all wrong, sorry. The mantra that "a well calibrated camera maps straight lines in the world to straight lines in the image", while true, does not lend itself to a well-posed definition of a metric for the quality of your calibration. You can compute an RMSE on straight lines in various ways, but they are all unprincipled hacks.

You can only define the RMSE error for the entire model of the projection from 3D points to their image. In other words, it only makes sense to speak of RMSE when you are doing bundle adjustment, solving jointly for the pose of the camera, and the linear and nonlinear intrinsic parameters of the lens. This is what you do when you calibrate a camera, or solve a structure-from-motion problem by bundle adjustment.

While it is theoretically true that a perfect estimation of the nonlinear lens distortion parameters "straightens" lines perfectly, it is quite tricky to use just this fact in order to define a metric for the quality of practically estimated distortion parameters. There are several reasons for this, among them:

  1. When you apply a least-squares straight-line fitting algorithm on points obtained by un-distorting with erroneous parameters you are are using a wrong model. Applying the un-distortion function itself to the image of a physical straight 3D line produces a curve in the image which is only straight if the parameters are well estimated. When they aren't, your line fit will be biased, which means that distance between the curve and the straight line it "fits" is not a purely random variable: it depends on where you measure it, and where the curve itself is located in the image.
  2. It is tricky to define a distance between a curve and a straight line that supposedly fits it. How do you choose which point on the line corresponds to a given point on the curve, or viceversa?
  3. A more principled approach would be to define an error measure based on the geometrical curvature of the curve resulting by un-distorting the image of a physical straight line. However, attempting to accurately measure curvature opens a can of worms by itself, since it amounts to estimating (explicitly or not) the first and second derivatives of the curve, which amplifies noise.

So, all in all, either one of your suggestion "works" in the sense that it gives you a number that, if small, is "suggestive" of likely good calibration. However, neither is a "correct" choice, because the basis of what you are trying to measure (and define an error for) is shaky.

Francesco Callari
  • 11,300
  • 2
  • 25
  • 40
  • Than you for suggestion, 1) Actually im applying distortion correction for fisheye image. In this straight line looks curved, so to correct it i am using distortion correction algorithm. i am confused with your statement "Applying the un-distortion function itself to the image of a physical straight 3D line produces a curve in the image which is only straight if the parameters are well estimated." – user4973313 Nov 16 '17 at 07:13
  • 2) Image which i have taken for evaluation is chessboard image, and i have selected corners on each line to fit a line. So, please recheck and clarify – user4973313 Nov 16 '17 at 07:16
  • 3) Also is there standard method to evaluate distortion correction algorithm. or should i take different test pattern – user4973313 Nov 16 '17 at 07:18
  • I don't have much to add to the answer without more details on what exactly your are doing. If you are using a chessboard you are working with points, I presume, not lines, so your mentioning of "line fitting" in the question is unclear. – Francesco Callari Nov 16 '17 at 16:51
  • Ok, 1) Actually, i have taken chessboard image using fisheye camera. Because of barrel distortion straight lines looks like curved. So, im doing distortion correction. As we know chessboard have many horizontal and vertical lines with corner points, to evaluate the algorithm i am marking corner points on straight line(undistorted image), and then fitting line. My assumption is if the line is straight then all points should lie on it and must fit proper line fit with less RMSE error. – user4973313 Nov 17 '17 at 04:50
  • 2) Since there are many lines on chessboard, i have to do line fit for each line separately, thus i get many RMSE values. 3) Now my doubt is how to make decision with single value for whole image. – user4973313 Nov 17 '17 at 04:52
  • At present, for example: if (x,y) is the marked corner point and (x,y1) is the estimated point of line fit. i have taken all y and corresponding y1 from all the lines (different line fit of each line), and then computing RMSE for whole image. Is the approach and metric is appropriate. – user4973313 Nov 17 '17 at 05:00
  • I get it, and what I am saying is that the RMSE number you get this way is rather meaningless, because it depends on the choice of the points. Try this exercise: draw a curve on a piece of paper (say, a segment of a parabola), then mark some points on it and fit a line to them. Now think about what the "error" at the points you selected tells you about the "error" you can expect at some other points, and what happens when you change your initial selection of points. Now, see what I mean when I say in my answer "your line fit is biased" ? – Francesco Callari Nov 17 '17 at 18:31
  • Ok, Francesco..got it..initially I tried by marking points manually, I observed bias but later thought of using OPENCV findChessboardCorner..Can I make use of this function to avoid bias or is there any other best method to evaluate the algorithm.. – user4973313 Nov 19 '17 at 13:29
  • Francesco. I was going through some of your answers for similar question, in that i have found a link "https://stackoverflow.com/questions/18052337/how-to-verify-that-t". There it is mentioned "Pick points on said line images, undistort their coordinates, fit mathematical lines, compute error." can you please tell me is it not similar to my approach, except that points are taken from distorted image(in your answer).Please clarify. – user4973313 Nov 20 '17 at 06:54
  • In that answer I do not make any claims as to statistical consistency. It is suggested as a "verification" method for a calibrated solution, not as a metric for its quality - it's just a backstop. As I said, any sensible error function you chose will be close to zero if the calibration is good. What I am trying to tell you is that its value when the calibration is NOT good may be meaningless, unless you choose a consistent metric. Consistency gives you the ability to compare values in a meaningful way. Will update that answer to clarify, thanks for pointing this out. – Francesco Callari Nov 20 '17 at 16:50
  • Thank you Francesco..Actually I am not able to come to conclusion, which metric best suits..at present I am getting error between 0.4 to 2..at the edges it is almost 3,4.. – user4973313 Nov 20 '17 at 17:47
  • Initially i tried by drawing line between two points and then found distance of all corner points (expected to be on line) from the line. The error used to be 1 or 2 pixels at the center and more at the edge. It is observed to be biased and moved to line fitting, as you say this is also not a proper metric. Will the line drawing method itself will work if i do any fine tuning, anyway corner points are detected using corner finding algorithm. But the error using line fitting and line drawing method is varying relatively for same points. – user4973313 Nov 21 '17 at 07:21