18

I am calibrating my camera following a tutorial. I understand the whole process of finding the intrinsic parameter and the distortion coefficients of the camera using the chessboard. What I don't understand is, why after that, we call the getOptimalNewCameraMatrix? Especially with the alpha parameter. Already read the documentation, but maybe because of lacking in knowledge of camera calibration, I really cannot understood it.

So, this is the original image. Original image

Below are samples of the undistorted images of the above image (using OpenCV's undistort).

For this one, I just undistort the image directly using the obtained intrinsic camera and distortion coefficients. Directly undistorted image

As for this one, I call the getOptimalNewCameraMatrix with alpha=0 (left) and alpha=1 (right) before I undistort it.Distorted image with alpha

From what I can see, the getOptimalNewCameraMatrix is preserving the original image without losing information? I hope someone can interpret what this function really do.

And if I want to build a 3D model with Structure from Motion (SfM) with pictures from this camera, should I first call the getOptimalNewCameraMatrix?

Thanks.

Manumerous
  • 455
  • 6
  • 21

4 Answers4

19

I had the same question. I could not find a clear direct answer by searching online, but after running some tests I have an idea what cv2.getOptimalNewCameraMatrix function actually does for the user.

I used the same images mentioned in openCV calibration tutorial (you can find the images in here named left01.jpg to left14.jpg (Actually left10.jpg seems to be missing but that's fine)) After the calibration steps -assuming the square sizes are 30 mm- in the tutorial I got this camera matrix:

mtx = [[534.07088364   0.         341.53407554]
       [  0.         534.11914595 232.94565259]
       [  0.           0.           1.        ]]

The new camera matrix obtained, on the other hand, using the command newcameramtx, roi = cv2.getOptimalNewCameraMatrix(mtx, dist, (w,h), 1, (w,h)) is

newcameramtx = [[457.92434692   0.         342.55548195]
                [  0.         456.2421875  233.34661351]
                [  0.           0.           1.        ]]

Notice that here alpha=1, but it could have been another value between [0,1]. Using another value would give a different newcameramtx.

So far nothing is new. Then, I "undistorted" all the used images during the calibration and saved them into two new folders. One folder (Let's call it the 1st folder) is filled with the undistorted images directly using the obtained intrinsic camera (ie. mtx) and distortion coefficients (dist), and the other one (Let's call it the 2nd folder) is filled with the undistorted images using the newcameramtx and the same distortion coefficients (dist). To be explicit, I used the commands cv2.undistort(img, mtx, dist, None, mtx) and cv2.undistort(img, mtx, dist, None, newcameramtx) respectively and saved the result images into the mentioned two folders. I did NOT cropped the undistorted images using the roi before saving. The result images are just like you have found and gave in the question.

Then, using the undistorted images in the 1st folder, I re-executed the calibration process and I found the following camera matrices:

mtx_1stFolder = [[533.72669262   0.         341.96641109]
                 [  0.         533.73880924 232.9742336 ]
                 [  0.           0.           1.        ]]
newcameramtx_1stFolder = [[533.41644287   0.         341.35446736]
                          [  0.         532.47808838 232.56343633]
                          [  0.           0.           1.        ]]

Doing the same for the 2nd folder gave the following camera matrices:

mtx_2ndFolder = [[458.04299944   0.         342.94915275]
                 [  0.         456.31672142 233.39481945]
                 [  0.           0.           1.        ]]
newcameramtx_2ndFolder = [[458.24960327   0.         342.20507346]
                          [  0.         455.25088501 232.99452259]
                          [  0.           0.           1.        ]]

Now, notice here how close mtx, mtx_1stFolder, and newcameramtx_1stFolder to each other. And similarly notice how close newcameramtx, mtx_2ndFolder, and newcameramtx_2ndFolder to each other as well.

Based on these observations my idea is that undistorting the images directly using the obtained intrinsic camera and distortion coefficients does not affect the camera matrix and it returns the same size images accordingly, but these resulting images loses some pixels. If you are fine with that loss, you are safe to not to use cv2.getOptimalNewCameraMatrix at all. If you need, however, the information given in the lost pixels, you also have an option to undistort your images accordingly setting up an alpha parameter between [0,1] into the same image size or any size you desire. This operation, however, would definitely destroy your camera matrix and you would need a new calibration upon the new undistorted images. cv2.getOptimalNewCameraMatrix function gives an estimation of what would be the new camera matrix without having you to require to do a new calibration. This explains why newcameramtx and mtx_2ndFolder are very close to each other.

This also seems to be explaining why mtx_2ndFolder and newcameramtx_2ndFolder are close to each other, because there is not much left to undistort.(And similarly for mtx_1stFolder and newcameramtx_1stFolder)

One thing I have not mentioned so far is the distortion coefficients(dist) for each calibration processes. This actually keeps me away from saying my idea is for sure, so I am all open to corrections on my idea. Intuitively, I am expecting some distortion coefficients for the first calibration since the images are obviously distorted. As expected, the distortion coefficient vector is found as:

dist = [[-0.29297164  0.10770696  0.00131038 -0.0000311   0.0434798 ]]

On the other hand, I was expecting that the calibration on the undistorted images would give less coefficients, ideally 0 since the all distortions are ideally removed. However, Calibration on the 1st folder gives:

dist = [[ 0.00760897 -0.07313015  0.0002162   0.0003522   0.1605208 ]]

Calibration on the 2nd folder gives:

dist = [[ 0.00135068 -0.02390678  0.0001996   0.0003413   0.0580141 ]]

For the first three terms, the results are as in the expectation way becoming less in the magnitude towards zero, however they are not drastically closer to 0. Also for the last two terms the coefficients are increased which is the opposite of my expectation. If anyone have an idea why this is the case I would highly appreciate.

Burak Aksoy
  • 191
  • 1
  • 5
  • One aspect you didn't consider is the `centerPrincipalPoint` flag. If you look at the image from `undistort` and compare it to the undistorted image with alpha=0, they show different results (the squares in the alpha=0 image don't look square to me). This is because by default, the principal point is recalculated by [`getOptimalNewCameraMatrix`](https://docs.opencv.org/4.7.0/d9/d0c/group__calib3d.html#ga7a6c4e032c97f03ba747966e6ad862b1) while the image centre is used when not using the new matrix. Set this flag to true to get the same result with alpha=0 as when not using the optimal matrix. – IndefiniteBen May 05 '23 at 14:24
8

You will have to lose some pixels if you want to have an image of the same shape as the original image.

The "alpha" parameter description from the opencv docs states...

Free scaling parameter between 0 (when all the pixels in the undistorted image are valid) and 1 (when all the source image pixels are retained in the undistorted image)

To me this means that with a value of 0, all the pixels will get geometrically transformed to be included. The radial and tangential distortion will be removed but your image will not be geometrically correct.

getOptimalNewCameraMatrix is used to use different resolutions from the same camera with the same calibration.

I get the best results when I use getOptimalNewCameraMatrix and then apply a crop for the region of interest that is also returned.

Here is a snippet of python I use for this purpose.

newcameramtx, roi = cv2.getOptimalNewCameraMatrix(mtx, dist, (w, h), 1, (w, h))
mapx, mapy = cv2.initUndistortRectifyMap(mtx, dist, None, newcameramtx, dim, 5)
image = cv2.remap(image, mapx, mapy, cv2.INTER_LINEAR)

x, y, w, h = roi
image = image[y:y + h, x:x + w]

That should take your last image (the pin cushion one on the right) and give you the most usable image.

I also split up the undistort method into its two parts as you can just keep calling remap on new images received from the camera provided that they are the same resolution

p0wdr.com
  • 307
  • 2
  • 4
  • 4
    The interpretation you've given of the alpha parameter is wrong. With alpha=1 all source pixels are undistorted, which can lead to a non-rectangular image region - visible in the black pincushioning. Because the image has to be rectangular in memory, the gaps are padded with black pixels. Setting alpha=1 effectively increases the focal length to have a rectangular undistorted image where all pixels correspond to a pixel in the original. You lose data from the periphery, but you don't have any padded pixels without valid data. – DoMakeSayThink Jan 04 '19 at 10:26
  • The latter half of my comment should read "Setting alpha=0..." – DoMakeSayThink Jan 15 '19 at 16:38
1
  • When we undistort the image using original camera matrix "mtx", the field of view/focal length of undistorted output image is kept the same as the original image i.e. ~534 in this case. Due to this, some of the pixels at the border are lost.
  • By specifying a free scale parameter >0, we are actually asking the algorithm to zoom out a bit so that the border pixels also get included.
  • This results in a larger field of view and a smaller focal length of the undistorted image compared to the source image.
  • Hence, as can be noticed in the new matrix (newcameramtx), the focal length has reduced to ~457 (from ~534) which gives the zoom out effect.
bilva
  • 51
  • 6
0

As you yourself have correctly recognized, it is in principle not necessary to call the getOptimalNewCameraMatrix function. However, the use of this function has some advantages:

  1. Alpha = 1. When called with alpha = 1, all original regions of the distorted image are also included in the rectified image. The new camera matrix is determined in such a way that the corner points of the distorted image are mapped to the corner points of the rectified image. This is the design criterion for the function. However, the use of alpha = 1 has the disadvantage that the rectified image also contains invalid areas that are colored black in your example. If you do not want to use these areas, you can use the ROI values returned by the function.

  2. Alpha = 0. When called with alpha = 0, you get an image that has no invalid areas and retains as much of the original image content as possible, taking this condition into account. In principle, it contains the rectangular area that can be selected in the image with alpha = 1, so that just no invalid areas are determined. The call with alpha = 0 generates images that are very similar to the original image in the central area with slight distortion. If one can do without the information in the peripheral corner areas, I recommend the use with alpha = 0.

enter image description here

See also https://answers.ros.org/question/392890/why-the-left-3x3-portion-of-projection-matrixp-is-different-from-the-intrinsic-camera-matrixk/