Structured-light 3D scanner - depth map from pixel correspondence

Question

I try to create Structured-light 3D scanner.

Camera calibration

Camera calibration is copy of OpenCV official tutorial. As resutlt I have camera intrinsic parameters(camera matrix).

Projector calibration

Projector calibration maybe is not correct but process was: Projector show chessboard pattern and camera take some photos from different angles. Images are cv.undistored with camera parameters and then result images are used for calibration with OpenCV official tutorial. As result I have projector intrinsic parameters(projector matrix).

Rotation and Transition

From cv.calibrate I have rotarion and transition vectors as results but vectors count are equal to images count and I thing it is not corect ones because I move camera and projector in calibration. My new idea is to project chessboard on scanning background, perform calibration and in this way I will have Rotation vector and Transition vector. I don't know is that correct way.

Scanning

Process of scanning is:

Generate patterns -> undistor patterns with projector matrix -> Project pattern and take photos with camera -> undistort taken photos with camera matrix

Camera-projector pixels map

I use GrayCode pattern and with cv.graycode.getProjPixel and have pixels mapping between camera and projector. My projector is not very high resolution and last patterns are not very readable. I will create custom function that generate mapping without the last patterns.

Problem

I don't know how to get depth map(Z) from all this information. My confution is because there are 3 coordinate systems - camera, projector and world coordinate system.

How to find 'Z' with code? Can I just get Z from pixels mapping between image and pattern?

Information that have:

p(x,y,1) = R*q(x,y,z) + T - where p is image point, q is real world point(maybe), R and T are rotation vector and transition vector. How to find R and T?
Z = B.f/(x-x') - where Z is coordinate(depth), B-baseline(distanse between camera and projector) I can measure it by hand but maybe this is not the way, (x-x') - distance between camera pixel and projector pixel. I don't know how to get baseline. Maybe this is Transition vector?
I tried to get 4 meaning point, use them in cv.getPerspectiveTransform and this result to be used in cv.reprojectImageTo3D. But cv.getPerspectiveTransform return 3x3 matrix and cv.reprojectImageTo3D use Q-4×4 perspective transformation matrix that can be obtained with stereoRectify.

Similar Questions:

How is point cloud data acquired from the structured light 3D scanning? - Answer is you need to define a vector that goes from the camera perspective center to the pixel in the image and then rotate this vector by the camera rotation. But I don't know how to define/find thid vercor and Rotation vector is needed.
How to compute the rotation and translation between 2 cameras? - Question is about R and T between two cameras but almost everywhere writes that projector is inverse camera. One good answer is The only thing you have to do is to make sure that the calibration chessboard is seen by both of the cameras. But I think if I project chessboard pattern it will be additional distored by wall(Projective transormation)

There are many other resources and I will update list with comment. I missed something and I can't figure out how to implement it.

you know how the projector reference image looks like. If you now imagine that the projector instead is a camera, it would see that reference image (with some lens distortion from projector etc. but independent from the scene 3D geometry). So you can interpret the whole setting as a stereo imaging setup, compute disparities and 3D reconstruction just in the stereo way. It might be even easier, but looking at it this way it might be easy to understand. — Micka, Feb 24 '22 at 16:17
I know that I repeat myself, but I thing that projected chessboard image and what camera see can't be used as stereo view point. The reason is that projection will be [distorted from wall](https://www.brilens.com/d/pic/keystone-correction-of-projector.png). Why this distortion doen't metter? If I image there is real chessboard and two cameras, result image on camera(camera in case 1 and case 2) will be different. — Morticia A. Addams, Feb 24 '22 at 16:46
From view of the prejector, the wall wont distort the image. — Micka, Feb 24 '22 at 18:43
Imagine the lcd of the projector to be a camera sensor instead and draw the projection rays. You will see that no matter how the wall looks like, the projector ray and the projection back to the imaginary camera sensor are identical rays. — Micka, Feb 24 '22 at 18:45
Maybe study how MediaPipeHandTracker detects the z-coordinates of hands. — Red, Feb 26 '22 at 22:40

cyborg · Accepted Answer · 2022-02-24T12:17:06.830

1

Lets assume p(x,y) is the image point and the disparity as (x-x'). You can obtain the depth point as,

disparity = x-x_ # x-x'
point_and_disparity = np.array([[[x, y, disparity]]], dtype=np.float32)

depth = cv2.perspectiveTransform(point_and_disparity, q_matrix)

edited Feb 24 '22 at 12:17

answered Feb 24 '22 at 12:05

cyborg

554
2
7

How I can get `q_matrix`? [cv.findHomography](https://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html#findhomography)? – Morticia A. Addams Feb 24 '22 at 14:38
1

You can get the q matrix using stereo calibration. – cyborg Feb 24 '22 at 15:02
1

https://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html#stereorectify – cyborg Feb 24 '22 at 15:07
1

This is good reference for streo calibration https://jayrambhia.com/blog/stereo-calibration – cyborg Feb 24 '22 at 15:08
`stereoRectify` uses `Rotation matrix` and `Translation vector`. In reference link `R` and `T` are result from `stereoCalibrate`, but i am not sure I can use this method if my second camera is projector. Because camera and projector don't '"see" same object from different angles, and Chessboard pattern is projection tranformed from scanner's background. – Morticia A. Addams Feb 24 '22 at 15:36