0

I saw a lot of similar questions, but not this exact one.

I know:

  • pixel coordinates (u,v) and depth (d) - 4 points
  • corresponding world coordinates (x,y,z) - 4 points
  • intrinsic matrix K of the camera (I didn't calibrate yet but used the camera default one)
  • extrinsic matrix [R|t] (I call its 4x4 version M_ext)
  • conversion from world coordinate points X (3D) to pixel coordinate points p (2D)

I don't know:

  • conversion from pixel coordinate points p (2D) + depth d to world coordinate points X (3D)

0. Get extrinsic matrix:

To get the extrinsic matrix I use opencvs function solvePnP:

ret, rvec, tvec = cv.solvePnP(world_points, img_points, K, np.float32([]), cv.SOLVEPNP_IPPE)

I get the extrinsic matrix [R|t] using the rotation and translation vector above ( Rt = np.hstack((cv.Rodrigues(rvec)[0], tvec))).

I want to apply both conversions for validation that my extrinsic matrix is correct.

1. World coordinate points to pixel coordinates:

camera_coordinates = K * M_ext * world_coordinates 
u = camera_coordinates[0] / camera_coordinates[2]
v = camera_coordinates[1] / camera_coordinates[2]

This conversion works, my pixel coordinates u, v are correct.

2. Pixel coordinates + depth to world coordinate points:

camera_coordinates = ( inv(K) * [u, v, 1] ) * d
world_coordinates = inv(M_ext) * [camera_coordinates, 1]

This inverse conversion does not yield my world coordinate points. Does anyone see where my error is?

meen
  • 13
  • 1
  • 5
  • so you are looking for projecting pixels to 3D with known depth, intrinsics and extrinsics? Have a look at: https://stackoverflow.com/questions/31265245/extracting-3d-coordinates-given-2d-image-points-depth-map-and-camera-calibratio/31266627#31266627 and the linked tutorial by nicolas burrus – Micka Apr 13 '21 at 13:51
  • In the tutorial they also project pixel + depth to 3D camera coordinates the same way I do, but there is no additional transformation from camera frame to another frame / no extrinsic matrix. So unfortunately it doesn't help me finding my error.. – meen Apr 13 '21 at 14:20
  • P3D' = R.P3D + T is transformation from 3d camera coordinate system to global coordinate system. – Micka Apr 13 '21 at 14:23
  • That's what I do, so I still don't see why it doesn't work. In solvePnP the rotation and translation vector "brings points from the model coordinate system to the camera coordinate system", so I applied the inverse to get from camera coordinates to global coordinates. – meen Apr 13 '21 at 14:33
  • ok, that's a misunderstanding. Model coordinates are typically not equal to the global coordinate system, unless you define the global coordinate system to be equal to the model coordinates. Do you see the model in all the cameras at the same time? – Micka Apr 13 '21 at 15:49
  • My model coordinate system is my global coordinate system (four points on a table, visible by my camera). I only have one (depth) camera and I want to know how to convert a pixel+depth point to my model/global coordinate system. – meen Apr 14 '21 at 08:59

1 Answers1

0

first, sorry for before . then i need say i was in the same situation as you, and the problem had been solved already. i tried several different strategies, the following 2 solved my problem:

strategy #1

  1. kicked out a few images from my calibration images set.
  2. calibrated my camera again.
  3. applied the inverse conversion with new parameters.

problem solved!

strategy #2

  1. took a few more cal-tab images and added to images set.
  2. calibrated my camera again.
  3. applied the inverse conversion with new parameters.

problem solved!

i think the problem is caused by some outlier data when try to calibrate camera. but i didn't read the source code of opencv , so i don't know in detail, . hope my experience can help.

hugh chan
  • 1
  • and i performed the conversion like this: world_coordinates = R.inv() * (d * K.inv() * pixel_coordinates - t); – hugh chan Apr 29 '23 at 14:43