19

I know that in the general case, making this conversion is impossible since depth information is lost going from 3d to 2d.

However, I have a fixed camera and I know its camera matrix. I also have a planar calibration pattern of known dimensions - let's say that in world coordinates it has corners (0,0,0) (2,0,0) (2,1,0) (0,1,0). Using opencv I can estimate the pattern's pose, giving the translation and rotation matrices needed to project a point on the object to a pixel in the image.

Now: this 3d to image projection is easy, but how about the other way? If I pick a pixel in the image that I know is part of the calibration pattern, how can I get the corresponding 3d point?

I could iteratively choose some random 3d point on the calibration pattern, project to 2d, and refine the 3d point based on the error. But this seems pretty horrible.

Given that this unknown point has world coordinates something like (x,y,0) -- since it must lie on the z=0 plane -- it seems like there should be some transformation that I can apply, instead of doing the iterative nonsense. My maths isn't very good though - can someone work out this transformation and explain how you derive it?

FusterCluck
  • 493
  • 1
  • 6
  • 19

3 Answers3

16

Here is a closed form solution that I hope can help someone. Using the conventions in the image from your comment above, you can use centered-normalized pixel coordinates (usually after distortion correction) u and v, and extrinsic calibration data, like this:

|Tx|   |r11 r21 r31| |-t1|
|Ty| = |r12 r22 r32|.|-t2|
|Tz|   |r13 r23 r33| |-t3|

|dx|   |r11 r21 r31| |u|
|dy| = |r12 r22 r32|.|v|
|dz|   |r13 r23 r33| |1|

With these intermediate values, the coordinates you want are:

X = (-Tz/dz)*dx + Tx
Y = (-Tz/dz)*dy + Ty

Explanation:

The vector [t1, t2, t3]t is the position of the origin of the world coordinate system (the (0,0) of your calibration pattern) with respect to the camera optical center; by reversing signs and inversing the rotation transformation we obtain vector T = [Tx, Ty, Tz]t, which is the position of the camera center in the world reference frame.

Similarly, [u, v, 1]t is the vector in which lies the observed point in the camera reference frame (starting from camera center). By inversing the rotation transformation we obtain vector d = [dx, dy, dz]t, which represents the same direction in world reference frame.

To inverse the rotation transformation we take advantage of the fact that the inverse of a rotation matrix is its transpose (link).

Now we have a line with direction vector d starting from point T, the intersection of this line with plane Z=0 is given by the second set of equations. Note that it would be similarly easy to find the intersection with the X=0 or Y=0 planes or with any plane parallel to them.

PeterJ
  • 3,705
  • 28
  • 51
  • 71
Milo
  • 2,062
  • 16
  • 38
  • In this case, is t3 = 1? to form homogenous coordinates – user1538798 Jan 06 '14 at 05:20
  • @user1538798: No. [t1, t2, t3] is a 3D position in the world. – Milo Jan 06 '14 at 10:21
  • a little confused over this. If [t1, t2, t3] is a 3D position in the world, would this still makes T = [Tx, Ty, Tz] a 3D position in the world reference? – user1538798 Jan 06 '14 at 10:38
  • @user1538798: Yes. Chech the paragraph just below "Explanation". – Milo Jan 06 '14 at 14:17
  • but the same matrix transform the vector [u v 1] from camera reference to [dx dy dz] in the world reference (in the second equation). while the same rotation matrix warp [-t1 -t2 -t3] of world reference into [Tx Ty Tz] of the same world reference? – user1538798 Jan 07 '14 at 08:44
6

Yes, you can. If you have a transformation matrix that maps a point in the 3d world to the image plane, you can just use the inverse of this transformation matrix to map a image plane point to the 3d world point. If you already know that z = 0 for the 3d world point, this will result in one solution for the point. There will be no need to iteratively choose some random 3d point. I had a similar problem where I had a camera mounted on a vehicle with a known position and camera calibration matrix. I needed to know the real world location of a lane marking captured on the image place of the camera.

Diarmaid O Cualain
  • 300
  • 2
  • 11
  • So for a concrete example, the transformation is [here.](http://docs.opencv.org/_images/math/50a3464c85a412907d91fd8895108ff692eb8d08.png) [X Y Z 1] is the world point, [u v 1] is the image pixel. How do I inverse the middle bit? – FusterCluck Jan 25 '13 at 13:47
  • Thanks, that was the hint I needed to solve the problem. I couldn't use the inverse transformation directly, because the camera point needs to be scaled by some amount to get the object point to z=0. But working out the scaling was trivial to solve. – FusterCluck Jan 27 '13 at 15:05
0

If you have Z=0 for you points in world coordinates (which should be true for planar calibration pattern), instead of inversing rotation transformation, you can calculate homography for your image from camera and calibration pattern.

When you have homography you can select point on image and then get its location in world coordinates using inverse homography. This is true as long as the point in world coordinates is on the same plane as the points used for calculating this homography (in this case Z=0)

This approach to this problem was also discussed below this question on SO: Transforming 2D image coordinates to 3D world coordinates with z = 0

kwmen
  • 63
  • 2
  • 5