What are the maths behind 3D billboard sprites? (was: 3D transformation matrix to 2D matrix)

Question

I have a 3D point in space. The point's exact orientation/position is expressed through a 4x4 transformation matrix.

I want to draw a billboard (3D Sprite) to this point. I know the projected position (i.e. 3D->2D) of the point; the billboard is facing the camera so that's very helpful too. What I don't know is the scaling that the billboard should have!

To make things more complex, the 4x4 matrix may have all sorts of transformations: 3D rotation, 3D scaling, 3D transposition. Assume that the camera is as simple as it can be: position at (0,0,0), no rotation.

So, can I "extract" the scaling of the billboard sprite from this 4x4 matrix?

-

WAS:

I have a 3D affine transformation 4x4 matrix. I need to convert it (project) to a 2D affine transformation 3x3 matrix, which looks like this:

3D rotations are irrelevant and if present may be discarded; I am only interested in translation and most importantly scaling.

Can anyone help with the equations for each of the six 4 values? (lets say t_x, t_y are also known)

Additional info:

The Matrix3D is the global transformation of a 3D point, say (0,0,0). Its purpose is to be projected on a 2D plane (the computer screen).

I know how to project a 3D point to 2D space, what I am looking for is to preserve additional transformation information beyond position, i.e. scaling: as you may know, the scaling property is also altered when projecting the point on a 2D plane.

I also forgot to mention that the perspective projection properties are also known, i.e.:

field of view (single value)
focal length (single value)
projection center (viewpoint position - 2D value)

Answer 1

if you not using spherical coordinate system then this task is not solvable because discarding Z-coordinate before projection will remove the distance form the projection point and therefore you do not know how to apply perspective.

You have two choices (unless I overlooked something):

1. apply 3D transform matrix

   and then use only x,y - coordinates of the result

2. create 3x3 transformation matrix for rotation/projection

   and add offset vector before or after applying it. Be aware that this approach do not use homogenous coordinates !!!

[Edit1] equations for clarity

Do not forget that 3x3 matrix + vector transforms are not cumulative !!! That is the reason why 4x4 transforms are used instead. Now you can throw away the last row of matrix/vector (Xz,Yz,Zz), (z0) and then the output vector is just (x', y'). Of course after this you cannot use the inverse transform because you lost Z coordinate.

Scaling is done by changing the size of axis direction vectors

Btw. if your projection plane is also XY-plane without rotations then:

x' = (x-x0)*d/(z-z0)
y' = (y-y0)*d/(z-z0)

(x,y,z) - point to project
(x',y') - projected point
(x0,y0,z0) - projection origin
d - focal length

[Edit2] well after question edit the meaning is completely different

I assume you want sprite always facing camera. It is ugly but simplifies things like grass,trees,...

M - your matrix
P - projection matrix inside M
If you have origin of M = (0,0,0) without rotations/scaling/skew then M=P
pnt - point of your billboard (center I assume) (w=1) [GCS]
dx,dy - half sizes of billboard [LCS]
A,B,C,D - projected edges of your billboard [GCS]
[GCS] - global coordinate system
[LCS] - local coordinate system

1. if you know the projection matrix

   I assume it is glFrustrum or gluPerspective ... then:

   (x,y,z,w)=(M*(P^-1))*pnt  // transformed center of billboard without projection
   A=P*(x-dx,y-dy,z,w)
   B=P*(x-dx,y+dy,z,w)
   C=P*(x+dx,y+dy,z,w)
   D=P*(x+dx,y-dy,z,w)
   

2. If your M matrix is too complex for #1 to work

   MM=(M*(P^-1))     // transform matrix without projection
   XX=MM(Xx,Xy,Xz)   // X - axis vector from MM [GCS](look at the image above on the right for positions inside matrix)
   YY=MM(Yx,Yy,Yz)   // Y - axis vector from MM [GCS]
   X =(M^-1)*XX*dx   // X - axis vector from MM [LCS] scaled to dx
   Y =(M^-1)*YY*dy   // Y - axis vector from MM [LCS] scaled to dy
   A = M*(pnt-X-Y)
   B = M*(pnt-X+Y)
   C = M*(pnt+X+Y)
   D = M*(pnt+X-Y)
   

[Edit3] scalling only

MM=(M*(P^-1))      // transform matrix without projection
sx=|MM(Xx,Xy,Xz)|  // size of X - axis vector from MM [GCS] = scale x
sy=|MM(Yx,Yy,Yz)|  // size of Y - axis vector from MM [GCS] = scale y

Answer 2

Scale matrix S looks like this:

sx 0  0  0
0  sy 0  0
0  0  sz 0
0  0  0  1

Translation matrix T looks like this:

1  0  0  0
0  1  0  0
0  0  1  0
tx ty tz 1

Z-axis rotation matrix Rlooks like this:

 cos(a) sin(a)  0  0
-sin(a) cos(a)  0  0
   0      0     1  0
   0      0     0  1

If you have a transformation matrix M, it is a result of a number of multiplications of R, T and S matrices. Looking at M, the order and number of those multiplications is unknown. However, if we assume that M=S*R*T we can decompose it into separate matrices. Firstly let's calculate S*R*T:

        ( sx*cos(a) sx*sin(a) 0  0)       (m11 m12 m13 m14)
S*R*T = (-sy*sin(a) sy*cos(a) 0  0) = M = (m21 m22 m23 m24)
        (     0         0     sz 0)       (m31 m32 m33 m34)
        (     tx        ty    tz 1)       (m41 m42 m43 m44)

Since we know it's a 2D transformation, getting translation is straightforward:

translation = vector2D(tx, ty) = vector2D(m41, m42)

To calculate rotation and scale, we can use sin(a)^2+cos(a)^2=1:

(m11 / sx)^2 + (m12 / sx)^2 = 1
(m21 / sy)^2 + (m22 / sy)^2 = 1

m11^2 + m12^2 = sx^2
m21^2 + m22^2 = sy^2

sx = sqrt(m11^2 + m12^2)
sy = sqrt(m21^2 + m22^2)

scale = vector2D(sx, sy)

rotation_angle = atan2(sx*m22, sy*m12)

Source

Hope this help you