Migrate from fixed-function to GLSL OpenGL: How should I modify my view (lookAt) matrix and perspection matrix?

Question

I'm trying to migrate my OpenGL rendering pipeline from fixed-point to modern OpenGL. My fixed-point 3D pipeline works well for years. Now when I switch to GLSL, the final projection and object orientation looks like "totally mirrored or swapped". I have excluded all other reasons (like fail to set matrix uniforms, etc.) since I can see the 3D scene is rendered from time to time by randomly dragging my mouse to pan the camera. So the question is narrowed down to the following:

The function that generates view matrix (my function getViewMat(pos, lookat, upaxis))
The function that generates perspective projection matrix (my function getPerspectiveMat(fov, aspect, nearZ, farZ))

Since these two functions generate correct view / projection matrix before for fixed-point pipeline, now I'm wondering whether / how I can modify them to make the generated matrices work for shader pipeline?

There are few stack overflow threads suggests that for shader pipeline, we need to manually flip Z value for projection matrix, but how about the view matrix? Many thanks for any suggestions.

Update: Soure code

/*
    Matrix4f has class members "float m00, m01, m02, m03, m10, m11, ..., m33" representing
    a row-dominant 4x4 matrix, when passed to GLSL, I have remembered to transpose them
*/
void Matrix4f::_getPerspectiveByFov(float fov, float aspectRatio, float nearZ, float farZ, int matrixType)
{
    float halfFov   = MathUtil::degToRad(fov) * 0.5f;
    float width, height;
    //We use the small side out of width and height as the base size, then calculate the other side by aspect ratio.
    float _tanFOV = (float)tan(halfFov);
    if(aspectRatio < 1.f)//width is smaller
    {
        width   = 2.f * nearZ * _tanFOV;
        height  = width / aspectRatio;
    }
    else    //height is smaller
    {
        height  = 2.f * nearZ * _tanFOV;
        width   = height * aspectRatio;
    }

    /*
    Formula from OpenGL reference, see function "glFrustum".
        |w      0       0       0|
        |0      h       0       0|
        |0      0       -C      D|
        |0      0       1       0|

    w = 2.f * nearZ / width
    h = 2.f * nearZ / height
    C = -(farZ + nearZ) / (farZ - nearZ)
    D = -2.f * farZ * nearZ / (farZ - nearZ);
    */
    float w = 2.f * nearZ / width;  // Is equal to: [ 1.f / tan(fov*0.5f) ]
    float h = 2.f * nearZ / height; // Is equal to: [ 1.f / tan(fov*0.5f) / aspectRatio ]
    float C = -(farZ + nearZ) / (farZ - nearZ);
    float D = -2.f * farZ * nearZ / (farZ - nearZ);

    //-----------------------
    m00 = w;
    m01 = 0.f;
    m02 = 0.f;
    m03 = 0.f;

    m10 = 0.f;
    m11 = h;
    m12 = 0.f;
    m13 = 0.f;

    m20 = 0.f;
    m21 = 0.f;
    m22 = -C;
    m23 = D;

    m30 = 0.f;
    m31 = 0.f;
    m32 = 1.f;
    m33 = 0.f;
}

void Matrix4f::_getLookAt(Vector3f& pos, Vector3f& lookat, Vector3f& upAxis)
{
    //Note _forward, _right, _up are working vector of type Vector3f

    _up.set(upAxis);
    _forward.sub(lookat, pos);
    _forward.normalize();
    _right.cross(_up, _forward);
    _right.normalize();
    _up.cross(_forward, _right);
    _up.normalize();

    m00 = _right.x;
    m10 = _right.y;
    m20 = _right.z;

    m01 = _up.x;
    m11 = _up.y;
    m21 = _up.z;

    m02 = _forward.x;
    m12 = _forward.y;
    m22 = _forward.z;

    // Locate the camera
    m03 = pos.x;
    m13 = pos.y;
    m23 = pos.z;

    m30 = 0.f;
    m31 = 0.f;
    m32 = 0.f;

    m33 = 1.f;
}

Thanks for your input, It's a little hard to add all source code, I'll try to modify my question. — Hongkun Wang, May 28 '18 at 21:17
1. My 4x4 matrices are row-dominant, I have transposed them when sending to shader (if I forgot, the rendering result is black-screen) 2. when I multiply matrix in shader, I did this: mat4 mat_vm = mat_view * mat_model; mat4 mat_pvm = mat_proj * mat_vm; — Hongkun Wang, May 28 '18 at 21:30
By following other posts suggestions, after I calculate the projection matrix, I negate z values: m02*-1, -m12*-1, -m22*-1, m32*-1, so just like what you suggested. But I start seeing black-screen until I randomely negate some values in view matrix to make it appear. But I do not know the correct way to adjust view matrix? — Hongkun Wang, May 28 '18 at 21:37
without seeing what are you doing in Vertex shader we can only guess ... — Spektre, May 30 '18 at 07:27

Spektre · Answer 1 · 2018-05-30T07:45:04.103

without seeing what are you doing in Vertex shader we can only guess. However You can use the same matrices in new OpenGL as in the old one. The only matrix you probably do not have is the gluPerspective so you can implement it your self (in case you do not have glu.h anymore). I do it like this:

void glPerspective(double fovy,double aspect,double zNear,double zFar)
    {
    double per[16],f;
    for (int i=0;i<16;i++) per[i]=0.0;
    // original gluProjection
//  f=divide(1.0,tan(0.5*fovy*deg))
//  per[ 0]=f/aspect;
//  per[ 5]=f;
    // corrected gluProjection
    f=divide(1.0,tan(0.5*fovy*deg*aspect));
    per[ 0]=f;
    per[ 5]=f*aspect;
    // z range
    per[10]=divide(zFar+zNear,zNear-zFar);
    per[11]=-1.0;
    per[14]=divide(2.0*zFar*zNear,zNear-zFar);
    glLoadMatrixd(per);
//  zNear=divide(-per[11],per[10]);     // get znear from perspective projection matrix
    }

It more or less mimics gluPerspective but it has repaired the inaccurate tan which caused problems when stacking up more frustrums together. The commented lines show how to compute original gluPerspective (also with tan repaired) and last comment shows how to get znear from matrix (for ray tracers)

Of coarse instead of glLoadMatrixd(per); you use the per as you need ... (like send as uniform)

Now to implement lookat you just construct the matrix your self. See:

Understanding 4x4 homogenous transform matrices

So you simply compute the X,Y,Z vectors and O origin position

Z = -forward
X = +right
Y = +up
O = +camera_position - znear

and feed them into your matrix at desired locations...

Now if your lookat is position of target object than

Z = normalize(camera_position-lookat)
Y = up
X = cross(Y,Z)

Not sure if X should be + or - just try and if scene is mirrored then negate it.

Although I have fixed my problem before I saw your answer, your info is very useful, thanks! My problem is that the pos-lookat-axis matrix generated in my lookAt() function is camera's world transformation matrix, I should use its inverse matrix to pass in the vertex shader as the "view" matrix. — Hongkun Wang, Jun 08 '18 at 14:22

Migrate from fixed-function to GLSL OpenGL: How should I modify my view (lookAt) matrix and perspection matrix?

1 Answers1