OpenGL camera - View matrix from quaternion behaves incorrectly and pitch is too restricted

Question

I'm creating the view matrix for my camera using its current orientation (quaternion) and its current position.

void Camera::updateViewMatrix()
{
    view = glm::gtx::quaternion::toMat4(orientation);

    // Include rotation (Free Look Camera)
    view[3][0] = -glm::dot(glm::vec3(view[0][0], view[0][1], view[0][2]), position);
    view[3][1] = -glm::dot(glm::vec3(view[1][0], view[1][1], view[1][2]), position);
    view[3][2] = -glm::dot(glm::vec3(view[2][0], view[2][1], view[2][2]), position);

    // Ignore rotation (FPS Camera)
    //view[3][0] = -position.x;
    //view[3][1] = -position.y;
    //view[3][2] = -position.z;

    view[3][3] = 1.0f;
}

There is a problem with this in that I do not believe the quaternion to matrix calculation is giving the correct answer. Translating the camera works as expected but rotating it causes incorrect behavior.

I am rotating the camera using the difference between the current mouse position and the the centre of the screen (resetting the mouse position each frame)

int xPos;
int yPos;
glfwGetMousePos(&xPos, &yPos);

int centreX = 800 / 2;
int centreY = 600 / 2;

rotate(xPos - centreX, yPos - centreY);

// Reset mouse position for next frame
glfwSetMousePos(800 / 2, 600 / 2);

The rotation takes place in this method

void Camera::rotate(float yawDegrees, float pitchDegrees)
{
    // Apply rotation speed to the rotation
    yawDegrees *= lookSensitivity;
    pitchDegrees *= lookSensitivity;

    if (isLookInverted)
    {
        pitchDegrees = -pitchDegrees;
    }

    pitchAccum += pitchDegrees;

    // Stop the camera from looking any higher than 90 degrees
    if (pitchAccum > 90.0f)
    {
        //pitchDegrees = 90.0f - (pitchAccum - pitchDegrees);
        pitchAccum = 90.0f;
    }

    // Stop the camera from looking any lower than 90 degrees
    if (pitchAccum < -90.0f)
    {
        //pitchDegrees = -90.0f - (pitchAccum - pitchDegrees);
        pitchAccum = -90.0f;
    }

    yawAccum += yawDegrees;

    if (yawAccum > 360.0f)
    {
        yawAccum -= 360.0f;
    }

    if (yawAccum < -360.0f)
    {
        yawAccum += 360.0f;
    }

    float yaw = yawDegrees * DEG2RAD;
    float pitch = pitchDegrees * DEG2RAD;

    glm::quat rotation;

    // Rotate the camera about the world Y axis (if mouse has moved in any x direction)
    rotation = glm::gtx::quaternion::angleAxis(yaw, 0.0f, 1.0f, 0.0f);

    // Concatenate quaterions
    orientation = orientation * rotation;

    // Rotate the camera about the world X axis (if mouse has moved in any y direction)
    rotation = glm::gtx::quaternion::angleAxis(pitch, 1.0f, 0.0f, 0.0f);

    // Concatenate quaternions
    orientation = orientation * rotation;
}

Am I concatenating the quaternions correctly for the correct orientation?

There is also a problem with the pitch accumulation in that it restricts my view to ~±5 degrees rather than ±90. What could be the cause of that?

EDIT:

I have solved the problem for the pitch accumulation so that its range is [-90, 90]. It turns out that glm uses degrees and not vectors for axis angle and the order of multiplication for the quaternion concatenation was incorrect.

// Rotate the camera about the world Y axis
// N.B. 'angleAxis' method takes angle in degrees (not in radians)
rotation = glm::gtx::quaternion::angleAxis(yawDegrees, 0.0f, 1.0f, 0.0f);

// Concatenate quaterions ('*' operator concatenates)
// C#: Quaternion.Concatenate(ref rotation, ref orientation)
orientation = orientation * rotation;

// Rotate the camera about the world X axis
rotation = glm::gtx::quaternion::angleAxis(pitchDegrees, 1.0f, 0.0f, 0.0f);

// Concatenate quaterions ('*' operator concatenates)
// C#: Quaternion.Concatenate(ref orientation, ref rotation)
orientation = rotation * orientation;

The problem that remains is that the view matrix rotation appears to rotate the drawn object and not look around like a normal FPS camera.

I have uploaded a video to YouTube to demonstrate the problem. I move the mouse around to change the camera's orientation but the triangle appears to rotate instead.

YouTube video demonstrating camera orientation problem

EDIT 2:

void Camera::rotate(float yawDegrees, float pitchDegrees)
{
    // Apply rotation speed to the rotation
    yawDegrees *= lookSensitivity;
    pitchDegrees *= lookSensitivity;

    if (isLookInverted)
    {
        pitchDegrees = -pitchDegrees;
    }

    pitchAccum += pitchDegrees;

    // Stop the camera from looking any higher than 90 degrees
    if (pitchAccum > 90.0f)
    {
        pitchDegrees = 90.0f - (pitchAccum - pitchDegrees);
        pitchAccum = 90.0f;
    }
    // Stop the camera from looking any lower than 90 degrees
    else if (pitchAccum < -90.0f)
    {
        pitchDegrees = -90.0f - (pitchAccum - pitchDegrees);
        pitchAccum = -90.0f;
    }

    // 'pitchAccum' range is [-90, 90]
    //printf("pitchAccum %f \n", pitchAccum);

    yawAccum += yawDegrees;

    if (yawAccum > 360.0f)
    {
        yawAccum -= 360.0f;
    }
    else if (yawAccum < -360.0f)
    {
        yawAccum += 360.0f;
    }

    orientation = 
        glm::gtx::quaternion::angleAxis(pitchAccum, 1.0f, 0.0f, 0.0f) * 
        glm::gtx::quaternion::angleAxis(yawAccum, 0.0f, 1.0f, 0.0f);
}

EDIT3:

The following multiplication order allows the camera to rotate around its own axis but face the wrong direction:

    glm::mat4 translation;
translation = glm::translate(translation, position);

view = glm::gtx::quaternion::toMat4(orientation) * translation;

EDIT4:

The following will work (applying the translation matrix based on the position after then rotation)

// Rotation
view = glm::gtx::quaternion::toMat4(orientation); 

// Translation
glm::mat4 translation;
translation = glm::translate(translation, -position);

view *= translation;

I can't get the dot product with each orientation axis to work though

// Rotation
view = glm::gtx::quaternion::toMat4(orientation); 

glm::vec3 p(
    glm::dot(glm::vec3(view[0][0], view[0][1], view[0][2]), position),
    glm::dot(glm::vec3(view[1][0], view[1][1], view[1][2]), position),
    glm::dot(glm::vec3(view[2][0], view[2][1], view[2][2]), position)
    );

// Translation
glm::mat4 translation;
translation = glm::translate(translation, -p);

view *= translation;

Answer 1

In order to give you a definite answer, I think that we would need the code that shows how you're actually supplying the view matrix and vertices to OpenGL. However, the symptom sounds pretty typical of incorrect matrix order.

Consider some variables:
V represents the inverse of the current orientation of the camera (the quaternion).
T represents the translation matrix holding the position of the camera. This should be an identity matrix with negation of the camera's position going down the fourth column (assuming that we're right-multiplying column vectors).
U represents the inverse of the change in orientation.
p represents a vertex in world space.
Note: all of the matrices are inverse matrices because the transformations will be applied to the vertex, not the camera, but the end result is the same.

By default the OpenGL camera is at the origin looking down the negative-z axis. When the view isn't changing (U==I), then the vertex's transformation from world coordinates to camera coordinates should be: p'=TVp. You first orient the camera (by rotating the world in the opposite direction) and then translate the camera into position (by shifting the world in the opposite direction).

Now there are a few places to put U. If we put U to the right of V, then we get the behavior of a first-person view. When you move the mouse up, whatever is currently in view rotates downward around the camera. When you move the mouse right, whatever is in view rotates to the left around the camera.

If we put U between T and V, then the camera turns relative to the world's axes instead of the camera's. This is strange behavior. If V happens to turn the camera off to the side, then moving the mouse up and down will make the world seem to 'roll' instead of 'pitch' or 'yaw'.

If we put U left of T, then the camera rotates around the world's axes around the world's origin. This can be even stranger because it makes the camera fly through world faster the farther the camera is from the origin. However, because the rotation is around the origin, if the camera happens to be looking at the origin, objects there will just appear to be turning around. This is sort of what you're seeing because of the dot-products that you're taking to rotate the camera's position.

You check to make sure that pitchAccum stays within [-90,90], but you've commented out the portion that would make use of that fact. This seems odd to me.

The way that you left-multiply pitch but right-multiply yaw makes it so that your quaternions aren't doing much for you. They're just holding your Euler angles. Unless orientation changes are coming in from other places, you could simply say that orientation = glm::gtx::quaternion::angleAxis(pitchAccum*DEG2RAD, 1.0f, 0.0f, 0.0f) * glm::gtx::quaternion::angleAxis(yawAccum*DEG2RAD, 0.0f, 1.0f, 0.0f); and overwrite the old orientation completely.

Answer 2

From what I understand in this tutorial, there might be a reason why pitch angle is restricted at 90 degrees.

Regardless of using quaternions or a look at matrix, at the end, we give an initial orientation to the Camera. In quaternions, this is the initial value of the orientation, in lookAt, it is the initial value of the up vector.

If the direction facing towards the camera is parallel to this initial vector, then the cross product of these will be zero, which means the camera might have any orientation if pitch is 90 or -90 degrees.

In the internal implementation of toMat4(orientation) this would result in one of your x_dir/y_dir/z_dir vectors to be a zero vector, which would mean that your can have any orientation. This is also discussed in this book, which says that if Y angle is 90 degrees, a degree of freedom is lost (Edward Angel and Dave Shreiner, Interactive Computer Graphics, A Top-Down Approach with WebGL, Seventh Edition, Addison-Wesley 2015.), which is discussed as Gimbal Lock.

I can see that you are aware of this problem, but in your code, the yaw angle is still set to 90 degrees if it overflows 90, leaving your Camera in an invalid state. You should consider something like this instead:

if (pitchAccum > 89.999f && pitchAccum <= 90.0f)
{
    pitchAccum = 90.001f;
}
else if (pitchAccum < -89.999f && pitchAccum >= -90.0f)
{
    pitchAccum = -90.001f;
}
if (pitchAccum >= 360.0f)
{
    pitchAccum = 0.0f;
}
else if (pitchAccum <= -360.0f)
{
    pitchAccum = 0.0f;
}

Or you can define another custom action of your choice when pitchAccum is 90 degrees.