Can't limit the camera pitch angle between [-90º, 90º] in OpenGL using vectors!

Question

I'm having a big problem limiting the camera pitch angle (between -90º and 90º) with the code below. This is somewhat a follow up to this question.

The problem, it seems, is that the camera rotates more than -90º or more than +90º and when that happens, I'll be looking down (or up) but but the view just rotated 180º around the Y axis.

Example: I'm facing north looking at the horizon and I start to look down until I can't go down anymore (limited by the code below). Then I start to look up and I'll be facing south.

void Camera::Rotate(Vector3D angle) {
    angle = angle * CAMERA_ROTATION_SPEED;

    accumPitchAngle += angle.x;

    if(accumPitchAngle > 90.0f) {
        angle.x = 90.0f - (accumPitchAngle - angle.x);
        accumPitchAngle = 90.0f;
    }

    if(accumPitchAngle < -90.0f) {
        angle.x = -90.0f - (accumPitchAngle - angle.x);
        accumPitchAngle = -90.0f;
    }

    // Rotate along the WORLD_SKY_VECTOR axis (yaw/heading rotation)
    // WORLD_SKY_VECTOR = (0.0f, 1.0f, 0.0f)
    if(angle.y != 0.0f) {
        Reference = RotateArbitraryAxis(Reference, WORLD_SKY_VECTOR, angle.y);
        RightVector = Vector3D::CrossProduct(Reference, WORLD_SKY_VECTOR);
        UpVector = Vector3D::CrossProduct(RightVector, Reference);
    }

    // Rotate along the x axis (pitch rotation)
    if(angle.x != 0.0f) {
        Reference = RotateArbitraryAxis(Reference, RightVector, angle.x);
        UpVector = Vector3D::CrossProduct(RightVector, Reference);
    }

    // Makes sure all vectors are perpendicular all the time
    Reference.Normalize();
    RightVector = Vector3D::CrossProduct(Reference, UpVector);
    RightVector.Normalize();
    UpVector = Vector3D::CrossProduct(RightVector, Reference);
    UpVector.Normalize();
}

Vector3D Camera::RotateArbitraryAxis(const Vector3D v, Vector3D u, double angle) {
    Vector3D result;

    u.Normalize();

    double scalar = Vector3D::DotProduct(v, u);
    double c = cos(Math::DegreesToRadians(angle));
    double s = sin(Math::DegreesToRadians(angle));
    double a = 1.0f - c;

    result.x = u.x * scalar * a + (v.x * c) + ((-u.z * v.y) + (u.y * v.z)) * s;
    result.y = u.y * scalar * a + (v.y * c) + (( u.z * v.x) - (u.x * v.z)) * s;
    result.z = u.z * scalar * a + (v.z * c) + ((-u.y * v.x) + (u.x * v.y)) * s;

    return result;
}

The problem is probably in the if(angle.y != 0.0f) statement, if I comment that code block, the problem doesn't exist at all. It has something to do with WORLD_SKY_VECTOR, but that code is like that to allow me to rotate the heading and keep the camera leveled. If I used the UpVector instead, problem solved. But that is only good for a flight simulator, I need to keep the horizon leveled, that's the reason behind WORLD_SKY_VECTOR. But it seems that's the cause of "side switching" when I point the camera straight down.

As per requested on a comment below... This is for a first person (and third person but I haven't started implementing that part yet) camera and when I look straight down, -90º (or straight up, +90º) and when the angle goes from -89º to -91º (or from +89º to +91º) I want the camera to prevent that and don't go beyond the -90º, +90º limit. When it reaches that limit I need the camera to be able to go back (either up if I'm at -90º or down if I'm at +90º). Right now this only works sometimes, other times I'll be facing the other way instead of the way I was initially looking at.

Answer 1

Problem 1 : The code belew "// Makes sure all vectors are perpendicular all the time" makes sure that your UP camera vector is actually up (with respect to WORLD_SKY_VECTOR). So :

• When you look completely down, "up" does not mean a lot. In your case, your camera_up shoud be towards North, not thowards the sky
• When you keep turning, past that point, the camera rotates to keep its up towarks the sky.

Look down at your chest. The top of your head is actually down. If you want to look at the same thing, from the same point, but with the top of your head up, you'd have to rotate it (not humanly feasible, sadly)

Problem 2 : Same as above. CrossProduct(Reference, WORLD_SKY_VECTOR) gives cross(up,down) which does not mean anything, just try on paper.

If you want to be able to look "upside down" :

• Compute your view direction vector. You know reliably that because you have your orientation and your pitch.
• Compute your right vector. It's the same, but with an orientation of 90° more, and a 0 pitch ( camera is always horizontal, i.e. you don't bend your head)
• Compute your up vector as cross(right,front).

So :

yaw += whatever;
pitch += whatever;
FrontVector = SphericalToCartesian(yaw, pitch);
RightVector = SphericalToCartesian(yaw + PI/2.0f, 0.0);
UpVector = cross(RightVector, FrontVector);

with SphericalToCartesian beeing something like if (pitch=0,yaw=0) means looking to south:

x = cos(pitch) * sin(yaw)
y = sin(pitch)
z = cos(pitch) * cos(yaw)

Answer 2

You can't take the cross product of two parallel vectors. I think that's where it's failing, i.e. when accumPitchAngle is ±90°.

You might want to limit it to -89.999° ~ +89.999°.

Edit: To start from the beginning, you want to convert pitch & yaw into a forward vector and an up vector for gluLookAt(), right? Then I suggest:

1) Use yaw (and yaw only) to create a right vector that is parallel to the ground.
2) Cross right and WORLD_SKY_VECTOR to get a forward vector that is correct in yaw but not in pitch.
3) Rotate forward around right through pitch degrees to get a forward vector that is correct in both pitch and yaw. (I think you've got this far.)
4) Cross forward and right to get a camera up vector that will work in all cases.

Answer 3

After while I come up with a solution which is pretty simple and straightforward. Like it was said many times in the answers and comments, the problem lies when the forward vector (my Reference) is looking straight up or down, which means this vector is parallel to WORLD_SKY_VECTOR and that's reason it gets all messy.

My thought behind this solution is that when I'm looking straight down (or up) and I want to rotate left or right, this is actually a roll rotation around the forward vector. Why not execute a roll movement when the pitch angle is at -90º or 90º instead?

Putting that together, I solved the problem by simply replacing the yaw/heading rotation with the following code:

if(angle.y != 0.0f) {
    if(abs(accumPitchAngle) == 90.0f) {
        RightVector = RotateArbitraryAxis(RightVector, Reference, -angle.y);
        UpVector = Vector3D::CrossProduct(RightVector, Reference);
    } else {
        Reference = RotateArbitraryAxis(Reference, WORLD_SKY_VECTOR, angle.y);
        RightVector = Vector3D::CrossProduct(Reference, WORLD_SKY_VECTOR);
        UpVector = Vector3D::CrossProduct(RightVector, Reference);
    }
}

This seems a fairly simple and straightforward working solution, I'll probably be marking my answer as accepted unless someone points out any strong problem against this implementation that I may not be considering. I'll wait a few hours before doing it, so that the previous answers reach some sort of conclusion.