Constrain pitch, yaw, & roll

Question

I've a rotation represented as a quaternion and am trying to constrain the pitch, yaw, & roll axes. I tried doing so thusly:

public struct Orientation
{
    public Vector3 up, forward;

    public Orientation(Vector3 up, Vector3 forward)
    {
        this.up = up;
        this.forward = forward;
    }
}

public static Orientation[] orientations = new Orientation[3]
{
    new Orientation(Vector3.right, Vector3.up),
    new Orientation(Vector3.up, Vector3.forward),
    new Orientation(Vector3.forward, Vector3.right)
};

public enum Axis
{
    Pitch,
    Yaw,
    Roll
};

private Vector3 ConstrainAxis(Vector3 vector, Axis axis, float from, float to)
{
    Orientation orientation = orientations[(int)axis];

    float theta = (to - from) * 0.5F;

    Vector3 cons = Quaternion.AngleAxis(from + theta, orientation.up) * orientation.forward;
    Vector3 proj = Vector3.ProjectOnPlane(vector, orientation.up);

    return ConstrainVector(cons.normalized, proj.normalized, theta);
}

private Vector3 ConstrainVector(Vector3 from, Vector3 to, float angle)
{
    float theta = Mathf.Abs(angle / Vector3.Angle(from, to));

    if(theta < 1.0F)
    {
        return Vector3.Slerp(from, to, theta);
    }

    return to;
}

Which turned out to be nothing more than an over-complicated way of constraining the individual components of an euler angle representation, of which both are subject to a strange jittering issue (gimbal lock related?).

What is the best approach to constraining these axes?

Answer 1

For joint constraints it is common practice to use "swing twist" parametrization. To represent current rotation as "swing twist" for quaternions, theare are good decomposition https://web.archive.org/web/20160909191250/https://www.alinenormoyle.com/weblog/?p=726

And constraint for "swing" and "twist" can be done with quaternions.

if we want to constrain swing to +-30 degrees , pseudocode looks like

Quaternion swing;
const double maxMagnitude = sin(0.5 * toRad(30));
const double maxMagnitudeW = sqrt(1.0 - maxMagnitude * maxMagnitude);
if (swing.vec().normSqr() > maxMagnitude * maxMagnitude)
{
    swing.vec() = swing.vec().normalized() * maxMagnitude;
    swing.w() = maxMagnitudeW;
}

Answer 2

Adding to minorlogic's answer: it is important to save the the sign of targetQuat's W component. Here is a three.js implementation of twist constraint. Also seems there are some singularities I have not checked for: http://www.allenchou.net/2018/05/game-math-swing-twist-interpolation-sterp/

const HEAD_YAW_MAX = 40
const MAX_MAGNITUDE = Math.sin(0.5 * THREE.Math.degToRad(HEAD_YAW_MAX));
const MAX_MAGNITUDE_W = Math.sqrt(1.0 - MAX_MAGNITUDE * MAX_MAGNITUDE);
const MAX_MAG_POW_2 = MAX_MAGNITUDE * MAX_MAGNITUDE;

in update function

const qT = this.headBone.quaternion;
v1.set(qT.x, qT.y, qT.z); //todo check singularity: rotation by 180
v1.projectOnVector(this.headBone.up); //up is direction around twist
// v1.set(0, qT.y, 0); //project on y axis
q1.set(v1.x, v1.y, v1.z, qT.w); //twist
q1.normalize();
q3.copy(q1).conjugate();
q2.multiplyQuaternions(qT, q3); //swing
q2.normalize();
v1.set(q1.x, q1.y, q1.z);
if (v1.lengthSq() > MAX_MAG_POW_2) {
   v1.setLength(MAX_MAGNITUDE);
   const sign = qT.w < 0 ? -1 : 1;
   q1.set(v1.x, v1.y, v1.z, sign * MAX_MAGNITUDE_W);
   this.headBone.quaternion.multiplyQuaternions(q2, q1); //swing * twist
}

The source for the swing twist parameterization algorithm: Component of a quaternion rotation around an axis

Answer 3

It appears that this question is relatively popular, so I will expand upon the answer that minorlogic had provided. With minorlogic's assistance I had ended up creating this "QuaternionExtension" class which can both decompose a quaternion into its swing & twist components and constrain a quaternion about an arbitrary axis. I apologize for not having shared this 6 years ago, but here it is now.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public static class QuaternionExtension
{
    public static void Decompose(this Quaternion quaternion, Vector3 direction, out Quaternion swing, out Quaternion twist)
    {
        Vector3 vector = new Vector3(quaternion.x, quaternion.y, quaternion.z);
        Vector3 projection = Vector3.Project(vector, direction);

        twist = new Quaternion(projection.x, projection.y, projection.z, quaternion.w).normalized;
        swing = quaternion * Quaternion.Inverse(twist);
    }

    public static Quaternion Constrain(this Quaternion quaternion, float angle)
    {
        float magnitude = Mathf.Sin(0.5F * angle);
        float sqrMagnitude = magnitude * magnitude;

        Vector3 vector = new Vector3(quaternion.x, quaternion.y, quaternion.z);

        if (vector.sqrMagnitude > sqrMagnitude)
        {
            vector = vector.normalized * magnitude;

            quaternion.x = vector.x;
            quaternion.y = vector.y;
            quaternion.z = vector.z;
            quaternion.w = Mathf.Sqrt(1.0F - sqrMagnitude) * Mathf.Sign(quaternion.w);
        }

        return quaternion;
    }
}