You have to Orthogonalize the vectors. A common way for the Orthogonalization is the Gram–Schmidt Orthonormalization.
This algorithm uses the circumstance that, the dot product of 2 vectors is equal the cosine of the angle between the 2 vectors multiplied by the magnitude (length) of both vectors.
dot( N, T ) == length( N ) * length( T ) * cos( angle_N_T )
This follows, that the dot product of 2 unit vectors (normalized vectors) is equal the cosine of the angle between the 2 vectors, because the length of a unit vector is 1.
uN = normalize( A )
uT = normalize( B )
cos( angle_T_N ) == dot( uT, uN )

If realNormal
is a normalized vector (its length is 1) and tangent
and binormal
are orthogonal, then the realTangent
and the the realBinormal
can be calculated like this:
realTangent = normalize( tangent - realNormal * dot(tangent, realNormal) );
realBinormal = binormal - realNormal * dot(binormal, realNormal);
realBinormal = normalize( realBinormal - realTangent * dot(realBinormal, realTangent) );
If tangent
and binormal
are normalized vectors too, then the normalize
function can be substituted by dividing with the dot product of the source vector and the real vector:
realTangent = tangent - realNormal * dot(tangent, realNormal);
realTangent /= dot(tangent, realTangent);
realBinormal = binormal - realNormal * dot(binormal, realNormal);
realBinormal = realBinormal - realTangent * dot(realBinormal, realTangent);
realBinormal /= dot(binormal, realBinormal);
See further How to calculate Tangent and Binormal?.