How to find the area that two triangles are in contact

Question

Given the positions of the vertices, and the surface normals. How can I calculate the area(which may be 0) that the 2 triangles are in contact? These triangles are also in 3D space so if they aren't lined up properly, just jammed into each other, the contact area should be 0. (In C#)

Answer 1

This is not a trivial problem, so let's break it down into steps.

1. Check if the two triangles are coplanar, otherwise the area is 0.
2. Project the triangles onto a 2D surface
3. Calculate the intersection polygon
4. Calculate the area

---

1. Checking for coplanarity

For the two triangles to be coplanar, all vertices of one triangle must lie in the plane determined by the other one.

Using the algorithm described here we can check for every vertex whether that is the case, but due to the fact that floating point numbers are not perfectly precise, you will need to define some error theshold to determine what still counts as coplanar.

Assuming va and vb are the vectors of the triangles A and B respectively, the code could look something like this.
(Note: I have never worked with Unity and am writing all of the code from memory, so please excuse if it isn't 100% correct).

public static bool AreTrianglesCoplanar(Vector3[] va, Vector3[] vb) {
    // make sure these are actually triangles
    Debug.Assert(va.Length == 3);
    Debug.Assert(vb.Length == 3);

    // calculate the (scaled) normal of triangle A
    var normal = Vector3.Cross(va[1] - va[0], va[2] - va[0]);

    // iterate all vertices of triangle B
    for(var vertex in vb) {
        // calculate the dot product between the normal and the vector va[0] --> vertex
        // the dot product will be 0 (or very small) if the angle between these vectors
        // is a right angle (90°)
        float dot = Vector3.Dot(normal, vertex - va[0]).

        // the error threshold
        const float epsilon = 0.001f;
        
        // if the dot product is above the threshold, the vertex lies outside the plane
        // in that case the two triangles are not coplanar
        if(Math.Abs(dot) > epsilon)
            return false;
    }

    return true;
}

2. Projecting the triangles into 2D space

We now know that all six vertices are in the same 2D plane embedded into 3D space, but all of our vertex coordinates are still three-dimensional. So the next step would be to project our points into a 2D coordinate system, such that their relative position is preserved.

This answer explains the math pretty well.
First, we need to find a set of three vectors forming an orthonormal basis (they must be orthoginal to each other and of length 1).

1. One of them is just the plane's normal vector, so we need two more vectors that are orthogonal to the first, and also orthogonal to each other.
2. By definition, all vectors in the plane defined by our triangles are orthogonal to the normal vector, so we can just pick one (for example the vector from va[0] to va[1]) and normalize it.
3. The third vector has to be orthogonal to both of the others, we can find such a vector by taking the cross product of the previous two.

We also need to choose a point in the plane as our origin point, for example va[0].

With all of these parameters, and using the formula from the linked amswer, we can determine our new projected (x, y) coordinates (t_1 and t_2 from the other answer). Note that -- because all of our points lie in the plane defining that normal vector -- the third coordinate (called s in the other answer) will always be (close to) zero.

public static void ProjectTo2DPlane(
        Vector3[] va, Vector3[] vb
        out Vector2[] vaProjected, out Vector2[] vbProjecte
    ) {
    // calculate the three coordinate system axes
    var normal = Vector3.Cross(va[1] - va[0], va[2] - va[0]).normalized;
    var e1 = Vector3.Normalize(va[1] - va[0]);
    var e2 = Vector3.Cross(normal, e1);

    // select an origin point
    var origin = va[0];

    // projection function we will apply to every vertex
    Vector2 ProjectVertex(Vector3 vertex) {
        float s = Dot(normal, vertex - origin);
        float t1 = Dot(e1, vertex - origin);
        float t2 = Dot(e2, vertex - origin);
        // sanity check: this should be close to zero
        // (otherwise the point is outside the plane)
        Debug.Assert(Math.Abs(s) < 0.001);
        return new Vector2(t1, t2);
    }

    // project the vertices using Linq
    vaProjected = va.Select(ProjectVertex).ToArray();
    vbProjected = vb.Select(ProjectVertex).ToArray();
}

Sanity check:

• The vertex va[0] should be projected to (0, 0).
• The vertex va[1] should be projected to (*, 0), so somewhere on the new x axis.

3. / 4. Calculating the intersection area in 2D

This answer this answer mentions the algorithms necessary for the last step.
The Sutherland-Hodgman algorithm successively clips one triangles with each side of the other. The result of this will be either a triangle, a quadrilateral or an empty polygon.

Finally, the shoelace formula can be used to calculate the area of the resulting clipping polygon.

Bringing it all together

Assuming you implemented the two functions CalculateIntersecionPolygon and CalculatePolygonArea, the final intersection area could be calculated like this:

public static float CalculateIntersectionArea(Mesh triangleA, Mesh triangleB) {
    var verticesA = triangleA.GetVertices();
    var verticesB = triangleB.GetVertices();

    if(!AreTrianglesCoplanar(verticesA, verticesB))
        return 0f; // the triangles are not coplanar

    ProjectTo2DPlane(verticesA, verticesB, out Vector2[] projectedA, out Vector2[] projectedB);

    CalculateIntersecionPolygon(projectedA, projectedB, out List<Vector2> intersection);

    if(intersection.Count == 0)
        return 0f; // the triangles didn't overlap

    return CalculatePolygonArea(intersection);
}