In which space to do Visible Surface Determination and how?

Question

The Visibility Problem could be solved by sorting polygons by depth (Painter's algorithm) and support with Newell's algorithm in cases where z extents overlap. Newell's algorithm is explained here and here. Point 3 and 4 involves comparing each vertex of polygon1 with the plane of polygon2.

To get correct result, we should do the comparison after clipping (as a polygon after clipping may change to entirely be on the opposite side of the other polygon's plane from the viewpoint). Clipping is done after projection (as the projection matrix defines the frustum to clip against), which means we have to compare vertices against plane after projection.

Further, we have to normalize the vectors as x, y and z values in projection space only are applicable in their own context of w.

After normalizing we don't anymore have a linear depth, meaning the z value of two polygons becomes closer if you move them closer to the far plane. Calculating normals, point-plane distance or anything needed to detect which side of a plane a point is, will fail in a non-linear-depth space.

So, how to carry out the step "Do all vertices of P lie deeper than the plane of Q"?

Answer 1

After narrowing the problem to "How could I get the clipped polygon back to a space where its normal vector could be calculated (meaning world space or eye space)", an approach appeared to me: We need to reverse our pipeline operations, description here.

To avoid the non-trivial math getting from window space back to clip space, we may cache the clip coords when going down the pipeline. Then we just have to transform the clip coords with the inverse of the projection matrix. The clipped coords will then be in eye space and we are able to do the vertices-plane comparison (eg. by dot product of plane-to-vector normal and a reference vector or test if there is a line(ray)-plane intersection) and finish our VSD operation.

If we are doing lighting calculations (in eye space), we probably already have the normal vector. One thing to remember when transforming normal vector down the pipeline (eg. from object space to world space) is to use the transposed inverse of the matrix so the normal vector is not translated in any direction.