A signed distance from a point P to a segment (A,B) is given by:
d = area(A, B, P) / length(B - A)
where
area(A, B, C) = (B.x - A.x)*(C.y - A.y) - (B.y - A.y)*(C.x - A.x)
Thus the distance you're looking for is given by:
dist0 = area(P, B, C) / length(C - B)
dist1 = area(A, P, C) / length(A - C)
dist2 = area(A, B, P) / length(B - A)
minimum distance = min(dist0, dist1, dist2)
The remaining question is how to map this into the GPU pipeline.
One way is to calculate vec3 dist
at each vertex of a triangle, then let OpenGL interpolate it across:
dist at A = vec3( area(A, B, C)/length(C - B), 0, 0 )
dist at B = vec3( 0, area(A, B, C)/length(A - C), 0 )
dist at C = vec3( 0, 0, area(A, B, C)/length(B - A) )
Then in the fragment shader you only need to calculate the minimum of the interpolated dist components:
in vec3 dist; // interpolated from the above
minimum distance = min(dist.x, min(dist.y, dist.z))
Alternatively, if you already have the barycentric coordinates available (e.g. through the NV_fragment_shader_barycentric extension), you can piggyback on that. Notice that the barycentric coordinates formula is identical to the above, up to the scaling factors:
bary0 = area(P, B, C) / area(A, B, C)
bary1 = area(A, P, C) / area(A, B, C)
bary2 = area(A, B, P) / area(A, B, C)
Thus, if bary
is already available, we only need to scale it by:
dist0 = bary0 * area(A, B, C) / length(C - B)
dist1 = bary1 * area(A, B, C) / length(A - C)
dist2 = bary2 * area(A, B, C) / length(B - A)
Since the scaling factors are constant across the triangle, this can shave off a few FLOPs.
EDIT: I missed that you asked for the actual vector rather than just the distance. The concept remains the same though: you'll need to pass either the normals or the vertices into the shader and based on the computed distances (using either of the above methods), select the normal of the closest edge, scale, and return it. Let's say that n0
, n1
, n2
are the normals opposite to vertices A
, B
, C
correspondingly; then you'd do the following in the fragment shader:
in vec3 dist; // interpolated from the above
in vec2 n0, n1, n2; // normals
...
vec2 result;
if(dist.x < min(dist.y, dist.z))
result = dist.x*n0;
else if(dist.y < dist.z)
result = dist.y*n1;
else
result = dist.z*n2;