GLSL Fragment Shader: detect that the current fragment is on the edge of a polygon

Question

I'm trying to write a shader that does antialiasing by reducing the opacity at the edge of polygons. So basically, if the fragment is on the edge of a polygon, divide the opacity by two. How can I detect if the fragment is on the edge of a polygon?

Answer 1

There is an approach you can take involving geometry shaders, which I have outlined here. That particular answer was for drawing a wireframe overlay, but the principle is actually the same. If you interpolate barycentric coordinates across your triangles you can tell all fragments that lie within a certain distance from the edge of the triangle and use that to modulate your opacity.

You must be aware, however, that doing this introduces order-dependence to your rendering. Traditional sample-based anti-aliasing or glorified blur filters like FXAA and MLAA/SMAA do not work by altering blend behavior and require no special draw order consideration. But if you want to blend the edges of your softened opaque geometry, you are going to need to draw back-to-front like you would for translucent geometry (the old GL_POLYGON_SMOOTH feature, which works remarkably similarly to what you are describing, has this same limitation).