Unity Shader highlighting overlaps

Question

I am trying to write a shader for unity that will highlight the overlapping fragments of meshes. It should work for one object overlapping itself as well as multiple objects.

The result should look like an attached image.

First I tried to accomplish this with collision detection but I think that the best way is writing a shader.

I'm not very familiar with shaders so if anyone could help me I would be grateful.

I think that it can be done by using stencil shaders like here http://docs.unity3d.com/Manual/SL-Stencil.html but this shaders only render intersection of two objects without rendering whole object.

I also found shader based on Depth (https://chrismflynn.wordpress.com/2012/09/06/fun-with-shaders-and-the-depth-buffer/) but this also work on two objects and doesn't work on one mesh that overlap itself

Regarding @Zze comment with idea about two Pass I have now two shaders. And it works on two objects when one have one shader and other have second one.

Maybe any one can help me how to combine it into one shader that will work also in object that will overlap itself?

ShaderOne

Shader "Custom/ShaderOne"
{
    SubShader {
        Tags { "RenderType"="Opaque" "Queue"="Geometry"}
        Pass {
            Stencil {
                Ref 2
                Comp always
                Pass keep 
                Fail decrWrap 
                ZFail keep
            }

            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            struct appdata {
                float4 vertex : POSITION;
            };
            struct v2f {
                float4 pos : SV_POSITION;
            };
            v2f vert(appdata v) {
                v2f o;
                o.pos = mul(UNITY_MATRIX_MVP, v.vertex);
                return o;
            }
            half4 frag(v2f i) : SV_Target {
                return half4(0,1,0,1);
            }
            ENDCG
        }
        Pass {
            Stencil {
                Ref 2
                Comp equal
            }

            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            struct appdata {
                float4 vertex : POSITION;
            };
            struct v2f {
                float4 pos : SV_POSITION;
            };
            v2f vert(appdata v) {
                v2f o;
                o.pos = mul(UNITY_MATRIX_MVP, v.vertex);
                return o;
            }
            half4 frag(v2f i) : SV_Target {
                return half4(0,0,1,1);
            }
            ENDCG
        }

    } 
}

ShaderTwo

Shader "Custom/ShaderTwo"
{
    SubShader {
        Tags { "RenderType"="Opaque" "Queue"="Geometry"}
        Pass {
            Stencil {
                Ref 2
                Comp always
                Pass replace
                ZFail keep
            }

            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            struct appdata {
                float4 vertex : POSITION;
            };
            struct v2f {
                float4 pos : SV_POSITION;
            };
            v2f vert(appdata v) {
                v2f o;
                o.pos = mul(UNITY_MATRIX_MVP, v.vertex);
                return o;
            }
            half4 frag(v2f i) : SV_Target {
                return half4(1,0,0,1);
            }
            ENDCG
        }
    } 
}

The result looks like an attached image

If the stencil shader renders the intersection points, then why don't you make a shader with 2 passes, the first one draws normally and then the second mimics the result of the stencil shader? — Zze, Feb 23 '16 at 21:25
You might want to look at the notion of depth peeling. See [here](http://www.opengl-tutorial.org/intermediate-tutorials/tutorial-10-transparency/) and [here](http://www.eng.utah.edu/~cs5610/handouts/order_independent_transparency.pdf). Also how many objects are you planning to have. If not many you might want to render each object to a texture and then combine them like a accumulation buffer. — mrVoid, Feb 26 '16 at 13:22
@mrVoid I think that depth testing will work in this situation only with camera top view - but when it will be perspective this will not work? I have headache from this. maybe you can provide me some example of using it with sample code? — seek, Feb 27 '16 at 19:12
Okay @seek. It seems that I have misunderstood what you are aiming for. So its not a 2D topdown view, its a 3d model view with sortof transparency layers drawn yes? I asked what is the amount of geometry that you will have. It might not be doable without implementing some code to accutally calculate geometry boolean operations. — mrVoid, Mar 01 '16 at 11:33

Pidhorskyi · Accepted Answer · 2016-04-16T19:12:33.680

This problem can be solved with help of Stencil buffer and one, two-pass shader. The idea is the following:

First pass compares the value in stencil buffer with 0. In both cases (pass/fail) increase the value in buffer.
Second pass compares the value in stencil buffer with 1. If reference value 1 is less, than we pass and highlight the overlapping pixel.

You might want to add more passes that are the same as the second, but with different reference value to highlight regions that overlaps two times, three times, etc.

In Unity's shaderlab notation, it should be something like that:

    Pass
    {
        Stencil {
            Ref 0
            Comp Equal
            Pass IncrSat 
            Fail IncrSat 
        }

        // Shader for not overlapping regions goes here.
    }

    Pass
    {
        Stencil {
            Ref 1
            Comp Less
        }

        // Shader for one-time overlapping regions goes here.
    }

    Pass
    {
        Stencil {
            Ref 2
            Comp Less
        }

        // Shader for two-time overlapping regions goes here.
    }

Example:

Shader:

Shader "Unlit/Stencil"
{
    Properties
    {
        _MainTex ("Texture", 2D) = "white" {}
    }
    SubShader
    {
        Tags { "RenderType"="Opaque" }
        LOD 100

        Pass
        {
            Stencil {
                Ref 0
                Comp Equal
                Pass IncrSat 
                Fail IncrSat 
            }

            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag

            #include "UnityCG.cginc"

            struct appdata
            {
                float4 vertex : POSITION;
            };

            struct v2f
            {
                float4 vertex : SV_POSITION;
            };

            v2f vert (appdata v)
            {
                v2f o;
                o.vertex = mul(UNITY_MATRIX_MVP, v.vertex);
                return o;
            }

            fixed4 frag (v2f i) : SV_Target
            {
                fixed4 col = fixed4(0.0, 0.0, 1.0, 1.0);
                return col;
            }
            ENDCG
        }

        Pass
        {
            Stencil {
                Ref 1
                Comp Less
            }

            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag

            #include "UnityCG.cginc"

            struct appdata
            {
                float4 vertex : POSITION;
            };

            struct v2f
            {
                float4 vertex : SV_POSITION;
            };

            v2f vert (appdata v)
            {
                v2f o;
                o.vertex = mul(UNITY_MATRIX_MVP, v.vertex);
                return o;
            }

            fixed4 frag (v2f i) : SV_Target
            {
                fixed4 col = fixed4(1.0, 1.0, 0.0, 1.0);
                return col;
            }
            ENDCG
        }

        Pass
        {
            Stencil {
                Ref 2
                Comp Less
            }

            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag

            #include "UnityCG.cginc"

            struct appdata
            {
                float4 vertex : POSITION;
            };

            struct v2f
            {
                float4 vertex : SV_POSITION;
            };

            v2f vert (appdata v)
            {
                v2f o;
                o.vertex = mul(UNITY_MATRIX_MVP, v.vertex);
                return o;
            }

            fixed4 frag (v2f i) : SV_Target
            {
                fixed4 col = fixed4(1.0, 0.0, 0.0, 1.0);
                return col;
            }
            ENDCG
        }
    }
}

Works great!. Thats exactly what I was looking for. Please tell me this shader is rendered on the top of the others, is it possible to add Queue to this? And the other thing is it possible to render this for double sided also? — seek, Apr 15 '16 at 12:06
> Please tell me this shader is rendered on the top of the others, is it possible to add Queue to this? Not sure that I've understood question correctly. You may replace shader in each Pass with your own, to get different shaders for overlapping and not overlapping areas. You may render it on top of the others using Queue. You may render double sided as well. You just need to set `Cull Off`. — Pidhorskyi, Apr 16 '16 at 18:29
I mean that I have other objects on the scene that should be rendered on the top of this object with your shader. — seek, Apr 16 '16 at 18:37
@seek, can you make an example? You can render multiple objects with the same shader in any order, it should work. — Pidhorskyi, Apr 16 '16 at 18:49
For example other object on the scene have shader Unlit/Color and I need that this object should be above object with your shader. Now every objects with different shaders are rendered behind these with your shader. — seek, Apr 16 '16 at 18:52
@seek, I got it. Remove `ZTest Always` from all passes, they are not necessary but they cause your problem. I'll update the answer. — Pidhorskyi, Apr 16 '16 at 19:10

Unity Shader highlighting overlaps

1 Answers1

Linked

Related