ARKit Rendering an object silhouette when behind other objects using SCNTechnique

Question

I want to render a gizmo object in my ARKit app so that it is still visible behind other geometry, but as a darkened silhouette. The Unity shader I found here produces the effect I am trying to reproduce in SceneKit/ARKit.

I've been looking into using SCNTechnique, and have been using the example described here. I have modified the example slightly, but the general gist is a 3 pass shader:

First pass renders the whole scene (draw: DRAW_SCENE)
second pass renders just parts of the gizmo where gizmo depth is greater than scene depth (draw: DRAW_SCENE with includeCategoryMask of the gizmo).
mix the outputs of the first and second passes so that the hidden parts rendered in pass 2 darken pass 1.

Pass 1 is working correctly, as the whole scene is rendered as expected with a black background (as set in the colorStates). However, no matter what I try, in pass 2 the colorStates' "clear" property is always ignored. I want pass 2 to have a black background like pass 1, but it always outputs the camera feed from ARKit. This causes a problem in pass 3 where I cannot correctly blend the colours because pass 2 contains all the camera feed detail, where it should be black. I'm not sure if this is a bug in ARKit, or I am misunderstanding SCNTechniques, but any help would be massively appreciated.

Here is my SCNTechnique:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>passes</key>
    <dict>
        <key>pass_scene</key>
        <dict>
            <key>colorStates</key>
            <dict>
                <key>clearColor</key>
                <string>0 0 0 1</string>
                <key>clear</key>
                <true/>
            </dict>
            <key>draw</key>
            <string>DRAW_SCENE</string>
            <key>inputs</key>
            <dict/>
            <key>outputs</key>
            <dict>
                <key>depth</key>
                <string>DEPTH</string>
                <key>color</key>
                <string>color_scene</string>
            </dict>
        </dict>
        <key>pass_gizmo</key>
        <dict>
            <key>includeCategoryMask</key>
            <integer>2</integer>
            <key>colorStates</key>
            <dict>
                <key>clearColor</key>
                <string>0 0 0 1</string>
                <key>clear</key>
                <true/>
            </dict>
            <key>depthStates</key>
            <dict>
                <key>func</key>
                <string>greater</string>
                <key>enableWrite</key>
                <false/>
                <key>clear</key>
                <false/>
            </dict>
            <key>outputs</key>
            <dict>
                <key>depth</key>
                <string>DEPTH</string>
                <key>color</key>
                <string>color_gizmo</string>
            </dict>
            <key>inputs</key>
            <dict>
                <key>color</key>
                <string>color_gizmo</string>
            </dict>
            <key>draw</key>
            <string>DRAW_SCENE</string>
        </dict>
        <key>mix</key>
        <dict>
            <key>colorStates</key>
            <dict>
                <key>clear</key>
                <true/>
            </dict>
            <key>depthStates</key>
            <dict>
                <key>clear</key>
                <false/>
            </dict>
            <key>inputs</key>
            <dict>
                <key>totalSampler</key>
                <string>color_scene</string>
                <key>gizmoSampler</key>
                <string>color_gizmo</string>
            </dict>
            <key>outputs</key>
            <dict>
                <key>color</key>
                <string>COLOR</string>
            </dict>
            <key>draw</key>
            <string>DRAW_QUAD</string>
            <key>program</key>
            <string>doesntexist</string>
            <key>metalFragmentShader</key>
            <string>gizmo_fragment</string>
            <key>metalVertexShader</key>
            <string>gizmo_vertex</string>
        </dict>
    </dict>
    <key>sequence</key>
    <array>
        <string>pass_scene</string>
        <string>pass_gizmo</string>
        <string>mix</string>
    </array>
    <key>targets</key>
    <dict>
        <key>color_scene</key>
        <dict>
            <key>type</key>
            <string>color</string>
        </dict>
        <key>color_gizmo</key>
        <dict>
            <key>type</key>
            <string>color</string>
        </dict>
    </dict>
    <key>symbols</key>
    <dict>
        <key>vertexSymbol</key>
        <dict>
            <key>semantic</key>
            <string>vertex</string>
        </dict>
    </dict>
</dict>
</plist>

The main region of interest being the colorStates property of the pass_gizmo pass never actually clearing / the camera feed is always rendered anyway;

<key>colorStates</key>
<dict>
<key>clearColor</key>
    <string>0 0 0 1</string>
    <key>clear</key>
    <true/>
</dict>

score 1 · Answer 1 · answered Sep 02 '20 at 05:06

I eventually managed to find a way to create the object silhouette shader that I wanted. It not quite as elegant as I was hoping to achieve, but it is the only solution that I could get to work within ARKit.

I never managed to solve my original issue, but I suspect it was to do with a dodgy combination of input/outputs on my passes.

If anyone stumbles upon this in future wanting to create a similar effect, I achieved it with the 3 pass SCNTechnique described below:

Pass 1, DRAW_SCENE draw the scene as usual. color output to COLOR and depth output to DEPTH
Pass 2, DRAW_SCENE + category mask draw the depth of my gizmo to a depth target: gizmo_depth
Pass 3, DRAW_QUAD custom metal fragment shader that compares the scene depth and the gizmo depth to find the hidden geometry and then multiplies the color of the scene to create a silhouette effect.

It's unfortunate that SCNTechnique is such a hassle to work with because I generally quite like the format. I struggled a lot with sequential passes not sharing targets between each other even though I'm sure they were correctly defined in the targets.

ARKit Rendering an object silhouette when behind other objects using SCNTechnique

1 Answers1