Darken/Lighten ARKit camera feed behind objects

Question

I'm currently working on an ARKit project where I would like to darken the actual camera feed so the objects in my 3D scene stand out more.

2 Solutions I found so far:

A) Manually applying CIFilter to the camera frames and setting those as background image to the SceneKit scene as answered in this SO post The problem here is that fps tanks significantly.

B) Set a background color like so:

sceneView.scene.background.contents = UIColor(white: 0.0, alpha: 0.2)

Sadly, colors with alpha <1 are still opaque, so no matter what alpha I set I can't see anything of the camera feed.

Can anyone think of a different trick to darken the camera feed?

score 5 · Accepted Answer · answered Dec 06 '17 at 21:01

Your option B doesn't work for two reasons:

the scene view is opaque, so there's nothing behind it for a partially transparent background color to blend with.
sceneView.scene.background is what actually displays the camera image, so if you set it to a color you're not displaying the camera image at all anymore.

Some other options (mostly untested) you might look into:

As referenced from the answer you linked, use SCNTechnique to set up multipass rendering. On the first pass, render the whole scene with the excludeCategoryMask (and your scene contents) set up to render nothing but the background, using a shader that dims (or blurs or whatever) all pixels. On the second pass, render only the node(s) you want to appear without that shader (use a simple pass-through shader).
Keep your option B, but make the scene view non-opaque. Set the view's backgroundColor (not the scene's background) to a solid color, and set the transparency of the scene background to fade out the camera feed against that color.
Use geometry to create a "physical" backdrop for your scene — e.g. an SCNPlane of large size, placed as a child of the camera at some fixed distance that's much farther than any other scene content. Set the plane's background color and transparency.
Use multiple views — the ARSCNView, made non-opaque and with a clear background, and another (not necessarily an SceneKit view) that just shows the camera feed. Mess with the other view (or drop in other fun things like UIVisualEffectView) to obscure the camera feed.
File a bug with Apple about sceneView.background not getting the full set of shading customization options that nodes and materials get (filters, shader modifiers, full shader programs) etc, without which customizing the background is much more difficult than customizing other aspects of the scene.

Thank you for the detailed answer! I like the multipass option most, but in the short term I resorted to a giant black SCNBox with transparency of 0.5 thats bigger then anything in my scene. It has to be smaller then zFar and zFar seems to be limited to ~13k, so my visible objects can't go further away than that, but it's a hacky solution for now. — Bersaelor, Dec 30 '17 at 21:24
What do you mean `transparency` of the scene `background`? I'm not finding that property.... — Hari Honor, Aug 17 '18 at 10:10

score 1 · Answer 2 · answered Apr 17 '20 at 08:42

I achieved this effect by creating a SCNNode with a SCNSphere geometry and keeping it attached to the camera using ARSCNView.pointOfView.

override func viewDidLoad() {
    super.viewDidLoad()

    let sphereFogNode = makeSphereNode()
    arView.pointOfView!.addChildNode(sphereFogNode)

    view.addSubview(arView)
}

private static func makeSphereGeom() -> SCNSphere {
    let sphere = SCNSphere(radius: 5)
    let material = SCNMaterial()

    material.diffuse.contents = UIColor(white: 1.0, alpha: 0.7)
    material.isDoubleSided = true

    sphere.materials = [material]
    return sphere
}

private static func makeSphereNode() -> SCNNode {
    SCNNode(geometry: makeSphereGeom())
}

Clipping Outside Sphere

This darkens the camera along with anything outside the bounds of the sphere. Hit testing (ARFrame.hitTest) does not respect the sphere boundary. You can receive results from outside the sphere.

Things which are outside your sphere will be seen through the sphere's opacity. It seems that non-transparent things will become transparent outside the sphere.

The white part is the plane inside a sphere and the grey is the plane outside the sphere. The plane is a solid white and non-transparent. I tried using SCNScene.fog* to clip SceneKit graphics outside the sphere, but it seems that fog doesn't occlude rendered content, just affects its appearance. SCNCamera.zFar doesn't work also as it clips based on the Z-distance, not on the straight line distance between the camera and target.

Just make your sphere big enough and everything will look fine.

score 0 · Answer 3 · answered Sep 19 '19 at 13:03

0

For those who want to implement rickster's option "Use geometry to create a "physical" backdrop for your scene" here is my code (plane and transparency of it made in .scnassets)

    guard let backgroundPlane = sceneView.scene.rootNode.childNode(withName: "background", recursively: false) else { return }
    backgroundPlane.removeFromParentNode()
    backgroundPlane.position = SCNVector3Make(0, 0, -2)
    sceneView.pointOfView?.addChildNode(backgroundPlane)

answered Sep 19 '19 at 13:03

AndrewK

907
8
14

Yup, that is one way to do it. Keep in midn though that `-2` is not very far away (2m) and if you have big 3D content it might be too close. In one of my apps I ended up putting the whole 3D scene in a giant ball of 1000m diameter. – Bersaelor Jan 06 '20 at 20:37

Darken/Lighten ARKit camera feed behind objects

3 Answers3

Clipping Outside Sphere