Darken/Lighten ARKit camera feed behind objects - ios

Question

I'm currently working on an ARKit project where I would like to darken the actual camera feed so the objects in my 3D scene stand out more.
2 Solutions I found so far:
A) Manually applying CIFilter to the camera frames and setting those as background image to the SceneKit scene as answered in this SO post
The problem here is that fps tanks significantly.
B) Set a background color like so:
sceneView.scene.background.contents = UIColor(white: 0.0, alpha: 0.2)
Sadly, colors with alpha <1 are still opaque, so no matter what alpha I set I can't see anything of the camera feed.
Can anyone think of a different trick to darken the camera feed?

Answer 1

Your option B doesn't work for two reasons:
the scene view is opaque, so there's nothing behind it for a partially transparent background color to blend with.
sceneView.scene.background is what actually displays the camera image, so if you set it to a color you're not displaying the camera image at all anymore.
Some other options (mostly untested) you might look into:
As referenced from the answer you linked, use SCNTechnique to set up multipass rendering. On the first pass, render the whole scene with the excludeCategoryMask (and your scene contents) set up to render nothing but the background, using a shader that dims (or blurs or whatever) all pixels. On the second pass, render only the node(s) you want to appear without that shader (use a simple pass-through shader).
Keep your option B, but make the scene view non-opaque. Set the view's backgroundColor (not the scene's background) to a solid color, and set the transparency of the scene background to fade out the camera feed against that color.
Use geometry to create a "physical" backdrop for your scene — e.g. an SCNPlane of large size, placed as a child of the camera at some fixed distance that's much farther than any other scene content. Set the plane's background color and transparency.
Use multiple views — the ARSCNView, made non-opaque and with a clear background, and another (not necessarily an SceneKit view) that just shows the camera feed. Mess with the other view (or drop in other fun things like UIVisualEffectView) to obscure the camera feed.
File a bug with Apple about sceneView.background not getting the full set of shading customization options that nodes and materials get (filters, shader modifiers, full shader programs) etc, without which customizing the background is much more difficult than customizing other aspects of the scene.

Answer 2

I achieved this effect by creating a SCNNode with a SCNSphere geometry and keeping it attached to the camera using ARSCNView.pointOfView.
override func viewDidLoad() {
super.viewDidLoad()
let sphereFogNode = makeSphereNode()
arView.pointOfView!.addChildNode(sphereFogNode)
view.addSubview(arView)
}
private static func makeSphereGeom() -> SCNSphere {
let sphere = SCNSphere(radius: 5)
let material = SCNMaterial()
material.diffuse.contents = UIColor(white: 1.0, alpha: 0.7)
material.isDoubleSided = true
sphere.materials = [material]
return sphere
}
private static func makeSphereNode() -> SCNNode {
SCNNode(geometry: makeSphereGeom())
}
Clipping Outside Sphere
This darkens the camera along with anything outside the bounds of the sphere. Hit testing (ARFrame.hitTest) does not respect the sphere boundary. You can receive results from outside the sphere.
Things which are outside your sphere will be seen through the sphere's opacity. It seems that non-transparent things will become transparent outside the sphere.
The white part is the plane inside a sphere and the grey is the plane outside the sphere. The plane is a solid white and non-transparent. I tried using SCNScene.fog* to clip SceneKit graphics outside the sphere, but it seems that fog doesn't occlude rendered content, just affects its appearance. SCNCamera.zFar doesn't work also as it clips based on the Z-distance, not on the straight line distance between the camera and target.
Just make your sphere big enough and everything will look fine.

Answer 3

For those who want to implement rickster's option "Use geometry to create a "physical" backdrop for your scene" here is my code (plane and transparency of it made in .scnassets)
guard let backgroundPlane = sceneView.scene.rootNode.childNode(withName: "background", recursively: false) else { return }
backgroundPlane.removeFromParentNode()
backgroundPlane.position = SCNVector3Make(0, 0, -2)
sceneView.pointOfView?.addChildNode(backgroundPlane)